Hey, I’m Robert. And I teach deep

learning here at SAS. Today, I’m going to talk to you

about an unsupervised learning method called autoencoders. Now, there are several different

variants of autoencoders. There’s denoising autoencoders

and sparse autoencoders. But you may be wondering

what is an autoencoder. Well, an autoencoder

is a neural network where you have inputs

on your input layer, and you have inputs

on your output layer. Your mathematical neurons that

are in your neural network are referred to as

encoders or decoders. And all these are are just

probabilistic representations of the input space. So each of your

mathematical neurons is going to take a look

at that input space and try to reformulate it

again given the input space. That’s why inputs are on

your input layer and inputs on your output layer. The idea, or our hope,

when we use an autoencoder is to desensitize the irrelevant

and sensitize the relevant. So I had mentioned that

there’s two variants that we’re going to look at, a

denoising autoencoder and a sparse autoencoder. And these are

complicated names that really describe simple ideas. A sparse autoencoder

is an autoencoder where we’re going to constrain

the objective function using an L1 or L2 regularization. That’s all it is. For a denoising

autoencoder, we’re simply applying dropout to

the input layer of our model. Let’s take a look at this. So here, I’m using SAS Studio. And the first thing

that I’m going to do is I’m going to

upload my images. Then I’m going to

view my images. And I’m going to view a

subset of each of the classes. The data that I’m

using in this example is the mnist_fashion data, where

I have 10 different outcomes. Next, I’m going to resize

the images to be a 32 by 32, because on my input

layer I’ve specified that the images are 32 by 32. I could have just used the

128 by 128 if I wanted. I then partition the data

and shuffle the data, per good practice. If I look at the results, I can

see the 10 different outcomes, that is the 10 different

pieces of clothing or types of clothing that we’re

going to try to model. I navigate to the

second program where I’ve created my sparse

denoising autoencoder model. This is actually a convolutional

sparse denoising autoencoder. It’s convolutional because

we’re modeling images. And we’re using pooling layers

and convolutional layers and so forth. So we begin by specifying our

build model action where we’re going to create the model,

titled sparse denoise autoenc for autoencoder. The type is going to be a CNN. All this does here,

this build model, is it just builds a

model shell that we’re going to populate with layers. Next, we’ve got this

add layer action where we’re adding an input layer. So after the layer equals,

when we have these curly brace, we’re specifying the

options for our layer. And this layer is

an input layer. These are grayscale

images, so they only have a single color channel

with the 32 height of 32. And notice I’m

applying the dropout. This creates the denoising

autoencoder, just a fancy name to describe a

relatively easy idea. So we’re going to

dropout about 40% of the inputs every mini batch. We’re just going to randomly

select 40 of the inputs and remove them from

the modeling process. Next, we add our first

convolutional layer. And this convolutional layer is

going to start with 16 neurons. The next convolutional layer

is going to have eight neurons. And notice that in this

convolutional layer, we’re actually applying

batch normalization. And when you use batch

normalization in SAS, you’re actually

adding a second layer. It’s adding a second layer, but

they’re really the same layer. This first layer

here, this add layer, this is a convolutional layer. And we know that because

type equals combo with eight filters. Notice that the activation

function is identity. So this first ADDLAYER

statement here is just really

combining the inputs. We’re not applying a nonlinear

transformation as of yet. We then pass the information

from this layer at line 24 to this next layer

where we’re going to normalize the information

with batch normalization. And then apply our

nonlinear transformation. If you have any questions

about batch normalization, definitely check out

our other YouTube video. We then pass this information

to our center layer, our center encoding layer, where we only

have one convolutional filter. And then we expand the

information back out using transpose convolutional layers. That basically up-samples

the information until we arrive at our last

convolutional layer and then our final output layer. Now, to train an autoencoder

in SAS, in the dlTrain action, you just negate

specifying a target. So notice in this code here,

there is no TARGET statement. But if you’re

looking at the code, you may realize that I have

specified an L1 regularization and an L2 regularization. So in this example, we

are creating a sparse denoising convolutional

autoencoder. Sounds fancy, but it’s

really pretty simple. I’m going to train this

model by running this code. We’ll build the model

structure, then train the model, and print out the output below. So now that the

results have appeared, let’s go scroll

down to the bottom. And we have an iteration

plot showing our training performance, the

blue line that seems to be monotonically

improving and our validation and performance. So I hope you like

this video where we looked at how to train

autoencoders in SAS. And we actually looked

at several variants– a denoising autoencoder,

a sparse autoencoder. And we created, we

combined the two to create a sparse denoising

convolutional autoencoder. Thank you for watching. Check out the links below

for more information. Please leave your

comments and questions. And I’ll get back to you. And if you’d like to

see one of these videos, definitely subscribe

to our channel. Take care. [MUSIC PLAYING]

You gotten beyond SAS DATA STEP and PROC SUMMARY which is my area.