Hey, I’m Robert. And I teach deep
learning here at SAS. Today, I’m going to talk to you
about an unsupervised learning method called autoencoders. Now, there are several different
variants of autoencoders. There’s denoising autoencoders
and sparse autoencoders. But you may be wondering
what is an autoencoder. Well, an autoencoder
is a neural network where you have inputs
on your input layer, and you have inputs
on your output layer. Your mathematical neurons that
are in your neural network are referred to as
encoders or decoders. And all these are are just
probabilistic representations of the input space. So each of your
mathematical neurons is going to take a look
at that input space and try to reformulate it
again given the input space. That’s why inputs are on
your input layer and inputs on your output layer. The idea, or our hope,
when we use an autoencoder is to desensitize the irrelevant
and sensitize the relevant. So I had mentioned that
there’s two variants that we’re going to look at, a
denoising autoencoder and a sparse autoencoder. And these are
complicated names that really describe simple ideas. A sparse autoencoder
is an autoencoder where we’re going to constrain
the objective function using an L1 or L2 regularization. That’s all it is. For a denoising
autoencoder, we’re simply applying dropout to
the input layer of our model. Let’s take a look at this. So here, I’m using SAS Studio. And the first thing
that I’m going to do is I’m going to
upload my images. Then I’m going to
view my images. And I’m going to view a
subset of each of the classes. The data that I’m
using in this example is the mnist_fashion data, where
I have 10 different outcomes. Next, I’m going to resize
the images to be a 32 by 32, because on my input
layer I’ve specified that the images are 32 by 32. I could have just used the
128 by 128 if I wanted. I then partition the data
and shuffle the data, per good practice. If I look at the results, I can
see the 10 different outcomes, that is the 10 different
pieces of clothing or types of clothing that we’re
going to try to model. I navigate to the
second program where I’ve created my sparse
denoising autoencoder model. This is actually a convolutional
sparse denoising autoencoder. It’s convolutional because
we’re modeling images. And we’re using pooling layers
and convolutional layers and so forth. So we begin by specifying our
build model action where we’re going to create the model,
titled sparse denoise autoenc for autoencoder. The type is going to be a CNN. All this does here,
this build model, is it just builds a
model shell that we’re going to populate with layers. Next, we’ve got this
add layer action where we’re adding an input layer. So after the layer equals,
when we have these curly brace, we’re specifying the
options for our layer. And this layer is
an input layer. These are grayscale
images, so they only have a single color channel
with the 32 height of 32. And notice I’m
applying the dropout. This creates the denoising
autoencoder, just a fancy name to describe a
relatively easy idea. So we’re going to
dropout about 40% of the inputs every mini batch. We’re just going to randomly
select 40 of the inputs and remove them from
the modeling process. Next, we add our first
convolutional layer. And this convolutional layer is
going to start with 16 neurons. The next convolutional layer
is going to have eight neurons. And notice that in this
convolutional layer, we’re actually applying
batch normalization. And when you use batch
normalization in SAS, you’re actually
adding a second layer. It’s adding a second layer, but
they’re really the same layer. This first layer
here, this add layer, this is a convolutional layer. And we know that because
type equals combo with eight filters. Notice that the activation
function is identity. So this first ADDLAYER
statement here is just really
combining the inputs. We’re not applying a nonlinear
transformation as of yet. We then pass the information
from this layer at line 24 to this next layer
where we’re going to normalize the information
with batch normalization. And then apply our
nonlinear transformation. If you have any questions
about batch normalization, definitely check out
our other YouTube video. We then pass this information
to our center layer, our center encoding layer, where we only
have one convolutional filter. And then we expand the
information back out using transpose convolutional layers. That basically up-samples
the information until we arrive at our last
convolutional layer and then our final output layer. Now, to train an autoencoder
in SAS, in the dlTrain action, you just negate
specifying a target. So notice in this code here,
there is no TARGET statement. But if you’re
looking at the code, you may realize that I have
specified an L1 regularization and an L2 regularization. So in this example, we
are creating a sparse denoising convolutional
autoencoder. Sounds fancy, but it’s
really pretty simple. I’m going to train this
model by running this code. We’ll build the model
structure, then train the model, and print out the output below. So now that the
results have appeared, let’s go scroll
down to the bottom. And we have an iteration
plot showing our training performance, the
blue line that seems to be monotonically
improving and our validation and performance. So I hope you like
this video where we looked at how to train
autoencoders in SAS. And we actually looked
at several variants– a denoising autoencoder,
a sparse autoencoder. And we created, we
combined the two to create a sparse denoising
convolutional autoencoder. Thank you for watching. Check out the links below
for more information. Please leave your
comments and questions. And I’ll get back to you. And if you’d like to
see one of these videos, definitely subscribe
to our channel. Take care. [MUSIC PLAYING]