GELU activation

GELUs full form is GAUSSIAN ERROR LINEAR UNIT

Activations like ReLU, ELU and PReLU have enabled faster and better convergence of Neural Networks than sigmoids.

Also, Dropout regularizes the model by randomly multiplying a few activations by 0.

Both of the above methods together decide a neuron’s output. Yet, the two work independently from each other. GELU aims to combine them.

Also, a new RNN regularizer called Zoneout stochastically multiplies the input by 1.

We want to merge all 3 functionalities by stochastically multiplying the input by 0 or 1 and getting the output value (of the activation function) deterministically.

We chose this distribution since neuron’s input follow a normal distribution, especially after Batch Normalization.

But the output of any activation function should be deterministic, not stochastic. So, we find the expected value of our transformation.

Since Φ(x) is a cumulative distribution of Gaussian distribution and is often computed with the error function, hence we define Gaussian Error Linear Unit (GELU) as-

References-

Related posts:

Algunos afirman que la corrupción es un fenómeno social, unos consideran que es el síntoma y otros la reconocen como la enfermedad misma.

What a lot of people don’t know is that Dementia is not a disease, but rather a syndrome. It’s basically a general term used to describe different disorders that affect the brain. Dementia occurs…

There are plans for a new skyscraper in Seattle. Yes, a new skyscraper. And this may be the tallest one in the city yet! People are planning for a bright future ahead and they are looking forward to…