Tjyylli

I was going through BERT paper which uses GELU (Gaussian Error Linear Unit) which states equation as
$$ GELU(x) = xP(X ≤ x) = xΦ(x).$$ which appriximates to $$0.5x(1 + tanh[sqrt{
2/π}(x + 0.044715x^3)])$$
Could you simplify the equation and explain how it has been approimated.

For these type of numerical approximations, the key idea is to find a similar function (based on experience), parameterize it, and then fit it to a set of points from the original function.

Lets expand the cumulative distribution $Phi(x)$:

$text{GELU}(x):=xPhi(x)=0.5xleft(1+text{erf}(frac{x}{sqrt{2}})right)$

Note that this is a definition, not an equation (a relation).

Knowing that $text{erf}(x)$ is very close to $text{tanh}(x)$

and first derivatives of $text{erf}(frac{x}{sqrt{2}})$ and $text{tanh}(sqrt{frac{2}{pi}}x)$ coincide at $x=0$, which is $sqrt{frac{2}{pi}}$, we proceed to fit
$$text{tanh}left(sqrt{frac{2}{pi}}(x+ax^2+bx^3+cx^4+dx^5)right)$$ (or more terms) to some points $left(x_i, text{erf}(frac{x_i}{sqrt{2}})right)$. I have fitted this function to 20 samples between $(-1.5, 1.5)$ (using this site), and here are the coefficients:

By setting $a=c=d=0$, $b$ was estimated to be $0.04495641$. With more samples from a wider range (that site only allowed 20), coefficient $b$ will be closer to paper's $0.044715$. Finally we get:

$text{GELU}(x)=xPhi(x)=0.5xleft(1+text{erf}(frac{x}{sqrt{2}})right)=0.5xleft(1+text{tanh}left(sqrt{frac{2}{pi}}(x+0.044715x^3)right)right)$

Note that if we did not utilize the relation between the first derivatives, term $sqrt{frac{2}{pi}}$ would have been included in the parameters as follows
$$0.5xleft(1+text{tanh}left(0.797885x+0.035677x^3right)right)$$
which is less beautiful (less analytical, more numerical)!

A similar relation holds between $text{erf}(x)$ and $2left(sigma(x)-frac{1}{2}right)$ (sigmoid), which is proposed in the paper as another approximation.

First note that $$Phi(x) = frac12 mathrm{erfc}left(-frac{x}{sqrt{2}}right) = frac12 left(1 + mathrm{erf}left(frac{x}{sqrt2}right)right)$$ by parity of $mathrm{erf}$. We need to show that $$mathrm{erf}left(frac x {sqrt2}right) approx tanhleft(sqrt{frac2pi} left(x + a x^3right)right)$$ for $a approx 0.044715$.

For large values of $x$, both functions are bounded in $[-1, 1]$. For small $x$, the respective Taylor series read $$tanh(x) = x - frac{x^3}{3} + o(x^3)$$ and $$mathrm{erf}(x) = frac{2}{sqrt{pi}} left(x - frac{x^3}{3}right) + o(x^3).$$
Substituting, we get that $$
tanhleft(sqrt{frac2pi} left(x + a x^3right)right) = sqrtfrac{2}{pi} left(x + left(a-frac{2}{3pi}right)x^3right) + o(x^3)
$$
and
$$
mathrm{erf}left(frac x {sqrt2}right) = sqrtfrac2pi left(x - frac{x^3}{6}right) + o(x^3).
$$
Equating coefficient for $x^3$, we find
$$
a approx 0.04553992412
$$
close to the paper's $0.044715$.