A simple warm-up

Jan 6, 2017 • neural networks • 0

Problem

The logistic sigmoid function is one of the more commonly used activation functions for neural networks. The function is given by

\begin{align} \sigma(x) = \frac{1}{1+ e^{-x}} \end{align}

One of the nice properties of this funciton is that its derivative can be written in terms of the function value itself.

Show that $\frac{d}{dx} \sigma(x) = \sigma(x) (1 - \sigma(x))$ .

Solution

show

The derivative of $\sigma(x)$ can be written as

\begin{align} \frac{d}{dx} \sigma(x) &= \frac{d}{dx} \frac{1}{1+ e^{-x}} \\ &= \frac{e^{-x}}{(1 + e^{-x})^{2}} \\ &= \frac{e^{-x} + 1 - 1}{(1+e^{-x})^2} \\ &= \frac{1}{1+e^{-x}} \left(\frac{1+e^{-x}}{1+e^{-x}} - \frac{1}{1+e^{-x}} \right)\\ &= \sigma(x) (1 - \sigma(x)) \end{align}