参考https://blog.csdn.net/kangyi411/article/details/78969642

Sigmoid

\sigma(x) = \frac {1} {1 + e^{-x}}
求导为\sigma(x)(1 - \sigma(x))

tanh

\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}
求导为1 - \tanh^2(x)

relu

f(x) = max\{0, x\}