http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture6.pdf

sigmoid

• Saturated neurons “kill” the gradients
• Sigmoid outputs are not zero-centered
• exp() is a bit compute expensive

tanh

• Squashes numbers to range [-1,1]
• zero centered (nice)
• Saturated neurons “kill” the gradients

ReLU

• Does not saturate (in +region)
• Very computationally efficient
• Converges much faster than sigmoid/tanh in practice (e.g. 6x)
• Actually more biologically plausible than sigmoid

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture7.pdf

• Adam is a good default choice in most cases
• If you can afford to do full batch updates then try out L-BFGS (and don’t forget to disable all sources of noise)