Tensorflow

keras中文文档：

https://keras.io/zh/losses/

keras英文文档：

https://tensorflow.google.cn/api_docs/python/tf/keras/layers/Conv2D

WDL知乎：(回顾Google经典CTR预估模型WDL)

https://zhuanlan.zhihu.com/p/100898327

LSTM: (“动手学深度学习” )

课程：https://zh.d2l.ai/chapter_recurrent-neural-networks/lstm.html
github: https://github.com/d2l-ai/d2l-zh
https://zhuanlan.zhihu.com/p/32085405
https://blog.csdn.net/hfutdog/article/details/96479716

Tensorflow编程

1. replica_device_setter 参数设备分配

如果有多个ps节点时，变量存储和更新该怎么分配呢？tf.train.replica_device_setter这个API给出了答案
worker分配和ps分配；Between-graph replication；

 if FLAGS.job_name == "ps":
        server.join()  # ps hosts only join
    elif FLAGS.job_name == "worker":
        with tf.device(tf.train.replica_device_setter(worker_device="/job:worker/task:%d" % FLAGS.task_index, cluster=cluster)):

https://zhuanlan.zhihu.com/p/90234576

2. MonitoredTrainingSession

变量的初始化、从已有checkpoint恢复训练、summary、log和checkpoint的保存等。

 with tf.train.MonitoredTrainingSession(master=server.target,
                                                   is_chief=(FLAGS.task_index == 0),
                                                   checkpoint_dir="./checkpoint_dir",
                                                   hooks=hooks) as mon_sess:
                while not mon_sess.should_stop():
                    # mon_sess.run handles AbortedError in case of preempted PS.
                    _, ls, step = mon_sess.run([train_op, loss, global_step])
                    if step % 100 == 0:
                        print("Train step %d, loss: %f" % (step, ls))

https://zhuanlan.zhihu.com/p/91608555

3. checkpoint

TensorFlow的checkpoint中变量的重命名
https://zhuanlan.zhihu.com/p/33153473
tf.contrib.framework.list_variables(checkpoint_dir)
var = tf.contrib.framework.load_variable(checkpoint_dir, var_name)
tf.contrib.framework.get_variables_to_restore()
tf.trainable_variables()
tensorflow/python/tools/inspect_checkpoint.py
中提到的tf.train.NewCheckpointReader类

• tf.train.NewCheckpointReader
  这种方法不需要model，只要有checkpoint文件就行。

reader=tf.train.NewCheckpointReader(checkpoint_path)
variable_map=reader.get_variable_to_shape_map()
for var_name in variable_map:
        print(reader.get_tensor(var_name))

4. reduce_sum axis 轴 塌缩

https://www.zhihu.com/question/51325408

5. 变量复用variable_scope get_variable()

https://blog.csdn.net/Jerr__y/article/details/70809528

6. dropout的作用: 防止过拟合

https://zhuanlan.zhihu.com/p/38200980

7. 激活函数

https://zhuanlan.zhihu.com/p/172254089

8. 交叉熵

https://zhuanlan.zhihu.com/p/63731947?group_id=1112146751385022464

9. tf.shape(x) PK x.get_shape().as_list()

https://blog.csdn.net/m0_37393514/article/details/82226754
tf.shape(x) 返回的是Tensor，因此要放在session.run()里
x.get_shape().as_list() 返回元祖，因此要as_list()

10. 反向传播

https://blog.csdn.net/u014313009/article/details/51039334

11. attention

https://zhuanlan.zhihu.com/p/47063917
https://zhuanlan.zhihu.com/p/47282410
https://blog.csdn.net/qq_43331398/article/details/103192522

12. transformer

https://zhuanlan.zhihu.com/p/44121378
https://arxiv.org/pdf/1706.03762.pdf
https://blog.csdn.net/longxinchen_ml/article/details/86533005

13. TRAINABLE_VARIABLES

1. TRAINABLE_VARIABLES is the collection of variables or training parameters which should be modified when minimizing the loss
   TRAINABLE_VARIABLES变量是需要在最小化Loss时进行数值调整的
2. UPDATE_OPS is a collection of ops (operations performed when the graph runs, like multiplication, ReLU, etc.), not variables. Specifically, this collection maintains a list of ops which need to run before each training step.
   其中会保存一些需要在训练操作之前完成的操作，可配合tf.control_dependencies函数使得update操作先执行，然后再训练

14. 生成对抗网络

https://zhuanlan.zhihu.com/p/33752313