TensorFlow知识点

1. 使用指定的GPU和显存

如果设备上装备了多块GPU，TF运行时默认使用所有与他可见GPU，而且默认使用尽可能多的显存。可是，很多情况下我们的程序实际上并不需要消耗那么多资源，TF的这种独占性的处理方式就造成了资源浪费。那么如何限制TF 可使用的GPU数目和显存容量呢？

1.1 设置TF可使用GPU设备

GPU设备在TF下是从零开始编号的，如果设备上有四块GPU设备，他们的编号依次是 0，1，2，3. 设备名称依次是 "/gpu:0","/gpu:1","/gpu:2","/gpu:3"。默认所有设备是TF运行时可见的，而TF默认也会占用所有与他可见的GPU设备。我们可以通过环境变量
CUDA_VISIBLE_DEVICES 来设置哪些GPU对TF可见，具体语法如下：

Environment Variable Syntax	Results
CUDA_VISIBLE_DEVICES=1	Only device 1 will be seen
CUDA_VISIBLE_DEVICES=0,1	Devices 0 and 1 will be visible
CUDA_VISIBLE_DEVICES="0,1"	Same as above, quotation marks are optional
CUDA_VISIBLE_DEVICES=0,2,3	Devices 0, 2, 3 will be visible; device 1 is masked

为保证设备使用上的灵活性， CUDA_VISIBLE_DEVICES 环境变量应为临时变量 ，默认情况下该临时变量不存在或未设置。终端中配置使用方法如下：

## Windows 下设置使用方法
# set  CUDA_DEVICE_ORDER=PCI_BUS_ID     # 设置按PCI_BUS_ID 顺序索引设备
set CUDA_VISIBLE_DEVICES=1     # 设置/gpu:1或1号GPU对后续CUDA程序可见
python mywork.py

## Linux 下设置使用方法
CUDA_VISIBLE_DEVICES=1 python mywork.py

CUDA_VISIBLE_DEVICES临时变量的设置还可放置在Python代码中设置，这样既能保证跨平台的统一性又免去了手动设置临时变量的麻烦。

import os
# os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"     # 见 Tensorflow issue #152
os.environ["CUDA_VISIBLE_DEVICES"] = "1"

通过这种方式设置的临时变量值会覆盖在终端中手动设置的值。
注1： 通过设置临时变量 CUDA_VISIBLE_DEVICES 来设置对TF可见的GPU设备并非TF的专属操作，临时变量CUDA_VISIBLE_DEVICES 是CUDA用来限制CUDA Application 可见GPU设备的手段，参见CUDA_VISIBLE_DEVICES 环境变量说明。
注2： CUDA_VISIBLE_DEVICES 设置的运行时可见设备序号可能与NVML 工具nvidia-smi输出的不一致，参见CUDA_DEVICE_ORDER 环境变量说明

1.2 设置TF可使用GPU内存容量

默认情况下，TF将占用尽可能多的可见GPU内存，用户可在启动TF Session时设置GPU参数来加以限制。典型设置分为定量使用和按需使用。

1.2.1 GPU内存定量使用

这种设置方法使得TF最多只能使用指定比例的可见显存。

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.7)
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))  

Tips： 可见显存 = 可见GPU显存总和
通过上述设置，TF最多只能使用70%的可见显存。

1.2.2 GPU内存按需使用

通过设置allow_growth GPU参数，TF可按需使用可见显存。

gpu_options = tf.GPUOptions(allow_growth=True)
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) 

2. Variables

A TensorFlow variable is the best way to represent shared, persistent state manipulated by your program. Variables are manipulated via the tf.Variable class.
Features:

1. A variable is a tensor whose value can be changed by running ops on it.
2. Modifications on a variable are visible across multiple tf.Sessions.
3. A variable exists outside the context of a single session.run call.

The best way to create a variable is to call the tf.get_variable function.
Usage： name = tf.get_variable(str_name,shape=[1], dtype=tf.float32, initializer=tf.glorot_uniform_initializer, collections, trainable)
params:

--name: 变量名，用于上下文中引用该变量
--str_name: 变量名, to name this variable's value when checkpointing and exporting models.
--shape: 可遍历值（list,tuple etc.）, 变量的维度和每一维的长度，类似numpy 数组。默认1维，即标量。
--dtype：变量数据类型，默认tf.float32
--initializer: 调用tf.global_variables_initializer()或session.run(name.initializer)时变量初始方式，默认tf.glorot_uniform_initializer。
--collections: 变量所在集合
--trainable: 变量是否可学习，将被放在tf.GraphKeys.TRAINABLE_VARIABLES集合if True
注：为保证变量使用上的一致性，name 和 str_name 常设成一样。

#### Variable 使用示例
## create a variable in tf.GraphKeys.LOCAL_VARIABLES collection, in which variables are not trainable
my_local = tf.get_variable("my_local", shape=(), 
collections=[tf.GraphKeys.LOCAL_VARIABLES])

## create a variable which is not trainable
my_non_trainable = tf.get_variable("my_non_trainable", 
                                   shape=(), 
                                   trainable=False)

## add an existing variable named my_local to a collection named my_collection_name
tf.add_to_collection("my_collection_name", my_local)

## retrieve a list of all the variables (or other objects)  in collection  named my_collection_name
tf.get_collection("my_collection_name")

## creates a variable named v and places it on the second GPU device
with tf.device("/device:GPU:1"):
    v = tf.get_variable("v", [1])

## initialize all variables
session.run(tf.global_variables_initializer())

## initialize variable my_variable only
session.run(my_variable.initializer)

## prints the names of all variables which have not yet been initialized
print(session.run(tf.report_uninitialized_variables()))

## initialize w with v's value
v = tf.get_variable("v", shape=(), initializer=tf.zeros_initializer())
w = tf.get_variable("w", initializer=v.initialized_value() + 1)

## set reuse=True to reuse variables
with tf.variable_scope("model"):
  output1 = my_image_filter(input1)
with tf.variable_scope("model", reuse=True):
  output2 = my_image_filter(input2)

## call scope.reuse_variables() to trigger a variable reuse
with tf.variable_scope("model") as scope:
  output1 = my_image_filter(input1)
  scope.reuse_variables()
  output2 = my_image_filter(input2)

## initialize a variable scope based on another one
## set reuse=true to share variables
with tf.variable_scope("model") as scope:
  output1 = my_image_filter(input1)
with tf.variable_scope(scope, reuse=True):
  output2 = my_image_filter(input2)

1. Any string is a valid collection name, and there is no need to explicitly create a collection.
2. Note that by default tf.global_variables_initializer does not specify the order in which variables are initialized. Therefore, if the initial value of a variable depends on another variable's value, it's likely that you'll get an error. Any time you use the value of a variable in a context in which not all variables are initialized, it is best to use variable.initialized_value() instead of variable.
3. Variable scopes allow you to control variable reuse when calling functions which implicitly create and use variables. They also allow you to name your variables in a hierarchical and understandable way.
4. Since depending on exact string names of scopes can feel dangerous, it's also possible to initialize a variable scope based on another one

Goto Tensorflow Docs -- Variables

3. Tensorboard multiple scalar summaries in one plot

Tensorboard 一张图中画两个变量，方便对比，如：

对比图

import tensorflow as tf
from numpy import random

writer_1 = tf.summary.FileWriter("./logs/plot_1")
writer_2 = tf.summary.FileWriter("./logs/plot_2")

log_var = tf.Variable(0.0)
tf.summary.scalar("loss", log_var)

write_op = tf.summary.merge_all()

session = tf.InteractiveSession()
session.run(tf.global_variables_initializer())

for i in range(100):
    # for writer 1
    summary = session.run(write_op, {log_var: random.rand()})
    writer_1.add_summary(summary, i)
    writer_1.flush()

    # for writer 2
    summary = session.run(write_op, {log_var: random.rand()})
    writer_2.add_summary(summary, i)
    writer_2.flush()