tf
主要总结整理TF2的使用过程总的基础的的知识点
图片数据的切片，混合，打包
测试数据集为：
import tensorflow as tf
import numpy as np
np.random.seed(10)#固定每次的随机数
features, labels = (np.random.sample((4, 2, 2)), # 模拟6组数据，每组数据3个特征
np.random.sample((4, 1))) # 模拟6组数据，每组数据对应一个标签，注意两者的维数必须匹配
print((features)) # 打印feature数据
print((labels)) # 打印标签数据
得到测试用的数据集为：
feture:
[[[0.77132064 0.02075195]
[0.63364823 0.74880388]]
[[0.49850701 0.22479665]
[0.19806286 0.76053071]]
[[0.16911084 0.08833981]
[0.68535982 0.95339335]]
[[0.00394827 0.51219226]
[0.81262096 0.61252607]]]
lables:
[[0.72175532]
[0.29187607]
[0.91777412]
[0.71457578]]
我们用features表示4张大小为[2,2]的图片数据，labels表示4张图片的标签数据，例如属于哪一类图片
from_tensor_slices
执行以下操作对数据集进行切片操作
data = tf.data.Dataset.from_tensor_slices((features, labels))
输出的结果是包含每一张图片的数据信息和特征信息的节点，4张图片对应的是4个Tensor节点
----------from_tensor_slices--------------
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=array([[0.77132064, 0.02075195],
[0.63364823, 0.74880388]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.72175532])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.49850701, 0.22479665],
[0.19806286, 0.76053071]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.29187607])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.16911084, 0.08833981],
[0.68535982, 0.95339335]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.91777412])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.00394827, 0.51219226],
[0.81262096, 0.61252607]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.71457578])>)
shuffle
对from_tensor_slices处理的数据，进行混合，混合就是打乱原数组之间的顺序，数组的数据大小和内容并没有改变;
混合的数据越大，混合程度越高
shuffle_data =data.shuffle(4)#4表示每次混合的buffer size，因为我们只有四个数据，直接混合所有数据
混合后的结果为：
可以跟from_tensor_slices数据进行比较下，原来的数据顺序为[0,1,2,3],混合后为[3,2,1,0]
----------shuffle--------------
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.00394827, 0.51219226],
[0.81262096, 0.61252607]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.71457578])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.169121084, 0.08833981],
[0.68535982, 0.95339335]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.91777412])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.49850701, 0.22479665],
[0.19806286, 0.76053071]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.29187607])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.77132064, 0.02075195],
[0.63364823, 0.74880388]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.72175532])>)
batch##
batch_data =data.batch(1)
对shuffle处理后的数据进行打包，如果为1，则数据内容和格式跟shuffle的数据相同，相当于没有处理
----------batch(1)--------------
(<tf.Tensor: shape=(1, 2, 2), dtype=float64, numpy=
array([[[0.77132064, 0.02075195],
[0.63364823, 0.74880388]]])>, <tf.Tensor: shape=(1, 1), dtype=float64, numpy=array([[0.72175532]])>)
(<tf.Tensor: shape=(1, 2, 2), dtype=float64, numpy=
array([[[0.00394827, 0.51219226],
[0.81262096, 0.61252607]]])>, <tf.Tensor: shape=(1, 1), dtype=float64, numpy=array([[0.71457578]])>)
(<tf.Tensor: shape=(1, 2, 2), dtype=float64, numpy=
array([[[0.16911084, 0.08833981],
[0.68535982, 0.95339335]]])>, <tf.Tensor: shape=(1, 1), dtype=float64, numpy=array([[0.91777412]])>)
(<tf.Tensor: shape=(1, 2, 2), dtype=float64, numpy=
array([[[0.49850701, 0.22479665],
[0.19806286, 0.76053071]]])>, <tf.Tensor: shape=(1, 1), dtype=float64, numpy=array([[0.29187607]])>)
如果batch(2)，则每个节点信息中包含两张图片数据，以此类推
(<tf.Tensor: shape=(2, 2, 2), dtype=float64, numpy=
array([[[0.00394827, 0.51219226],
[0.81262096, 0.61252607]],
[[0.77132064, 0.02075195],
[0.63364823, 0.74880388]]])>, <tf.Tensor: shape=(2, 1), dtype=float64, numpy=
array([[0.71457578],
[0.72175532]])>)
(<tf.Tensor: shape=(2, 2, 2), dtype=float64, numpy=
array([[[0.49850701, 0.22479665],
[0.19806286, 0.76053071]],
[[0.16911084, 0.08833981],
[0.68535982, 0.95339335]]])>, <tf.Tensor: shape=(2, 1), dtype=float64, numpy=
array([[0.29187607],
[0.91777412]])>)
总结一下：
- from_tensor_slices对原图片数据进行切片，有多上张图片就切成多少个数据
- shuffle,对切好后的数据进行混合，交换下顺序
- batch对数据进行打包，便于批量化处理；batch后的节点数为（数据集大小/batch大小），两个数字可能不是整除，会导致一个batch大小可能小于等于batch size
完整代码：shuffle_and_batch.py
import tensorflow as tf
import numpy as np
from tensorflow.keras import datasets
np.random.seed(10)#固定每次的随机数
features, labels = (np.random.sample((4, 2, 2)), # 模拟6组数据，每组数据3个特征
np.random.sample((4, 1))) # 模拟6组数据，每组数据对应一个标签，注意两者的维数必须匹配
print((features)) # 打印feature数据
print((labels)) # 打印标签数据
print('----------from_tensor_slices--------------')
for element in features:
print(element)
#切片转换，将数据转化成tesor节点数据
data = tf.data.Dataset.from_tensor_slices((features, labels))
for element in data:
print(element)
print('----------shuffle--------------')
#shuffle_data = tf.data.Dataset.from_tensor_slices((features,labels)).shuffle(1000).batch(128)
shuffle_data =data.shuffle(4)
for element in shuffle_data:
print(element)
print('----------batch--------------')
batch_data =shuffle_data.batch(2)
for element in batch_data:
print(element)
batch_data.repeat(2)
output:
runfile('G:/GitHub/tensorflow/Spyder/TF2/shuffle_and_batch.py', wdir='G:/GitHub/tensorflow/Spyder/TF2')
[[[0.77132064 0.02075195]
[0.63364823 0.74880388]]
[[0.49850701 0.22479665]
[0.19806286 0.76053071]]
[[0.16911084 0.08833981]
[0.68535982 0.95339335]]
[[0.00394827 0.51219226]
[0.81262096 0.61252607]]]
[[0.72175532]
[0.29187607]
[0.91777412]
[0.71457578]]
----------from_tensor_slices--------------
[[0.77132064 0.02075195]
[0.63364823 0.74880388]]
[[0.49850701 0.22479665]
[0.19806286 0.76053071]]
[[0.16911084 0.08833981]
[0.68535982 0.95339335]]
[[0.00394827 0.51219226]
[0.81262096 0.61252607]]
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.77132064, 0.02075195],
[0.63364823, 0.74880388]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.72175532])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.49850701, 0.22479665],
[0.19806286, 0.76053071]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.29187607])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.16911084, 0.08833981],
[0.68535982, 0.95339335]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.91777412])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.00394827, 0.51219226],
[0.81262096, 0.61252607]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.71457578])>)
----------shuffle--------------
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.49850701, 0.22479665],
[0.19806286, 0.76053071]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.29187607])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.16911084, 0.08833981],
[0.68535982, 0.95339335]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.91777412])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.77132064, 0.02075195],
[0.63364823, 0.74880388]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.72175532])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.00394827, 0.51219226],
[0.81262096, 0.61252607]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.71457578])>)
----------batch--------------
(<tf.Tensor: shape=(2, 2, 2), dtype=float64, numpy=
array([[[0.77132064, 0.02075195],
[0.63364823, 0.74880388]],
[[0.16911084, 0.08833981],
[0.68535982, 0.95339335]]])>, <tf.Tensor: shape=(2, 1), dtype=float64, numpy=
array([[0.72175532],
[0.91777412]])>)
(<tf.Tensor: shape=(2, 2, 2), dtype=float64, numpy=
array([[[0.00394827, 0.51219226],
[0.81262096, 0.61252607]],
[[0.49850701, 0.22479665],
[0.19806286, 0.76053071]]])>, <tf.Tensor: shape=(2, 1), dtype=float64, numpy=
array([[0.71457578],
[0.29187607]])>)
网友评论