如何使用keras的generator

当数据量非常大时，一次将所有的图片加载到内存中会导致训练失败，因而需要使用generator的方式加载部分的图片

https://medium.com/@fromtheast/implement-fit-generator-in-keras-61aa2786ce98

https://github.com/keras-team/keras/issues/7729

https://github.com/keras-team/keras/issues/1627

Implement fit_generator( ) in Keras

Here is an example of fit_generator():

model.fit_generator(generator(features, labels, batch_size), samples_per_epoch=50, nb_epoch=10)

Breaking it down:

generator(features, labels, batch_size): generates batches of samples indefinitely

sample_per_epoch: number of samples you want to train in each epoch

nb_epoch: number of epochs

As you can manually define sample_per_epoch and nb_epoch, you have to provide codes for generator. Here is an example:

Assume features is an array of data with shape (100,64,64,3) and labels is an array of data with shape (100,1). We use data from features and labelsto train our model.

defgenerator(features, labels, batch_size):

# Create empty arrays to contain batch of features and labels#

batch_features = np.zeros((batch_size, 64, 64, 3))

batch_labels = np.zeros((batch_size,1))

whileTrue:

for i in range(batch_size):

# choose random index in features

index= random.choice(len(features),1)

batch_features[i] = some_processing(features[index])

batch_labels[i] = labels[index]

yieldbatch_features, batch_labels

With the generator above, if we define batch_size = 10, that means it will randomly taking out 10 samples from features and labels to feed into each epoch until an epoch hits 50 sample limit. Then fit_generator() destroys the used data and move on repeating the same process in new epoch.

One great advantage about fit_generator() besides saving memory is user can integrate random augmentation inside the generator, so it will always provide model with new data to train on the fly.

For more information on fit_generator() arguments, refer to Keras website:

Sequential - Keras Documentation

Fits the model on data generated batch-by-batch by a Python generator. The generator is run in parallel to the model…keras.io

I hope you found the content is helpful. If so, please hit ❤ to share and I really appreciate any feedback. Until next time!