Traceback (most recent call last):
File "task2_cakir.py", line 626, in <module> | 0/200 [00:00<?, ?it/s]
sys.exit(main(sys.argv))
File "task2_cakir.py", line 560, in main
app.system_training()
File "/home/fanyiyi_env/DCASE2017-baseline-system-master-gpu/dcase_framework/decorators.py", line 38, in function_wrapper
to_return = func(*args, **kwargs)
File "/home/fanyiyi_env/DCASE2017-baseline-system-master-gpu/dcase_framework/application_core.py", line 3222, in system_training
validation_files=validation_files
File "/home/fanyiyi_env/DCASE2017-baseline-system-master-gpu/dcase_framework/learners.py", line 2890, in learn
class_weight=class_weight
File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
return func(*args, **kwargs)
File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/engine/training.py", line 2147, in fit_generator
class_weight=class_weight)
File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/engine/training.py", line 1839, in train_on_batch
outputs = self.train_function(ins)
File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 1224, in __call__
return self.function(*inputs)
File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py", line 917, in __call__
storage_map=getattr(self.fn, 'storage_map', None))
File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/theano/gof/link.py", line 325, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py", line 903, in __call__
self.fn() if output_subset is None else\
File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/theano/scan_module/scan_op.py", line 963, in rval
r = p(n, [x[0] for x in i], o)
File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/theano/scan_module/scan_op.py", line 952, in p
self, node)
File "theano/scan_module/scan_perform.pyx", line 259, in theano.scan_module.scan_perform.perform (/home/fanyiyi_env/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.2.1511-Core-x86_64-2.7.15-64/scan_perform/mod.cpp:3136)
File "pygpu/gpuarray.pyx", line 1777, in pygpu.gpuarray.GpuArray.copy
File "pygpu/gpuarray.pyx", line 718, in pygpu.gpuarray.pygpu_copy
File "pygpu/gpuarray.pyx", line 406, in pygpu.gpuarray.array_copy
pygpu.gpuarray.GpuArrayException: cuMemAlloc: CUDA_ERROR_OUT_OF_MEMORY: out of memory
Apply node that caused the error: for{gpu,scan_fn}(Subtensor{int64}.0, GpuSubtensor{:int64:}.0, GpuIncSubtensor{Set;:int64:}.0, GpuIncSubtensor{Set;:int64:}.0, GpuIncSubtensor{Set;:int64:}.0, GpuIncSubtensor{Set;:int64:}.0, GpuIncSubtensor{Set;:int64:}.0, GpuIncSubtensor{Set;:int64:}.0, Subtensor{int64}.0, conv2d_1/kernel, conv2d_1/bias, batch_normalization_1/gamma, batch_normalization_1/beta, batch_normalization_1/moving_mean, batch_normalization_1/moving_variance, conv2d_2/kernel, conv2d_2/bias, batch_normalization_2/gamma, batch_normalization_2/beta, batch_normalization_2/moving_mean, batch_normalization_2/moving_variance, conv2d_3/kernel, conv2d_3/bias, batch_normalization_3/gamma, batch_normalization_3/beta, batch_normalization_3/moving_mean, batch_normalization_3/moving_variance, gru_1/kernel, gru_1/bias, gru_1/recurrent_kernel, GpuFromHost<None>.0, GpuFromHost<None>.0)
Toposort index: 363
Inputs types: [TensorType(int64, scalar), GpuArrayType<None>(float32, 3D), GpuArrayType<None>(float32, 3D), GpuArrayType<None>(int32, 3D), GpuArrayType<None>(int32, 3D), GpuArrayType<None>(int32, 3D), GpuArrayType<None>(int32, 3D), GpuArrayType<None>(int32, 3D), TensorType(int64, scalar), GpuArrayType<None>(float32, 4D), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, 4D), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, 4D), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, matrix), GpuArrayType<None>(float32, vector), GpuArrayType<None>(float32, matrix), GpuArrayType<None>(uint8, scalar), GpuArrayType<None>(float32, 4D)]
Inputs shapes: [(), (1501, 32, 96), (1502, 32, 96), (1502, 15360, 6), (1502, 15360, 6), (1502, 15360, 6), (1502, 15360, 6), (1502, 15360, 6), (), (5, 5, 1, 96), (96,), (96,), (96,), (96,), (96,), (5, 5, 96, 96), (96,), (96,), (96,), (96,), (96,), (5, 5, 96, 96), (96,), (96,), (96,), (96,), (96,), (96, 288), (288,), (96, 288), (), (32, 1, 1501, 40)]
Inputs strides: [(), (4, 576384, 6004), (12288, 384, 4), (368640, 24, 4), (368640, 24, 4), (368640, 24, 4), (368640, 24, 4), (368640, 24, 4), (), (1920, 384, 384, 4), (4,), (4,), (4,), (4,), (4,), (184320, 36864, 384, 4), (4,), (4,), (4,), (4,), (4,), (184320, 36864, 384, 4), (4,), (4,), (4,), (4,), (4,), (1152, 4), (4,), (1152, 4), (), (240160, 240160, 160, 4)]
Inputs values: [array(1501), 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', array(1501), 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', gpuarray.array(1, dtype=uint8), 'not shown']
Outputs clients: [[Shape(for{gpu,scan_fn}.0), GpuSubtensor{::int64}(for{gpu,scan_fn}.0, Constant{-1}), GpuSubtensor{:int64:}(for{gpu,scan_fn}.0, Constant{-1})], [Shape(for{gpu,scan_fn}.1), GpuSubtensor{::int64}(for{gpu,scan_fn}.1, Constant{-1}), GpuSubtensor{:int64:}(for{gpu,scan_fn}.1, Constant{-1})], [Shape(for{gpu,scan_fn}.2), GpuSubtensor{::int64}(for{gpu,scan_fn}.2, Constant{-1}), GpuSubtensor{:int64:}(for{gpu,scan_fn}.2, Constant{-1})], [Shape(for{gpu,scan_fn}.3), GpuSubtensor{::int64}(for{gpu,scan_fn}.3, Constant{-1}), GpuSubtensor{:int64:}(for{gpu,scan_fn}.3, Constant{-1})], [Shape(for{gpu,scan_fn}.4), GpuSubtensor{::int64}(for{gpu,scan_fn}.4, Constant{-1}), GpuSubtensor{:int64:}(for{gpu,scan_fn}.4, Constant{-1})], [Shape(for{gpu,scan_fn}.5), GpuSubtensor{::int64}(for{gpu,scan_fn}.5, Constant{-1}), GpuSubtensor{:int64:}(for{gpu,scan_fn}.5, Constant{-1})], [InplaceGpuDimShuffle{0,1,2}(for{gpu,scan_fn}.6), Shape(for{gpu,scan_fn}.6), GpuSubtensor{::int64}(for{gpu,scan_fn}.6, Constant{-1})]]
Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):
File "/home/fanyiyi_env/DCASE2017-baseline-system-master-gpu/dcase_framework/application_core.py", line 3222, in system_training
validation_files=validation_files
File "/home/fanyiyi_env/DCASE2017-baseline-system-master-gpu/dcase_framework/learners.py", line 2718, in learn
self.create_model(input_shape=input_shape)
File "task2_cakir.py", line 252, in create_model
dropout=rnn_dropout_W, recurrent_dropout=rnn_dropout_U, implementation=2)(model)
File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/layers/recurrent.py", line 482, in __call__
return super(RNN, self).__call__(inputs, **kwargs)
File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 603, in __call__
output = self.call(inputs, **kwargs)
File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/layers/recurrent.py", line 1515, in call
initial_state=initial_state)
File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/layers/recurrent.py", line 589, in call
input_length=timesteps)
File "/home/fanyiyi_env/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 1424, in rnn
go_backwards=go_backwards)
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
Exception KeyError: KeyError(<weakref at 0x7f3c8b47ae68; to 'tqdm' at 0x7f3c6c938d90>,) in <bound method tqdm.__del__ of Learn: 0%| | 0/200 [00:00<?, ?it/s]> ignored
解决方法：将keras2.1.2升级为2.2.0即可解决
后续：
之后，增加了各层中的隐藏单元个数，由原来的96——>160,结果又出现了上面的报错，将隐藏单元的数值降低，以后，报错消失。
网友评论