ubuntu18.04 tensorflow 1.14运行程序的时候出现cuDNN failed to initialize

ubuntu18.04 tensorflow 1.14运行程序的时候出现cuDNN failed to initialize

今天再运行程序的时候出现了下面的错误：

2020-02-12 13:06:06.483007: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.2020-02-12 13:06:06.567509: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.02020-02-12 13:06:06.587495: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED2020-02-12 13:06:06.589641: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED2020-02-12 13:06:06.592919: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED2020-02-12 13:06:06.594236: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED2020-02-12 13:06:06.596050: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED2020-02-12 13:06:06.597353: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED2020-02-12 13:06:06.608829: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.72020-02-12 13:06:06.611414: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR2020-02-12 13:06:06.612753: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERRORTraceback (most recent call last): File "train_mspeech.py", line 47, in ms.TrainModel(datapath, epoch = 50, batch_size = 8, save_step = 500) File "/home/eric/Documents/speech_recognition/ASRT_SpeechRecognition/SpeechModel251.py", line 179, in TrainModel self._model.fit_generator(yielddatas, save_step) File "/home/eric/anaconda3/lib/python3.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper return func(*args, **kwargs) File "/home/eric/anaconda3/lib/python3.7/site-packages/keras/engine/training.py", line 1418, in fit_generator initial_epoch=initial_epoch) File "/home/eric/anaconda3/lib/python3.7/site-packages/keras/engine/training_generator.py", line 217, in fit_generator class_weight=class_weight) File "/home/eric/anaconda3/lib/python3.7/site-packages/keras/engine/training.py", line 1217, in train_on_batch outputs = self.train_function(ins) File "/home/eric/anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__ return self._call(inputs) File "/home/eric/anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call fetched = self._callable_fn(*array_vals) File "/home/eric/anaconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1458, in __call__ run_metadata_ptr)tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found. (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node conv2d_1/convolution}}]] [[ctc/ToInt32_2/_299]] (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node conv2d_1/convolution}}]]0 successful operations.0 derived errors ignored.

解决方法

注释掉下面的代码或者删除就行了，我的tensorflow-gpu是1.14，cuda10.0，cudnn7.0

config = tf.ConfigProto()config.gpu_options.per_process_gpu_memory_fraction = 0.95#config.gpu_options.allow_growth=True #不全部占满显存, 按需分配set_session(tf.Session(config=config))

参考文献

[1]. Error : Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.​​https://github.com/tensorflow/tensorflow/issues/24828​​

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。