[tensorflow gpu 설정 에러 원인과 해결] "Memory growth cannot differ between GPU devices"

에러 문구 

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 271, in constant
    return _constant_impl(value, dtype, shape, name, verify_shape=False,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 283, in _constant_impl
    return _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 308, in _constant_eager_impl
    t = convert_to_eager_tensor(value, ctx, dtype)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 105, in convert_to_eager_tensor
    ctx.ensure_initialized()
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 552, in ensure_initialized
    config_str = self.config.SerializeToString()
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 1101, in config
    gpu_options = self._compute_gpu_options()
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 1164, in _compute_gpu_options
    raise ValueError("Memory growth cannot differ between GPU devices")
ValueError: Memory growth cannot differ between GPU devices

상황

tf.config.experimental.set_memory_growth를 호출하여 메모리 증가를 허용할 때 에러 발생

gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
  try:
    tf.config.experimental.set_memory_growth(gpus[0], True)
  except RuntimeError as e:
    # 프로그램 시작시에 메모리 증가가 설정되어야만 합니다
    print(e)

원인

gpu가 여러 개에  메모리 증가를 모두 허용해야 함

 

해결 방안

gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
  try:
    for i in range(len(gpus)):
    	tf.config.experimental.set_memory_growth(gpus[i], True)
  except RuntimeError as e:
    # 프로그램 시작시에 메모리 증가가 설정되어야만 합니다
    print(e)