NetVLAD Descriptors for Training #58
Comments
|
Hi and sorry for the late reply. This is an old codebase and I am not providing much support for it. What version of Tensorflow are you using? Afaik this codebase only supports v1.12. It's unclear from the log whether TensorFlow was installed with GPU capability - it should. Could you check that in earlier logs? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I've been able to export the SuperPoint descriptors but am getting an error when running the script to export the NetVLAD descriptors. It seems to be indicating that I have an incompatible device although I'm not sure what that could be, I have a SkyLake processor and GTX 1080 Ti graphics card. This occurs in base_model.py when executing
Any idea how to work around this issue? Complete output is below. thanks
Traceback (most recent call last):
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in do_call
return fn(*args)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1339, in run_fn
self.extend_graph()
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1374, in extend_graph
tf_session.ExtendSession(self.session)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node {{colocation_node vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/RandomUniform}} was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0].
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index=-1 requested_device_name='/device:GPU:0' assigned_device_name='' resource_device_name='/device:GPU:0' supported_device_types=[CPU] possible_devices_=[]
Assign: CPU
Identity: CPU XLA_CPU
VariableV2: CPU
Mul: CPU XLA_CPU
Add: CPU XLA_CPU
Sub: CPU XLA_CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/shape (Const)
vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/min (Const)
vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/max (Const)
vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/RandomUniform (RandomUniform)
vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/sub (Sub)
vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/mul (Mul)
vgg16_netvlad_pca/average_rgb/Initializer/random_uniform (Add)
vgg16_netvlad_pca/average_rgb (VariableV2) /device:GPU:0
vgg16_netvlad_pca/average_rgb/Assign (Assign) /device:GPU:0
vgg16_netvlad_pca/average_rgb/read (Identity) /device:GPU:0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/brian/.vscode/extensions/ms-python.python-2021.3.680753044/pythonFiles/lib/python/debugpy/main.py", line 45, in
cli.main()
File "/home/brian/.vscode/extensions/ms-python.python-2021.3.680753044/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
run()
File "/home/brian/.vscode/extensions/ms-python.python-2021.3.680753044/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
runpy.run_path(target_as_str, run_name=compat.force_str("main"))
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/runpy.py", line 85, in run_code
exec(code, run_globals)
File "hfnet/export_predictions.py", line 59, in
**config['model']) as net:
File "/home/brian/Desktop/hfnet/hfnet/models/base_model.py", line 125, in init
self.build_graph()
File "/home/brian/Desktop/hfnet/hfnet/models/base_model.py", line 293, in build_graph
l])
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in run
feed_dict_tensor, options, run_metadata)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in do_run
run_metadata)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node node vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/RandomUniform (defined at /home/brian/Desktop/hfnet/hfnet/models/netvlad_original.py:78) placed on device Device assignments active during op 'vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/RandomUniform' creation:
with tf.device(None): </home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/variables.py:1696>
with tf.device(/gpu:0): </home/brian/Desktop/hfnet/hfnet/models/base_model.py:230> was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0].
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index=-1 requested_device_name='/device:GPU:0' assigned_device_name='' resource_device_name='/device:GPU:0' supported_device_types=[CPU] possible_devices=[]
Assign: CPU
Identity: CPU XLA_CPU
VariableV2: CPU
Mul: CPU XLA_CPU
Add: CPU XLA_CPU
Sub: CPU XLA_CPU
RandomUniform: CPU XLA_CPU
Const: CPU XLA_CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/shape (Const)
vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/min (Const)
vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/max (Const)
vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/RandomUniform (RandomUniform)
vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/sub (Sub)
vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/mul (Mul)
vgg16_netvlad_pca/average_rgb/Initializer/random_uniform (Add)
vgg16_netvlad_pca/average_rgb (VariableV2) /device:GPU:0
vgg16_netvlad_pca/average_rgb/Assign (Assign) /device:GPU:0
vgg16_netvlad_pca/average_rgb/read (Identity) /device:GPU:0
Device assignments active during op 'vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/RandomUniform' creation:
with tf.device(None): </home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/variables.py:1696>
with tf.device(/gpu:0): </home/brian/Desktop/hfnet/hfnet/models/base_model.py:230>
Original stack trace for 'vgg16_netvlad_pca/average_rgb/Initializer/random_uniform/RandomUniform':
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/brian/.vscode/extensions/ms-python.python-2021.3.680753044/pythonFiles/lib/python/debugpy/main.py", line 45, in
cli.main()
File "/home/brian/.vscode/extensions/ms-python.python-2021.3.680753044/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
run()
File "/home/brian/.vscode/extensions/ms-python.python-2021.3.680753044/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
runpy.run_path(target_as_str, run_name=compat.force_str("main"))
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "hfnet/export_predictions.py", line 59, in
**config['model']) as net:
File "/home/brian/Desktop/hfnet/hfnet/models/base_model.py", line 125, in init
self._build_graph()
File "/home/brian/Desktop/hfnet/hfnet/models/base_model.py", line 276, in _build_graph
self._pred_graph(self.pred_in)
File "/home/brian/Desktop/hfnet/hfnet/models/base_model.py", line 231, in _pred_graph
pred_out = self._model(data, Mode.PRED, **self.config)
File "/home/brian/Desktop/hfnet/hfnet/models/netvlad_original.py", line 78, in _model
'average_rgb', 3, dtype=image_batch.dtype)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1496, in get_variable
aggregation=aggregation)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1239, in get_variable
aggregation=aggregation)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 562, in get_variable
aggregation=aggregation)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 514, in _true_getter
aggregation=aggregation)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 929, in _get_single_variable
aggregation=aggregation)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 259, in call
return cls._variable_v1_call(*args, **kwargs)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 220, in _variable_v1_call
shape=shape)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 198, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2511, in default_variable_creator
shape=shape)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 263, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 1568, in init
shape=shape)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 1698, in _init_from_args
initial_value(), name="initial_value", dtype=dtype)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 901, in
partition_info=partition_info)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py", line 533, in call
shape, -limit, limit, dtype, seed=self.seed)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/random_ops.py", line 247, in random_uniform
rnd = gen_random_ops.random_uniform(shape, dtype, seed=seed1, seed2=seed2)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/ops/gen_random_ops.py", line 820, in random_uniform
name=name)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/brian/miniconda3/envs/hfnet/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()
The text was updated successfully, but these errors were encountered: