colemurray/age-gender-estimation-tutorial

Tutorial for creating multi-task age and gender estimator in Tensorflow

age-gender-estimation-tutorial's Issues

Where is the model created?

Hi Cole, thanks for the tutorial.

I'm a little bit confused about the model we should create. I don't know what path I should use in the parameter --model-cnn when training, I mean I don't have a model file. How do I create it? I found the model-cnn.py file but it contains just some definitions of functions.

Thanks for your help!

problem during export model

Hello everyone,
i have a problem during the export, i got an error as the following:

line 480, in build_all_signature_defs
'{}'.format(type(export_outputs)))
ValueError: export_outputs must be a dict and not<class 'NoneType'>

i used your scripts for the entire process, the only difference is that, i don't use Doker.
Did i make any mistakes?

Thanks guys
Ric

Benchmark on current network

Did you perform any benchmarks on the current network? It would be great if there were accuracy estimates on a standard dataset.

tensorflow.python.framework.errors_impl.NotFoundError: /opt/app/var/train.csv; No such file or directory

Hi, Thanks for this great tutorial. Please help i get an error when running the train process.
command:
#building the image (CPU)
docker build -t colemurray/age-gender-estimation-tutorial:cpu .

#running the train process (CPU)
docker run -v $PWD:/opt/app \-e PYTHONPATH=$PYTHONPATH:/opt/app \-it colemurray/age-gender-estimation-tutorial:cpu \python3 /opt/app/bin/train.py \--img-dir /opt/app/var/crop \--train-csv /opt/app/var/train.csv \--val-csv /opt/app/var/val.csv \--model-dir /opt/app/var/cnn-model \--img-size 224 \--num-steps 200000

#error-output

INFO:tensorflow:Using config: {'_keep_checkpoint_max': 5, '_save_checkpoints_secs': None, '_global_id_in_cluster': 0, '_keep_checkpoint_every_n_hours': 10000, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f59660a6e80>, '_eval_distribute': None, '_is_chief': True, '_evaluation_master': '', '_log_step_count_steps': 100, '_protocol': None, '_device_fn': None, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_service': None, '_tf_random_seed': None, '_save_summary_steps': 100, '_master': '', '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_save_checkpoints_steps': 1500, '_task_type': 'worker', '_model_dir': '/opt/app/var/cnn-model', '_experimental_distribute': None, '_train_distribute': None}
INFO:tensorflow:Not using Distribute Coordinator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 1500 or save_checkpoints_secs None.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2020-01-23 02:16:32.502323: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-01-23 02:16:32.583359: W tensorflow/core/framework/allocator.cc:122] Allocation of 411041792 exceeds 10% of system memory.
2020-01-23 02:16:32.846042: W tensorflow/core/framework/allocator.cc:122] Allocation of 411041792 exceeds 10% of system memory.
2020-01-23 02:16:33.128939: W tensorflow/core/framework/allocator.cc:122] Allocation of 411041792 exceeds 10% of system memory.
2020-01-23 02:16:33.366611: W tensorflow/core/framework/allocator.cc:122] Allocation of 411041792 exceeds 10% of system memory.
2020-01-23 02:16:33.714329: W tensorflow/core/framework/allocator.cc:122] Allocation of 411041792 exceeds 10% of system memory.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into /opt/app/var/cnn-model/model.ckpt.
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
**tensorflow.python.framework.errors_impl.NotFoundError: /opt/app/var/train.csv; No such file or directory**
	 [[{{node IteratorGetNext}} = IteratorGetNext[output_shapes=[[?,224,224,3], [?], [?]], output_types=[DT_FLOAT, DT_INT64, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](OneShotIterator)]]

If you don't mind please help and add more documentation of it. Thank you so much.

Out of memory issue

Hello,

I am trying to reproduce your amazing work, but I am getting an out of mermory issue for some reasons.
I have tried to reduce batch_size, and reduce prefetch, none of these worked.

Here is a part of the log :

Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9605 MB memory) -> physical GPU (device: 0, name: TITAN
Xp COLLECTORS EDITION, pci bus id: 0000:01:00.0, compute capability: 6.1)
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into D:\StageVYNCK\GenderClass\VyncketUn\cnn_model\model.ckpt.
2019-05-17 12:19:47.423066: E tensorflow/stream_executor/cuda/cuda_driver.cc:868] failed to alloc 536870912 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-05-17 12:19:47.425793: W .\tensorflow/core/common_runtime/gpu/cuda_host_allocator.h:44] could not allocate pinned host memory of size: 536870912
2019-05-17 12:19:47.428146: E tensorflow/stream_executor/cuda/cuda_driver.cc:868] failed to alloc 483183872 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-05-17 12:19:47.430317: W .\tensorflow/core/common_runtime/gpu/cuda_host_allocator.h:44] could not allocate pinned host memory of size: 483183872
2019-05-17 12:19:47.432615: E tensorflow/stream_executor/cuda/cuda_driver.cc:868] failed to alloc 434865664 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-05-17 12:19:47.435390: W .\tensorflow/core/common_runtime/gpu/cuda_host_allocator.h:44] could not allocate pinned host memory of size: 434865664
2019-05-17 12:19:47.437544: E tensorflow/stream_executor/cuda/cuda_driver.cc:868] failed to alloc 536870912 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-05-17 12:19:47.439808: W .\tensorflow/core/common_runtime/gpu/cuda_host_allocator.h:44] could not allocate pinned host memory of size: 536870912
2019-05-17 12:19:57.455649: E tensorflow/stream_executor/cuda/cuda_driver.cc:868] failed to alloc 536870912 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-05-17 12:19:57.457975: W .\tensorflow/core/common_runtime/gpu/cuda_host_allocator.h:44] could not allocate pinned host memory of size: 536870912
2019-05-17 12:19:57.459555: E tensorflow/stream_executor/cuda/cuda_driver.cc:868] failed to alloc 536870912 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-05-17 12:19:57.461229: W .\tensorflow/core/common_runtime/gpu/cuda_host_allocator.h:44] could not allocate pinned host memory of size: 536870912
2019-05-17 12:19:57.462758: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (cuda_host_bfc) ran out of memory trying to allocate 392.00MiB. Current allocation summary follows.
2019-05-17 12:19:57.464715: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 12, Chunks in use: 12. 3.0KiB allocated for chunks. 3.0KiB in use in bin. 120B client-requested in use in bin.
2019-05-17 12:19:57.466972: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.2KiB client-requested in use in bin.
2019-05-17 12:19:57.469227: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.472371: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.474488: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): Total Chunks: 6, Chunks in use: 6. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin.
2019-05-17 12:19:57.476930: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): Total Chunks: 3, Chunks in use: 3. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin.
2019-05-17 12:19:57.479308: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.482225: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.484920: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.487328: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.489671: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 2, Chunks in use: 2. 808.0KiB allocated for chunks. 808.0KiB in use in bin. 808.0KiB client-requested in use in bin.
2019-05-17 12:19:57.492187: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 1, Chunks in use: 1. 581.0KiB allocated for chunks. 581.0KiB in use in bin. 404.0KiB client-requested in use in bin.
2019-05-17 12:19:57.494839: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.498008: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.500185: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 6, Chunks in use: 6. 24.00MiB allocated for chunks. 24.00MiB in use in bin. 24.00MiB client-requested in use in bin.
2019-05-17 12:19:57.502562: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.504796: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.507932: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.510289: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.512693: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.514864: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 1, Chunks in use: 1. 491.59MiB allocated for chunks. 491.59MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-05-17 12:19:57.517460: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 392.00MiB was 256.00MiB, Chunk State:
2019-05-17 12:19:57.518808: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400000 of size 256
2019-05-17 12:19:57.520036: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400100 of size 256
2019-05-17 12:19:57.521273: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400200 of size 256
2019-05-17 12:19:57.522748: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400300 of size 256
2019-05-17 12:19:57.524784: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400400 of size 256
2019-05-17 12:19:57.526171: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400500 of size 256
2019-05-17 12:19:57.527399: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400600 of size 256
2019-05-17 12:19:57.528672: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400700 of size 256
2019-05-17 12:19:57.529895: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400800 of size 256
2019-05-17 12:19:57.531114: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400900 of size 8192
2019-05-17 12:19:57.532375: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203402900 of size 8192
2019-05-17 12:19:57.534155: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203404900 of size 8192
2019-05-17 12:19:57.535444: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203406900 of size 256
2019-05-17 12:19:57.536714: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203406A00 of size 256
2019-05-17 12:19:57.537947: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203406B00 of size 256
2019-05-17 12:19:57.539200: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203406C00 of size 4096
2019-05-17 12:19:57.540521: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203407C00 of size 4096
2019-05-17 12:19:57.541766: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203408C00 of size 4096
2019-05-17 12:19:57.543185: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203409C00 of size 413696
2019-05-17 12:19:57.544617: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000020346EC00 of size 594944
2019-05-17 12:19:57.545895: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203600000 of size 4194304
2019-05-17 12:19:57.547194: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203A00000 of size 4194304
2019-05-17 12:19:57.548500: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203E00000 of size 4194304
2019-05-17 12:19:57.549777: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204200000 of size 413696
2019-05-17 12:19:57.551093: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204265000 of size 512
2019-05-17 12:19:57.552500: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204265200 of size 512
2019-05-17 12:19:57.553739: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204265400 of size 512
2019-05-17 12:19:57.554978: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204265600 of size 4194304
2019-05-17 12:19:57.556259: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204665600 of size 4194304
2019-05-17 12:19:57.557575: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204A65600 of size 4194304
2019-05-17 12:19:57.559832: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204E65600 of size 4096
2019-05-17 12:19:57.561373: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204E66600 of size 4096
2019-05-17 12:19:57.562672: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204E67600 of size 4096
2019-05-17 12:19:57.563905: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204E68600 of size 515471872
2019-05-17 12:19:57.565208: I tensorflow/core/common_runtime/bfc_allocator.cc:638] Summary of in-use Chunks by size:
2019-05-17 12:19:57.566461: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 12 Chunks of size 256 totalling 3.0KiB
2019-05-17 12:19:57.567741: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-05-17 12:19:57.569629: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 4096 totalling 24.0KiB
2019-05-17 12:19:57.570891: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 8192 totalling 24.0KiB
2019-05-17 12:19:57.572133: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 413696 totalling 808.0KiB
2019-05-17 12:19:57.573406: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 594944 totalling 581.0KiB
2019-05-17 12:19:57.574675: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 4194304 totalling 24.00MiB
2019-05-17 12:19:57.575958: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 515471872 totalling 491.59MiB
2019-05-17 12:19:57.577300: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 517.00MiB
2019-05-17 12:19:57.579081: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit: 68719476736
InUse: 542113792
MaxInUse: 542113792
NumAllocs: 67
MaxAllocSize: 536870912

2019-05-17 12:19:57.581960: W tensorflow/core/common_runtime/bfc_allocator.cc:271] *********************************************************************************xxxxxxxxxxxxxxxxxxx
Traceback (most recent call last):
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
return fn(*args)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
[[{{node dense/kernel/Adam}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\StageVYNCK\GenderClass\VyncketUn\bin\train.py", line 45, in
tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 471, in train_and_evaluate
return executor.run()
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 611, in run
return self.run_local()
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 712, in run_local
saving_listeners=saving_listeners)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 358, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1124, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1158, in _train_model_default
saving_listeners)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1403, in _train_with_estimator_spec
log_step_count_steps=log_step_count_steps) as mon_sess:
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\monitored_session.py", line 508, in MonitoredTrainingSession
stop_grace_period_secs=stop_grace_period_secs)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\monitored_session.py", line 934, in init
stop_grace_period_secs=stop_grace_period_secs)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\monitored_session.py", line 648, in init
self._sess = _RecoverableSession(self._coordinated_creator)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1122, in init
_WrappedSession.init(self, self._create_session())
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1127, in _create_session
return self._sess_creator.create_session()
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\monitored_session.py", line 812, in create_session
hook.after_create_session(self.tf_sess, self.coord)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 568, in after_create_session
self._save(session, global_step)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 599, in _save
self._get_saver().save(session, self._save_path, global_step=step)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\saver.py", line 1171, in save
{self.saver_def.filename_tensor_name: checkpoint_file})
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
run_metadata_ptr)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
run_metadata)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
[[node dense/kernel/Adam (defined at D:/StageVYNCK/genderclass/vyncketun\age_gender\cnn_estimator.py:99) ]]

Caused by op 'dense/kernel/Adam', defined at:
File "D:\StageVYNCK\GenderClass\VyncketUn\bin\train.py", line 45, in
tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 471, in train_and_evaluate
return executor.run()
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 611, in run
return self.run_local()
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 712, in run_local
saving_listeners=saving_listeners)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 358, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1124, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1154, in _train_model_default
features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1112, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "D:/StageVYNCK/genderclass/vyncketun\age_gender\cnn_estimator.py", line 30, in model_fn
return get_training_spec(learning_rate, joint_loss)
File "D:/StageVYNCK/genderclass/vyncketun\age_gender\cnn_estimator.py", line 99, in get_training_spec
global_step=tf.train.get_global_step())
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\optimizer.py", line 413, in minimize
name=name)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\optimizer.py", line 595, in apply_gradients
self._create_slots(var_list)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\adam.py", line 135, in _create_slots
self._zeros_slot(v, "m", self._name)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\optimizer.py", line 1153, in _zeros_slot
new_slot_variable = slot_creator.create_zeros_slot(var, op_name)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\slot_creator.py", line 183, in create_zeros_slot
colocate_with_primary=colocate_with_primary)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\slot_creator.py", line 157, in create_slot_with_initializer
dtype)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\slot_creator.py", line 65, in _create_slot_var
validate_shape=validate_shape)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1479, in get_variable
aggregation=aggregation)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1220, in get_variable
aggregation=aggregation)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 547, in get_variable
aggregation=aggregation)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 499, in _true_getter
aggregation=aggregation)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 911, in _get_single_variable
aggregation=aggregation)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variables.py", line 213, in call
return cls._variable_v1_call(*args, **kwargs)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variables.py", line 176, in _variable_v1_call
aggregation=aggregation)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variables.py", line 155, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 2495, in default_variable_creator
expected_shape=expected_shape, import_scope=import_scope)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variables.py", line 217, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variables.py", line 1395, in init
constraint=constraint)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variables.py", line 1509, in _init_from_args
name=name)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\state_ops.py", line 79, in variable_op_v2
shared_name=shared_name)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\gen_state_ops.py", line 1425, in variable_v2
shared_name=shared_name, name=name)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op
op_def=op_def)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\framework\ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()

InternalError (see above for traceback): Dst tensor is not initialized.
[[node dense/kernel/Adam (defined at D:/StageVYNCK/genderclass/vyncketun\age_gender\cnn_estimator.py:99) ]]

Do you have any idea for me to resolve this ? thanks a lot

Dockerfile.gpu references to non-existing requirements-gpu.txt

While executing docker build -f Dockerfile.gpu:

Step 3/5 : ADD $PWD/requirements-gpu.txt /requirements.txt
ADD failed: stat /var/lib/docker/tmp/docker-builder598871921/requirements-gpu.txt: no such file or directory

Can't create docker

Hi,
I did install Docker version 18.09.7 on my Ubuntu 18.04. I did follow your instruction to install docker image as this command
$docker build -t colemurray/age-gender-estimation-tutorial
I got this error
"docker build" requires exactly 1 argument.
See 'docker build --help'.
Any advice?
Thank you.

Can i use colab

i dont have a gpu ,So can u tell me how to use colab

Can this tutorial being used to detect both age and gender in one foreground network

Hi, Cole

Could you pls upload some testing method and demo trained models?

And, can this model directly predict both age and gender in once computation?

Or .. bascially, this is two seperate network, once for age classification and the other is for gender, and run one by one.

Missing '=' in requirements for opencv

Unable to find image gpu locally.

I have successfully built the images and done all the steps correctly.
But when i run command:
sudo docker run -v $PWD:/opt/app -e PYTHONPATH=$PYTHONPATH:/opt/app -it colemurray/age-gender-estimation-tutorial:gpu python3 /opt/app/bin/train.py --img-dir /opt/app/var/crop --train-csv /opt/app/var/train.csv --val-csv /opt/app/var/val.csv --model-dir /opt/app/var/cnn-model --img-size 224 --num-steps 200000

I get the following error:
Unable to find image 'colemurray/age-gender-estimation-tutorial:gpu' locally
docker: Error response from daemon: manifest for colemurray/age-gender-estimation-tutorial:gpu not found.
See 'docker run --help'.

Please help @ColeMurray @aclex @Gius-8 .
Thanks in advance.

Use epochs

Hi!
| was wondering how is it better to modify the code to use epochs? It seems that for more advanced networks architecture it could make sense. Maybe you do have some ideas, cause I've tried several solutions and it doesn't work properly for me (I'm quite new to Tensorflow and especially to Tensorflow Estimators API).

Use this model in an app

Can I use this model in an ios app which will detect the age of any person?
Can I convert this model to the core ML model to use it in an ios app to detect the age of any person?

About InvalidArgumentError

I have run your code with all the requirement file you specified in requirement.txt
but I got InvalidArgumentError error continuously while running same code.
I have just change directory in which data available.you can see that on code.

the error occur continuously on 22 line

you can see here

help me out with this.
Thanks in advance

and my system specification is