mlcommons/storage

MLPerf™ Storage Benchmark Suite

storage's Issues

Single host runreport not working?

I did a run on a single host:

./benchmark.sh run --workload unet3d --num-accelerators 8 --results-dir unet3d_results-1 --param dataset.data_folder=/panfs/pgh-realm-75-4/r6-256/unet3d/ --param dataset.num_files_train=3200

after the run it seems like json files were created ok?

[root@pgh-echo-1-1 storage]# ls unet3d_results-1/ 0_output.json 1_output.json 2_output.json 3_output.json 4_output.json 5_output.json 6_output.json 7_output.json configs dlio.log per_epoch_stats.json summary.json

When I do the reportgen it says I need a multi-host type format?

[root@pgh-echo-1-1 storage]# ./benchmark.sh reportgen -r unet3d_results-1 2023-05-04 16:11:20 Error: Directory structure unet3d_results-1/summary.json is not correct. It has be in format result_dir/run(1..n)/host(1..n)/summary.json

I don't see instructions in the readme for how to do multi-host but I will look in benchmark.sh for more details.

I tried to fake the multi home layout it asked for but that did not work.

[root@pgh-echo-1-1 storage]# mkdir unet3d_results-onehost-8 [root@pgh-echo-1-1 storage]# mkdir unet3d_results-onehost-8/run1 [root@pgh-echo-1-1 storage]# mkdir unet3d_results-onehost-8/run1/host1/ [root@pgh-echo-1-1 storage]# cp -rf unet3d_results-1/* unet3d_results-onehost-8/run1/host1/ [root@pgh-echo-1-1 storage]# ./benchmark.sh reportgen -r unet3d_results-onehost-8/ 2023-05-04 16:14:21 Error: AU value didn't pass the threshold in the run reported by unet3d_results-onehost-8/run1/host1/summary.json

Let me know if there are any suggestions.

Update DLRM mentions in readme, etc.

First of all, the https://github.com/mlcommons/storage/blob/main/README.md is beautiful and really nicely done!

Since DLRM isn't in first version, so rather than saying "todo", can we explicitly indicate that?

Hydra Error when trying to run datagen

Version: commit bd21330 (HEAD, tag: v0.5-rc1)

The Benchmark seems to install ok.

[kmannthey@pgh-echo-1-1 storage]$ pip3.10  install -r dlio_benchmark/requirements.txt
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu
Requirement already satisfied: absl-py==1.3.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 4)) (1.3.0)
Requirement already satisfied: antlr4-python3-runtime==4.9.3 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 5)) (4.9.3)
Requirement already satisfied: astunparse==1.6.3 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 6)) (1.6.3)
Requirement already satisfied: cachetools==5.2.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 7)) (5.2.0)
Requirement already satisfied: certifi==2022.9.24 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 8)) (2022.9.24)
Requirement already satisfied: charset-normalizer==2.1.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 9)) (2.1.1)
Requirement already satisfied: flatbuffers==22.10.26 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 10)) (22.10.26)
Requirement already satisfied: gast==0.4.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 11)) (0.4.0)
Requirement already satisfied: google-auth==2.14.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 12)) (2.14.1)
Requirement already satisfied: google-auth-oauthlib==0.4.6 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 13)) (0.4.6)
Requirement already satisfied: google-pasta==0.2.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 14)) (0.2.0)
Requirement already satisfied: grpcio==1.51.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 15)) (1.51.0)
Requirement already satisfied: h5py==3.7.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 16)) (3.7.0)
Requirement already satisfied: hydra-core==1.2.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 17)) (1.2.0)
Requirement already satisfied: idna==3.4 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 18)) (3.4)
Requirement already satisfied: keras==2.11.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 19)) (2.11.0)
Requirement already satisfied: libclang==14.0.6 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 20)) (14.0.6)
Requirement already satisfied: Markdown==3.4.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 21)) (3.4.1)
Requirement already satisfied: MarkupSafe==2.1.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 22)) (2.1.1)
Requirement already satisfied: mpi4py==3.1.4 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 23)) (3.1.4)
Requirement already satisfied: numpy==1.23.5 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 24)) (1.23.5)
Requirement already satisfied: oauthlib==3.2.2 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 25)) (3.2.2)
Requirement already satisfied: omegaconf==2.2.3 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 26)) (2.2.3)
Requirement already satisfied: opt-einsum==3.3.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 27)) (3.3.0)
Requirement already satisfied: packaging==21.3 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 28)) (21.3)
Requirement already satisfied: pandas==1.5.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 29)) (1.5.1)
Requirement already satisfied: Pillow==9.3.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 30)) (9.3.0)
Requirement already satisfied: protobuf==3.19.6 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 31)) (3.19.6)
Requirement already satisfied: pyasn1==0.4.8 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 32)) (0.4.8)
Requirement already satisfied: pyasn1-modules==0.2.8 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 33)) (0.2.8)
Requirement already satisfied: pyparsing==3.0.9 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 34)) (3.0.9)
Requirement already satisfied: python-dateutil==2.8.2 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 35)) (2.8.2)
Requirement already satisfied: pytz==2022.6 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 36)) (2022.6)
Requirement already satisfied: PyYAML==6.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 37)) (6.0)
Requirement already satisfied: requests==2.28.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 38)) (2.28.1)
Requirement already satisfied: requests-oauthlib==1.3.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 39)) (1.3.1)
Requirement already satisfied: rsa==4.9 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 40)) (4.9)
Requirement already satisfied: six==1.16.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 41)) (1.16.0)
Requirement already satisfied: tensorboard==2.11.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 42)) (2.11.0)
Requirement already satisfied: tensorboard-data-server==0.6.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 43)) (0.6.1)
Requirement already satisfied: tensorboard-plugin-wit==1.8.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 44)) (1.8.1)
Requirement already satisfied: tensorflow==2.11.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 45)) (2.11.0)
Requirement already satisfied: tensorflow-io==0.28.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 46)) (0.28.0)
Requirement already satisfied: tensorflow-estimator==2.11.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 47)) (2.11.0)
Requirement already satisfied: termcolor==2.1.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 48)) (2.1.1)
Requirement already satisfied: torch==1.13.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 49)) (1.13.0+cpu)
Requirement already satisfied: torchaudio==0.13.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 50)) (0.13.0+cpu)
Requirement already satisfied: torchvision==0.14.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 51)) (0.14.0+cpu)
Requirement already satisfied: typing_extensions==4.4.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 52)) (4.4.0)
Requirement already satisfied: urllib3==1.26.12 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 53)) (1.26.12)
Requirement already satisfied: Werkzeug==2.2.2 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 54)) (2.2.2)
Requirement already satisfied: wrapt==1.14.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from -r dlio_benchmark/requirements.txt (line 55)) (1.14.1)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from astunparse==1.6.3->-r dlio_benchmark/requirements.txt (line 6)) (0.40.0)
Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.10/site-packages (from tensorboard==2.11.0->-r dlio_benchmark/requirements.txt (line 42)) (65.5.0)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from tensorflow==2.11.0->-r dlio_benchmark/requirements.txt (line 45)) (0.28.0)
[kmannthey@pgh-echo-1-1 storage]$ 

When I try and run datagen I get errors about hydra.

[kmannthey@pgh-echo-1-1 storage]$ ./benchmark.sh datagen --workload unet3d --num-parallel 8 --param dataset.num_subfolders_train=10 --param dataset.data_folder=unet3d_data
Traceback (most recent call last):
  File "dlio_benchmark/src/dlio_benchmark.py", line 19, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'
Traceback (most recent call last):
  File "dlio_benchmark/src/dlio_benchmark.py", line 19, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'
Traceback (most recent call last):
  File "dlio_benchmark/src/dlio_benchmark.py", line 19, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'
Traceback (most recent call last):
  File "dlio_benchmark/src/dlio_benchmark.py", line 19, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'
Traceback (most recent call last):
  File "dlio_benchmark/src/dlio_benchmark.py", line 19, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'
Traceback (most recent call last):
  File "dlio_benchmark/src/dlio_benchmark.py", line 19, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'
Traceback (most recent call last):
  File "dlio_benchmark/src/dlio_benchmark.py", line 19, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'
Traceback (most recent call last):
  File "dlio_benchmark/src/dlio_benchmark.py", line 19, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'
[kmannthey@pgh-echo-1-1 storage]$ 

hydra-core is installed.

Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: hydra-core in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (1.2.0)
Requirement already satisfied: omegaconf~=2.2 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from hydra-core) (2.2.3)
Requirement already satisfied: antlr4-python3-runtime==4.9.* in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from hydra-core) (4.9.3)
Requirement already satisfied: packaging in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from hydra-core) (21.3)
Requirement already satisfied: PyYAML>=5.1.0 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from omegaconf~=2.2->hydra-core) (6.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /net/nfs.paneast.panasas.com/home/kmannthey/.local/lib/python3.10/site-packages (from packaging->hydra-core) (3.0.9)
[kmannthey@pgh-echo-1-1 storage]$ 

Any ideas?