Document some examples of use

[rom1504]

Document some examples of use

[rom1504]

one is cah

get http://3080.rom1504.fr/cah/cah_dataframe_unique/ then

img2dataset --url_list part_one.parquet --input_format "parquet" --url_col "URL" --caption_col "TEXT" --output_format webdataset --output_folder the_part_one_wds --thread_count 256 --image_size 256

running...

[rom1504]

img2dataset --url_list part_one.parquet --input_format "parquet" --url_col "URL" --caption_col "TEXT" --output_format webdataset --output_folder the_part_one_wds --thread_count 128 --image_size 256 took 18h to get 18M samples stored at http://3080.rom1504.fr/cah/cah_wds_part_one/
Average 20MB/s

I had to reduce the thread count due to a slow memory leak.
Learned from this experiment:

• need to increase a lot more the concurrency to increase download speed
• need to decrease number of process to decrease ram usage
• removing mem leaks would be great

will try doing that using asks + trio / raw multi threading

[rom1504]

done