tar_patch_4096 with webdataset API

[mahmoodlab]

tar_patch_4096 with webdataset API

[JuanDuranMcgill]

For example, do I take the directories from each patch, turn them into json files, and then store them with "from webdataset import TarWriter"? I am trying to figure out how the information is stored.

[Richarizardd]

Hi @JuanDuranMcgill

You do not need to turn the images into a *.tar format for WebDataset. Originally, all 400K-ish [4096 × 4096] images were saved in a WebDataset format, to prevent reading directly from Openslide. However, like CLAM, its much easier to just extract features directly from the WSI rather than to 1) save the image patch, then 2) extract features.

How are you extracting features? I would modify the CLAM feature extraction pipeline to directly use HIPT_4K for feature extraction of specified 4096 x 4096 patches.