about validation_for_net

[zaccharieramzi]

about validation_for_net

[zaccharieramzi]

Hi @fharman ,

It's not possible to compute image quality metrics for the test dataset since it doesn't have the ground truth images.
The way to do this would be to reconstruct all the images of the test dataset, put them in a single folder, zip it and upload to, for example, Dropbox, and submit this zipped folder containing your reconstructions to https://fastmri.org/.

This way your reconstructions will appear in the leaderboards.

Hope this helps

[fharman]

Thank you for your return. I try to reach your result. So i wrote down this.

Thank you again Zaccharie for everything.

Best regards,

[zaccharieramzi]

Which result are you referring to?

[fharman]

I mean , i hope to reach a result in PSNR(28.4db- 33.2 db) with high SSIM value. But when i tried, zero filled result is higher than unet.(i thought it is less than full val. dataset).

Also, in metrics std is multiplied with 2. Why?

[zaccharieramzi]

So you are talking about the Proton Density with Fat Suppression (PDFS) U-net results from the journal paper?
I am very surprised that your U-net results are less than the Zero-Filled, since the U-net is starting from the Zero-Filled image so it should at least perform the same.
What are your results exactly?

Re the *2 in std representation: I have no clue why it's there. I took this code from the original fastMRI repo but never paid attention to this part !!!
Thank you very much for signaling this to me, I will submit a correction ASAP.

[fharman]

So you are talking about the Proton Density with Fat Suppression (PDFS) U-net results from the journal paper?
I am very surprised that your U-net results are less than the Zero-Filled, since the U-net is starting from the Zero-Filled image so it should at least perform the same.
What are your results exactly?

Re the *2 in std representation: I have no clue why it's there. I took this code from the original fastMRI repo but never paid attention to this part !!!
Thank you very much for signaling this to me, I will submit a correction ASAP.

Hi Zaccharie,

My result is low in SSIM as you see in below. So i hoped to reach a result close to Challenge Leaderboard.(0.65-0.75 in SSIM).

PSNR-mean (std) (dB) SSIM-mean (std) # params Runtime (s)
zfilled 25.91 (3.012) 0.5049 (0.1147) 0 0.4042
unet 28.15 (4.37) 0.4313 (0.1223) 481801 0.4002

Also I want to ask you that is there any step for test dataset? Because all codes for validation in reconstruction. But as you know test dataset is zero-filled in default. So ı pruned the codes but i think there could be a way for seeing the reconstruction of test dataset in unet despite of validation dataset.

Thank you for all your support,

Best Regards,

[zaccharieramzi]

Hi @fharman ,

I think your results are low both in PSNR and SSIM compared to my results, even for the zero-filled version.
Are you showing me the results for both contrasts or for one contrast only?
For how many epochs did you train the U-net?

Also you shouldn't compare against the leaderboard for the knee dataset, since the results are slightly higher on the leaderboard than on the validation set.

Again, for the test set we don't have access to the ground truth so it's impossible to know the image quality metrics. And for simple inference for later submission, I think that unfortunately I don't have the scripts for the old functional models, but I will gladly receive a contribution on that.
I gave you the instructions on how to submit the test results here.

[fharman]

Hi again, Yes my validation results are unfortunately low.So i tried to reach close to your result but did not. I just only used fastmri knee dataset and that is pdfs. I used for testing your 300 epoch checkpoint result that you put in checkpoints folder. I know test dataset did not have the ground truth and consequently i did not have metrics values. But for seeing the results of reconstruction of the unet in fastmri knee test dataset, i asked whether a way for this or not despite the pruning the all codes.Because as you know test datasets have the default aliasing artifacts.So i dont need the use of Zerofilled2DSequence and Masked2DSequence. Thank you for all your support, Best Regards 12 Kas 2020 Per 18:42 tarihinde Zaccharie Ramzi <[email protected]> şunu yazdı:

…

Hi @fharman <https://github.com/fharman> , I think your results are low both in PSNR and SSIM compared to my results, even for the zero-filled version. Are you showing me the results for both contrasts or for one contrast only? For how many epochs did you train the U-net? Also you shouldn't compare against the leaderboard for the knee dataset, since the results are slightly higher on the leaderboard than on the validation set. Again, for the test set we don't have access to the ground truth so it's impossible to know the image quality metrics. And for simple inference for later submission, I think that unfortunately I don't have the scripts for the old functional models, but I will gladly receive a contribution on that. I gave you the instructions on how to submit the test results here <#104 (comment)> . — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO4IQ2VH2YZZOV435C7G3JDSPP667ANCNFSM4TMUBXOQ> .

[zaccharieramzi]

Aaaaah ok, I didn't know you used my checkpoints.
I think at some point there was a breaking change in the U-net implementation... But I don't remember when this happened... Sorry about that.
I will notify this change in the Readme somehwere.

What you can do to still test with these checkpoints is to checkout to the commit they were added (git checkout bcd3fdd).

Re the test set, you have a mode argument for the ZeroFilled2DSequence, which you can set to 'testing' (same for MaskedSequence).

[zaccharieramzi]

Actually, I just realized that I had planned a release for that, you can see this here.

You can do git checkout old_unet to be in a state where the UNet saved checkpoints correspond to the UNet architecture.

[fharman]

Thank you very much Zaccharie Ramzi <[email protected]>, 16 Kas 2020 Pzt, 15:59 tarihinde şunu yazdı:

…

Actually, I just realized that I had planned a release for that, you can see this here <https://github.com/zaccharieramzi/fastmri-reproducible-benchmark/releases/tag/old_unet> . You can do git checkout old_unet to be in a state where the UNet saved checkpoints correspond to the UNet architecture. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO4IQ2XXWNEK2OTXHLCOWNLSQEO3LANCNFSM4TMUBXOQ> .

[zaccharieramzi]

Glad I could help (this is very old work so that I did when starting my PhD so it's not really well organised, sorry for that)!
Please let me know if this works.

[fharman]

Hi Zaccharie, I had a result like this(with your old checkpoint) PSNR-mean (std) (dB)SSIM-mean (std)# paramsRuntime (s) unet 28.44 (2.717) 0.4738 (0.1999) 481801 0.7312 zfilled 28.45 (2.624) 0.5799 (0.09451) 0 0.4972 it is really bad in comparison with new version. It is hoped to have better than zero filled. I took the validation path like this AF = 4 contrast = 'CORPDFS_FBK' # CORPDFS_FBK CORPD_FBK None val_path = '/content/drive/My Drive/UNET_Colab/knee_singlecoil_val/singlecoil_val/' val_gen_zero = ZeroFilled2DSequence(val_path, af=AF, norm=True, mode= 'validation', contrast=contrast) #val_gen_zero.filenames = val_gen_zero.filenames[:2] val_gen_scaled = Masked2DSequence(val_path, mode='validation' , af=AF, scale_factor=1e6, contrast=contrast) #val_gen_scaled.filenames = val_gen_scaled.filenames[:2] Thank you for your support, Best regards Zaccharie Ramzi <[email protected]>, 16 Kas 2020 Pzt, 16:31 tarihinde şunu yazdı:

…

Glad I could help (this is very old work so that I did when starting my PhD so it's not really well organised, sorry for that)! Please let me know if this works. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO4IQ2QUMGVIRPYNS4X2LJTSQEST7ANCNFSM4TMUBXOQ> .

[zaccharieramzi]

Ok I just checked and I also had this issue at the release old_unet. It must mean that there is a bug somewhere and I will fix it eventually.

What you can do is to checkout to the commit I had mentioned earlier (the one where I added the weights), and everything should work just fine git checkout bcd3fdd (this one doesn't have a separation per contrast).

You will need to install PyTorch and ModOpt for everything to run smoothly (pip install torch modopt).
Sorry about that.

[fharman]

Hi Zaccharie, My results for validation dataset with unet_af4_1569210349 are shown below. It is really bad(with 99 validation file). What should I do to reach your result in Fastmri SingleCoil Knee Dataset Result [image: image.png] Thank you for support and everything. Best regards, Zaccharie Ramzi <[email protected]>, 16 Kas 2020 Pzt, 20:40 tarihinde şunu yazdı:

…

Ok I just checked and I also had this issue at the release old_unet. It must mean that there is a bug somewhere and I will fix it eventually. What you can do is to checkout to the commit I had mentioned earlier <#104 (comment)> (the one where I added the weights), and everything should work just fine git checkout bcd3fdd (this one doesn't have a separation per contrast). You will need to install PyTorch and ModOpt for everything to run smoothly (pip install torch modopt). Sorry about that. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO4IQ2T72SDMIPPR5UPPC4LSQFP2BANCNFSM4TMUBXOQ> .

[zaccharieramzi]

Hi @fharman ,

I didn't get your image. But if they are not like in the paper, you probably didn't checked out to the commit I sent you git checkout bcd3fdd.

Could you share you colab with me for me to check that everything is as expected?

[fharman]

Hi again, I could not find your checkout git checkout bcd3fdd. My colab link is https://colab.research.google.com/drive/17e--_Nd_DUrc9eWqEDCEA1mpvC-Vc9J4#scrollTo=Fqc6TvjZsxkG Thank you for your support, Best regards, 17 Kas 2020 Sal 15:55 tarihinde Zaccharie Ramzi <[email protected]> şunu yazdı:

…

Hi @fharman <https://github.com/fharman> , I didn't get your image. But if they are not like in the paper, you probably didn't checked out to the commit I sent you git checkout bcd3fdd. Could you share you colab with me for me to check that everything is as expected? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO4IQ2UTREEKLLSMXNARPBDSQJXEHANCNFSM4TMUBXOQ> .

[zaccharieramzi]

Ah I think it's because you didn't do the checkout correctly.
In colab, after cloning the repo, you can do !git checkout bcd3fdd.
Checking out to an older commit allows you to have the code at the time of writing of this commit.
Don't hesitate to read a bit about git and versioning to know more about this, I think it will help you greatly in the future.

Also don't forget to grant me the access to the colab for me to review it.

[fharman]

Hi, Yes i downloaded checkout bcd3fdd as unet_af4_1569210349 (300 epoch). But for 50 validation files, SSIM is still lower than Zerofilled. I hope to reach at least close to your merits in FastMRI Single Coil Results. This is my colab link. https://colab.research.google.com/drive/17e--_Nd_DUrc9eWqEDCEA1mpvC-Vc9J4#scrollTo=wkg4uvuxsxjf (By the way i used 50 Validation Files for evaluation because of limited Colab RAM(12 GB). I plan to split all data like this and then averaged this according to split number and share the results with you. ) Best regards, Zaccharie Ramzi <[email protected]>, 17 Kas 2020 Sal, 17:53 tarihinde şunu yazdı:

…

Ah I think it's because you didn't do the checkout correctly. In colab, after cloning the repo, you can do !git checkout bcd3fdd. Checking out to an older commit allows you to have the code at the time of writing of this commit. Don't hesitate to read a bit about git and versioning to know more about this, I think it will help you greatly in the future. Also don't forget to grant me the access to the colab for me to review it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO4IQ2WAM2H3MKCHANC6Z53SQKFADANCNFSM4TMUBXOQ> .

[zaccharieramzi]

No but you need also the code corresponding to this commit not only the checkpoints.

And you still didn't grant me access permissions to your colab.

[fharman]

Sorry ,i granted it.But because of internet ,it was not approved. Please check it again. 18 Kas 2020 Çar 01:30 tarihinde Zaccharie Ramzi <[email protected]> şunu yazdı:

…

No but you need also the code corresponding to this commit not only the checkpoints. And you still didn't grant me access permissions to your colab. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO4IQ2RUZTTSTGK4TBS5WU3SQL2PVANCNFSM4TMUBXOQ> .

[zaccharieramzi]

I just checked, I still don't have access to it.

[fharman]

https://colab.research.google.com/drive/17e--_Nd_DUrc9eWqEDCEA1mpvC-Vc9J4 for 50 validation files. please confirm me if it is not accessed Zaccharie Ramzi <[email protected]>, 18 Kas 2020 Çar, 01:37 tarihinde şunu yazdı:

…

I just checked, I still don't have access to it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO4IQ2QXIUBNB6MVTYONL2TSQL3LZANCNFSM4TMUBXOQ> .

[zaccharieramzi]

Still not it.

In the top right hand corner, when using a colab notebook, you have a share button (here in French, "Partager").
Click on it, and add my email address to the colaborators: [email protected].

Before I check, please make sure you are on the right git commit(bcd3fdd ), to compute the metrics.

[fharman]

Hi Zaccharie, I am realy sorry for this. I still don't know why this happened. Because I did the same thing like in the image. I am doing again. Best regards, Zaccharie Ramzi <[email protected]>, 18 Kas 2020 Çar, 11:17 tarihinde şunu yazdı:

…

Still not it. [image: colab_share (1)] <https://user-images.githubusercontent.com/6387497/99503532-9d809580-297e-11eb-907b-ea5ec2452de6.png> In the top right hand corner, when using a colab notebook, you have a share button (here in French, "Partager"). Click on it, and add my email address to the colaborators: ***@***.*** Before I check, please make sure you are on the right git commit(bcd3fdd ), to compute the metrics. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO4IQ2VV45O7N2VMDCZ5K6DSQN7JZANCNFSM4TMUBXOQ> .

[zaccharieramzi]

No I have the access (the last one was the right move).

In your notebook you didn't checkout to the commit I mentioned. How did you do it?

[fharman]

Hi Zaccharie, bcd3fdd As you see, i could download the checkpoints/unet_af4_1569210349-300.hdf5. So I changed the path and name in all_net_params. Zaccharie Ramzi <[email protected]>, 18 Kas 2020 Çar, 11:28 tarihinde şunu yazdı:

…

No I have the access (the last one was the right move). In your notebook you didn't checkout to the commit I mentioned. How did you do it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO4IQ2ROGFEABVGHTLX7KCDSQOARPANCNFSM4TMUBXOQ> .

[zaccharieramzi]

Yes but it's not about the checkpoints only it's about the version of the source code to run it.

[fharman]

Hi again, Ok, i see . Sorry for the confusion. I will run it inside that folder and python codes. I will send you the results. Thank you. best regards, Zaccharie Ramzi <[email protected]>, 18 Kas 2020 Çar, 12:15 tarihinde şunu yazdı:

…

Yes but it's not about the checkpoints only it's about the version of the source code to run it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO4IQ2WZSSCHMDS7FB2G3JLSQOGDPANCNFSM4TMUBXOQ> .

[zaccharieramzi]

In light of your answer in the other issue, can we close this @fharman ?

[fharman]

Yes Zaccharie , Thank you very much for support. Best, 19 Kas 2020 Per 18:35 tarihinde Zaccharie Ramzi <[email protected]> şunu yazdı:

…

In light of your answer in the other issue <#95 (comment)>, can we close this @fharman <https://github.com/fharman> ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO4IQ2VU7LURYDZS67V2KELSQU3MZANCNFSM4TMUBXOQ> .