Getting NaN values when running nc_train.py

[SivaHitesh047]

Getting NaN values when running nc_train.py

[chaithyagr]

Can you please tell what is the exact reproducible code you ran?
Also, please let us know the exact environment you used (tensorflow version etc).

[chaithyagr]

To me, on seeing your output, it is quite bizarre to have that high loss initially... Are you using density compensation?
How did you set up the data?
From what I remember you need to convert the fastmri dataset into tfrecords to have preprocessed data (kspace data, smaps, density compensation, etc).

[SivaHitesh047]

I ran the nc_train.py from the training_scripts folder keeping all the parameters as it is. I used python -m nc_train command to run the file. I am not using density compensation.

The dataset I used for train set are the .h5 files.

Here are some of the module versions I used in the environment.

tfkbnufft - 0.2.4
tensorflow -2.8.0
tensorflow-gpu - 2.4.1
tensorflow-nufft - 0.7.3
scikit-image - 0.19.3
keras - 2.8.0
keras-preprocessing - 1.1.2

[chaithyagr]

Thank you for the info:

The dataset I used for train set are the .h5 files.

Do you mean from fastMRI dataset? I am afraid that is not right. See here how the dataset needs to be organized:

fastmri-reproducible-benchmark/fastmri_recon/models/utils/fourier.py

Lines 280 to 289 in 4a4ec09

	if self.multicoil:
	if self.density_compensation:
	kspace, ktraj, smaps, shape, dcomp, = inputs
	else:
	kspace, ktraj, smaps, shape = inputs
	else:
	if self.density_compensation:
	kspace, ktraj, shape, dcomp = inputs
	else:
	kspace, ktraj, shape = inputs

Ideally you will need to generate the tfrecords dataset using: https://github.com/zaccharieramzi/fastmri-reproducible-benchmark/blob/master/fastmri_recon/data/scripts/multicoil_nc_tf_records_generation.py
(Assuming you want to run multicoil in 2D)

I am not using density compensation.

I Highly recommend you use density compensation. You can have scaling issues otherwise which can affect the convergence. See the paper Sec III D where the importance of it, particularly for non-Cartesian MRI is discussed.
By default we do use density compensation.

[SivaHitesh047]

Hey, thank you so much chaithya! It's working now. I was trying out the single coil only.

[SivaHitesh047]

I was wondering about adding k-space network (dual layers) to the NCPDNet (similar to how it is in the PDNet). Can you please suggest me on what to do? Also, is there any reason why you didn’t implement dual layers in NCPDNet.

[chaithyagr]

Usually k-space doesnt exactly satisfy the same properties of equivariance in image domain and hence such a direct network would not exactly help the cause in my opinion. However, if u still want to do the same, see that NCPDNet is derived from CrossDomainNet, which is exactly network in k-space then in image space.

What you would need to do is to modify here:

fastmri-reproducible-benchmark/fastmri_recon/models/subclassed_models/ncpdnet.py

Lines 75 to 81 in 4a4ec09

	self.kspace_net = [measurements_residual for i in range(self.n_iter)]
	def measurements_residual(concatenated_kspace):
	current_kspace = concatenated_kspace[..., 0:1]
	original_kspace = concatenated_kspace[..., 1:2]
	return current_kspace - original_kspace

Note that currently we only have data consistency here. I would recommend you to have this, and in addition add your k-space network.
Do let us know if you see any improvements over the baseline, and after benchmarking, we would not mind adding it as a feature.

[zaccharieramzi]

I am going to close this issue since it seems resolved for now (at least the original nan problem).
Thanks @chaithyagr for handling it.