neuralcodinglab/brain2pix

Brain2Pix: Supplementary materials

Introduction

Welcome to the repository that contains supplementary materials and the source code for the paper "Brain2Pix: Fully convolutional naturalistic video reconstruction from brain activity".

The brain2pix model consists of 2 parts. 1) Making RFSimages, and 2) training the GAN-like model. For reproducing the experiment, first check out the data_preprocessing files for making the RFSimages and then the experiment files for training the model.

Folders

data_preprocessing: this folder entails all the steps of transforming raw brain signals into RFSimages.

experiment: codes containing the model and training loop for the experiments.

visualizations: reconstruction videos in GIF format and figures in PFD format.

Results

Main results -- FixedRF (test set):

fixed RFSimage | reconstruction | ground truth

Main results -- LearnedRF (test set):

learned RFSimage | reconstruction | ground truth

Additional results (test set):

Codes:

More information on the codes of the experiments in the README inside the "experiment" folder.

To replicate the main experiment, please see the "experiment/learnableRF" and "experiment/fixedRF" folders.

Datasets:

Dr. Who: The Dr. Who dataset is publicly available. The first author of the dataset paper (Seeliger et. al., 2019) mentioned (on http://disq.us/p/23bj45d) that the link will be activated soon. For now it is available by contacting them.

vim2: This dataset was taken from http://crcns.org/, originally published by Nishimoto et. al, 2011.

References:

Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. L. (2011). Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology, 21(19), 1641-1646.

Seeliger, K., et al. "A large single-participant fMRI dataset for probing brain responses to naturalistic stimuli in space and time." bioRxiv (2019): 687681.

Shen, G., Dwivedi, K., Majima, K., Horikawa, T., & Kamitani, Y. (2019). End-to-end deep image reconstruction from human brain activity. Frontiers in Computational Neuroscience, 13, 21.

brain2pix's Issues

Update the README file

It would be good to use readme-md-generator (https://github.com/kefranabg/readme-md-generator) to make sure that README styles of different projects are consistent with one another.