ME2 Synthetic Training

[reesekneeland]

Use GNet to encode the NSD training images into predicted patterns of brain activity, then use these synthetic patterns to train ME2 in its default configuration, and test inference on NSD-Imagery.

The point of this task is to see if we can expand the distribution of training samples in the ME2 model by creating synthetic brain data to train it with. A 100% synthetic model will be almost certainly be worse than the original, but is a first step towards our actual goal, which is incorporating synthetic data as part of the training dataset.

Future steps will be replacing smaller and smaller fractions of the training dataset with synthetic data to see how much we can get away with.

To use GNet, see the final_evaluations.ipynb file section that calculates brain correlation scores. The code to perform inference with the GNet model is already there in the MindEye repo, and the weights are on the ME2 huggingface.