Just to clarify, the paper [0] does use both implanted electrodes and fMRI data, but it is actually quite transparent about which data came from which source. The authors worked with two datasets: the B2G dataset, which includes multi-unit activity from implanted Utah arrays in macaques, and the Shen-19 dataset, which uses noninvasive fMRI from human participants.
You’re right that fMRI measures blood flow rather than direct neural activity, and the authors acknowledge that limitation. But the study doesn’t treat it as a direct window into brain function. Instead, it proposes a predictive attention mechanism (PAM) that learns to selectively weigh signals from different brain areas, depending on the task of reconstructing perceived images from those signals.
The “thermal imager” analogy might make sense in a different context, but in this case, the model is explicitly designed to deal with those signal differences and works across both modalities. If you’re curious, the paper is available here:
Fair point. Side-channel attacks show how much signal you can pull from noise. But fMRI is a different kind of beast. It’s slow, indirect, and coarse. You’re not measuring neural activity directly, just blood flow changes that lag by a few seconds.
The paper [0] doesn’t pretend otherwise. It trains a model (PAM) to learn which brain regions carry useful info for reconstructing images, and applies this to both fMRI data from humans and intracranial recordings from macaques. The two signal types are handled separately.
If you want an analogy, it’s less like tapping power lines and more like trying to figure out which YouTube video someone is watching by measuring heat on the back of their laptop every few seconds. There’s a pattern in there, but pulling it out takes work.
You’re right that fMRI measures blood flow rather than direct neural activity, and the authors acknowledge that limitation. But the study doesn’t treat it as a direct window into brain function. Instead, it proposes a predictive attention mechanism (PAM) that learns to selectively weigh signals from different brain areas, depending on the task of reconstructing perceived images from those signals.
The “thermal imager” analogy might make sense in a different context, but in this case, the model is explicitly designed to deal with those signal differences and works across both modalities. If you’re curious, the paper is available here:
[0] https://www.biorxiv.org/content/10.1101/2024.06.04.596589v2....