Poster
in
Workshop: UniReps: Unifying Representations in Neural Models
Language decoding from human brain activity via contrastive learning
Matteo Ferrante · Nicola Toschi · Alexander Huth
Keywords: [ neural representation of language ] [ brain decoding ] [ sentence identification ] [ fMRI ] [ contrastive learning ] [ language ]
We propose a novel contrastive learning approach to decode brain activity into sentences by mapping fMRI recordings and text embeddings into a shared representational space. Using data from three subjects, we trained a cross-subject fMRI encoder and demonstrated effective sentence identification with a retrieval module. Our model shows strong alignment between brain activity and linguistic features, with top-1 accuracy up to 49.2\% and top-10 accuracy up to 84\%, significantly outperforming chance levels. These results highlight the potential of contrastive learning for cross-subject language decoding,