Succeeded to image images in the head from brain activity data using machine learning

Although you may see scenes in the science fiction novels and movies in which code is inserted in the head and memory and images are output as data from the brain,fMRI (Functional Nuclear Magnetic Resonance)By analyzing human brain activity visualized by machine learning, it may be possible to actually visualize the image in the brain, with researchers at Kyoto UniversityATR (International Telecommunications Research Institute, Inc.)It has been proposed by researchers. In the following papers, the image actually imagined to the head isMachine learningIt has been reported that it succeeded in outputting to the image by using.

Deep image reconstruction from human brain activity | bioRxiv
https://www.biorxiv.org/content/early/2017/12/30/240317

Human visual information is the "cerebrum occipital lobe"Visual cortexAt the site calledProcessed hierarchicallyI know that. Therefore, the part where the nerve activity is active in visual cortex is measured with fMRI and the obtained signal data is DNN (Deep Neural Network) And input it. This experiment is that DNN predicted a rough image from the input data and optimized the predicted image with DGN (depth generation network) and imaged it.


First of all, while showing the photographs of living things and artifacts to the subject, measure the activity of the brain in real time and reconstruct the image from there. The image below shows the images (the top row) actually viewed and the images (the bottom three rows) output from each of the three subjects. Though there are rough differences in the three subjects, when looking at the same image roughly the same image is outputted.


Also, the image optimized by DGN (with DGN) was a much clearer depiction than the image not optimized (No DGN), and the coincidence rate between the original image and the output image was also high.


In the following movie, the picture actually seen on the left is shown on the right, and the image is shown optimized by outputting from the activity of the brain of the subject to the right.

Deep image reconstruction: Natural images - YouTube


Experiments have been done to recognize shapes as well as colors and compositions. The following movie is the result, the image displayed on the left is the figure actually seen by the subject, the image displayed on the right is outputted and optimized.

Deep image reconstruction: Geometric shapes - YouTube


Although figures are somewhat fuzzy, it is clear what kind of figure the original is. In particular, as shown in the lower part of the image below, the alphabet shows a more clearly shaped shape than the figure, and the matching ratio between the original image and the image also exceeds 90%.


Furthermore, experiments were also conducted to reconstruct the images imagined in the head by using the data so far as to whether it is possible to visually reconstruct the subjective contents. After showing the photographs and figures in advance to the subjects, we measure the activity of the brain while asking them to think in the head without looking at anything. In the following movies, the image displayed beforehand is shown on the left, while the image on the right shows how to optimize the image outputted from the brain activity when remembered.

Deep image reconstruction: Visual imagery - YouTube


Complex photographs such as birds and airplanes are not able to obtain sufficiently optimized data because it is difficult to recall the concrete image. In simple figures, it is considerably dimpled compared with the case of measuring while looking at images, but the image of the original figure appears vaguely.


At the same time, the research team notes that the matching rate between the original image and the generated image is more judged by human eyes than judging by the program, and by combining the hierarchies of the plurality of DNNs, human subjective We are hoping that it will be possible to reconfigure the image more finely.