Deep Learning to Study the Brain to Improve...Deep Learning

Mar 13, 2018

Author Bio Image

Casimir Wierzynski

Senior Director, Office of the CTO, Artificial Intelligence Products Group

The interplay between AI and neuroscience is one of my favorite topics, and one that several of us on the AI team at Intel have personally worked on. I’m excited to share news about a new collaboration between Intel and MIT research labs designed to advance both of these fields. Before I do, I’d like to set the scene:

The relationship between neural networks and the brain has been a fascinating dialogue over the last fifty years. One side of this dialogue—the flow of knowledge from neuroscience to AI—is well known. The machine learning community has imported several concepts from neuroscience, starting, of course, with the (simplified) neuron, with linear inputs and a non-linear output, as an atomic unit of computation. Another idea from neuroscience is the arrangement of neurons into layers, analogous to the cytoarchitecture of mammalian neocortex. Recently, even more features of the brain have been transplanted into machine learning models. Some of these have been deliberate, such as the use of experience replay buffers for reinforcement learning, inspired by the observation that sequences of pyramidal cell activity are replayed in the hippocampus during sleep and periods of inactivity. Other features have appeared due to resource constraints, such as reduced bit-widths for weights, analogous to the limited precision of biological synapses. (This could be seen as an example of convergent evolution between brains and computers.)

But the other side of the dialogue, where machine learning informs neuroscience, is less well known and equally riveting. Much of this activity has been in the use of connectionist models that are complex enough to imitate features of brain activity and behavior and yet still mathematically tractable. These models provide an intellectual framework around experimental observations, make testable predictions, and suggest new experiments. One example is the Hopfield model of associative memory formation, which combines the recurrent (loopy) connectivity patterns found in the hippocampus with Hebbian learning (“neurons that fire together, wire together”). Two simple equations suffice to replicate phenomena such as auto-associative memory (using part of a memory to recall the rest), false memories, and attractor patterns found in the activity of hippocampal place cells. Another is Olshausen’s model of neural responses in the visual cortex, which combines the variational principle of sparsity with the concept of overcompleteness from linear algebra to explain why edge detectors (and not, say, Fourier bases) are learned in the brain from the statistics of natural images.

This brings us to our new project with MIT. An exciting thread in the AI-neuroscience dialogue has recently emerged where deep learning is serving neuroscience in a new way—namely as a tool for analyzing vast quantities of neuroscience data. In short, researchers are using brain-inspired algorithms to study the brain that inspired them. One example is in the field of connectomics, a recent effort to map large volumes of brain at nanometer scale. At this scale, neurons are huge, and one can even trace all of their input and output connections—that is, their axons, dendrites, and synapses. Assembling these connectomes—3D wiring diagrams of the brain—requires three major steps: (1) create a series of brain slices around 30 nanometers thick; (2) image this stack with an electron microscope; and (3) extract contours of neurons from the images and fuse them across the slices into a 3D reconstruction—a tour de force of experimental and analytical skill. It is also a big data problem; at this resolution, the neocortex of a rat would generate about an exabyte of image data.

We have just started collaborating with Prof. Nir Shavit’s lab at MIT CSAIL, which has been a leader in the area of computational connectomics, the analytics required to achieve step 3. His lab has pioneered ways of accelerating a novel deep learning-based image processing pipeline through clever use of multi-threaded programming and vector instructions on Intel® Xeon® architectures. We are excited to help him develop his next-gen processing pipeline.

What will we researchers learn from these studies? How will revealing the complete wiring diagram of the brain (or brain regions) help us design better neural networks? One possibility is in the area of novel architectures. Many significant performance jumps in neural networks have been due to innovations in connectivity patterns, such as inception modules and skip connections. Once researchers gain access to the full wiring diagram of the brain, and how it varies across brain regions and individuals, it is possible that new connectivity motifs will become apparent and inspire even more productive architectural variety in engineered systems. There are some caveats, of course: the connectome is only part of the story of how the brain computes, since a static wiring diagram doesn’t easily show how connections change with time and experience. Nevertheless, given how far deep learning has progressed so far using only a few key concepts from neuroscience, I look forward to the imminent explosion of neuroscience discovery that will hopefully lead us to vastly more powerful learning systems.

Image credit: Mouse nerve bundle, Y. Meirovitch and K. Kang from N. Shavit and J. Lichtman labs.

Author Bio Image

Casimir Wierzynski

Senior Director, Office of the CTO, Artificial Intelligence Products Group

Related Blog Posts

Using Intel® Xeon® for Multi-node Scaling of TensorFlow* with Horovod*

TensorFlow* is one of the leading deep learning and machine learning frameworks today. Earlier in 2017, Intel worked with Google* to incorporate optimizations for Intel® Xeon® processor-based platforms using Intel® Math Kernel Library (Intel® MKL) [1].  Optimizations such as these with multiple popular frameworks have led to orders of magnitude improvement in performance. Intel has…

Read more

#Technical Blog

MLT: The Keras of Kubernetes*

Running distributed machine learning workloads has been a hot topic lately.  Intel has shared documents walk through the process of using Kubeflow* to run distributed TensorFlow* jobs with Kubernetes, as well as a blog on using the Volume Controller for Kubernetes (KVC) for data management on clusters, and a blog describing a real-world use case…

Read more

#Technical Blog #Tutorial

Analyzing and Understanding Visual Data

Currently, more than 75% of all internet traffic is visual (video/images). Total traffic is exploding, projected to jump from 1.2 zettabytes per year in 2016, to 3.3 zettabytes in 2021, and visual data will comprise roughly 2.6 zettabytes of that. A major challenge for applications is how to process and understand this visual data, a…

Read more

#Research #Technical Blog