Fast and Stable Algorithms for Deep Learning



Journal Title

Journal ISSN

Volume Title



Brain tumors are the leading cause of cancer-related deaths among multiple age groups. An important step in clinical management of patients diagnosed with tumors is tumor segmentation, which separates abnormal (cancerous) from normal tissues. In recent years, classification methods that employ deep learning were shown to yield excellent results for automatic tumor segmentation. In 2017, there has appeared interesting work that establishes a connection between the theory of differential equations, an area that is well explored in applied mathematics, and residual networks (a specific type of architecture in deep learning). In my summer research project, I explored this connection. My research project dealt with the development and implementation of efficient and stable methods for the forward propagation process in deep residual neural networks. The forward propagation maps features (e.g., brain images) to labels or classes (e.g., tumor vs. normal tissue). The forward propagation of a residual network can be interpreted as the numerical time integration of a differential equation. This allows us to rigorously assess stability issues seen in deep learning. My task was to implement and test different methods for the forward propagation. In particular, I implemented an explicit Euler time integration scheme and the Verlet method. I replicated test problems available in the literature to explore and understand the connection between time integration and forward propagation, and its implications on stability. My research program concludes with an exploration of more complex operators that appear in the learning problem (i.e., the estimation of weights used in the network).