Machine programming (MP) is a new field of research that uses automation to improve software development productivity (e.g., the time it takes a developer to write code) and quality (e.g., performance, correctness, security, maintainability, etc.). We generally consider MP as a fusion of machine learning and formal methods, which rely heavily on programming languages and systems.
PublicationToward Scalable Verification for Safety-Critical Deep Networks(2018-01-01) Kuper, Lindsey; Gottschlich, Justin E; Gottschlich, Justin E; Julian, Kyle; Barrett, Clark; Kochenderfer, Mykel JThe increasing use of deep neural networks for safety-critical applications, such as autonomous driving and flight control, raises concerns about their safety and reliability. Formal verification can address these concerns by guaranteeing that a deep learning system operates as intended, but the state of the art is limited to small systems. In this work-in-progress report we give an overview of our work on mitigating this difficulty, by pursuing two complementary directions: devising scalable verification techniques, and identifying design choices that result in deep learning systems that are more amenable to verification. PublicationThe Three Pillars of Machine Programming(2018-01-01) Gottschlich, Justin E; Gottschlich, Justin E; Solar-Lezama, Armando; Tatbul, Nesime; Carbin, Michael; Rinard, Martin; Barzilay, Regina; Amarasinghe, Saman; Tenenbaum, Joshua B; Mattson, TimothyIn this position paper, we describe our vision of the future of machine programming through a categorical examination of three pillars of research. Those pillars are:(i) intention,(ii) invention, and (iii) adaptation. Intention emphasizes advancements in the human-to-computer and computer-to-machine-learning interfaces. Invention emphasizes the creation or refinement of algorithms or core hardware and software building blocks through machine learning (ML). Adaptation emphasizes advances in the use of ML-based constructs to autonomously evolve software. PublicationPrecision and Recall for Time Series(2018-01-01) Tatbul, Nesime; Lee, Tae J; Zdonik, Stan; Gottschlich, Justin E; Gottschlich, Justin EClassical anomaly detection is principally concerned with point-based anomalies, those anomalies that occur at a single point in time. Yet, many real-world anomalies are range-based, meaning they occur over a period of time. Motivated by this observation, we present a new mathematical model to evaluate the accuracy of time series classification algorithms. Our model expands the well-known Precision and Recall metrics to measure ranges, while simultaneously enabling customization support for domain-specific preferences. PublicationControlFlag: A Self-supervised Idiosyncratic Pattern Detection System for Software Control Structures(2020-01-01) Gottschlich, Justin E; Gottschlich, Justin ESoftware debugging has been shown to utilize upwards of 50% of developers’ time. Machine programming, the field concerned with the automation of software (and hardware) development, has recently made progress in both research and production-quality automated debugging systems. In this paper, we present ControlFlag, a system that detects possible idiosyncratic violations in software control structures. ControlFlag also suggests possible corrections in the event a true error is detected. A novelty of ControlFlag is that it is entirely self-supervised; that is, it requires no labels to learn about the potential idiosyncratic programming pattern violations. In addition to presenting ControlFlag’s design, we also provide an abbreviated experimental evaluation. PublicationSoftware Language Comprehension using a Program-Derived Semantics Graph(2020-01-01) Iyer, Roshni G; Sun, Yizhou; Gottschlich, Justin E; Gottschlich, Justin ETraditional code transformation structures, such as abstract syntax trees (ASTs), conteXtual flow graphs (XFGs), and more generally, compiler intermediate representations (IRs), may have limitations in extracting higher-order semantics from code. While work has already begun on higher-order semantics lifting (e.g., Aroma’s simplified parse tree (SPT), verified lifting’s lambda calculi, and Halide’s intentional domain specific language (DSL)), research in this area is still immature. To continue to advance this research, we present the program-derived semantics graph (PSG), a new graphical structure to capture semantics of code. The PSG is designed to provide a single structure for capturing program semantics at multiple levels of abstraction. The PSG may be in a class of emerging structural representations that cannot be built from a traditional set of predefined rules and instead must be learned. In this paper, we describe the PSG and its fundamental structural differences compared to state-of-the-art structures. Although our exploration into the PSG is in its infancy, our early results and architectural analysis indicate it is a promising new research direction to automatically extract program semantics. PublicationMLSys: The New Frontier of Machine Learning Systems(2019-01-01) Ratner, Alexander; Alistarh, Dan; Alons, Gustavo; Andersen, David G; Bailis, Peter; Bird, Sarah; Carlini, Nicholas; Catanzaro, Bryan; Chayes, Jennifer; Chung, Eric; Dally, Bill; Dean, Jeff; Dhillon, Inderjit S; Dimakis, Alexandros; Dubey, Pradeep; Elkan, Charles; Fursin, Grigori; Ganger, Gregory R; Getoor, Lise; Gibbons, Phillip B; Gibson, Garth A; Gottschlich, Justin E; Gottschlich, Justin E; Han, Song; Hazelwood, Kim; Huang, Furong; Jaggi, Martin; Jamieson, Kevin; Jordan, Michael I; Joshi, Gauri; Khalaf, Rania; Knight, Jason; Konecny, Jakub; Kraska, Tim; Kumar, Arun; Kyrillidis, Anastasios; Lakshmiratan, Aparna; Li, Jing; Madden, Samuel; McMahan, H B; Meijer, Erik; Mitliagkas, Ioannis; Monga, Rajat; Murray, Derek; Olukotun, Kunle; Papailiopoulos, Dimitris; Pekhimenko, Gennady; Re, Christopher; Rekatsinas, Theodoros; Rostamizadeh, Afshin; De Sa, Christopher; Sedghi, Hanie; Sen, Siddhartha; Smith, Virginia; Smola, Alex; Song, Dawn; Sparks, Evan; Stoica, Ion; Sze, Vivienne; Udell, Madeleine; Vanschoren, Joaquin; Venkataraman, Shivaram; Vinayak, Rashmi; Weimer, Markus; Wilson, Andrew G; Xing, Eric; Zaharia, Matei; Zhang, Ce; Talwalkar, AmeetMachine learning (ML) techniques are enjoying rapidly increasing adoption. However, designing and implementing the systems that support ML models in real-world deployments remains a significant obstacle, in large part due to the radically different development and deployment profile of modern ML methods, and the range of practical concerns that come with broader adoption. We propose to foster a new systems machine learning research community at the intersection of the traditional systems and ML communities, focused on topics such as hardware systems for ML, software systems for ML, and ML optimized for metrics beyond predictive accuracy. To do this, we describe a new conference, MLSys, that explicitly targets research at the intersection of systems and machine learning with a program committee split evenly between experts in systems and ML, and an explicit focus on topics at the intersection of the two. PublicationGreenhouse: A Zero-Positive Machine Learning System for Time-Series Anomaly Detection(2018-01-01) Gottschlich, Justin E; Gottschlich, Justin E; Tatbul, Nesime; Metcalf, Eric; Zdonik, StanThis short paper describes our ongoing research on Greenhouse - a zero-positive machine learning system for time-series anomaly detection. PublicationLearned Garbage Collection(2020-01-01) Cen, Lujing; Marcus, Ryan; Gottschlich, Justin E; Gottschlich, Justin E; Alizadeh, Mohammad; Kraska, TimSeveral programming languages use garbage collectors (GCs) to automatically manage memory for the programmer. Such collectors must decide when to look for unreachable objects to free, which can have a large performance impact on some applications. In this preliminary work, we propose a design for a learned garbage collector that autonomously learns over time when to perform collections. By using reinforcement learning, our design can incorporate user-defined reward functions, allowing an autonomous garbage collector to learn to optimize the exact metric the user desires (e.g., request latency or queries per second). We conduct an initial experimental study on a prototype, demonstrating that an approach based on tabular Q learning may be promising. PublicationA Zero-Positive Learning Approach for Diagnosing Software Performance Regressions(2019-01-01) Gottschlich, Justin E; Gottschlich, Justin E; Tatbul, Nesime; Turek, Javier S; Mattson, Timothy; Muzahid, AbdullahThe field of machine programming (MP), the automation of the development of software, is making notable research advances. This is, in part, due to the emergence of a wide range of novel techniques in machine learning. In this paper, we apply MP to the automation of software performance regression testing. A performance regression is a software performance degradation caused by a code change. We present AutoPerf–a novel approach to automate regression testing that utilizes three core techniques:(i) zero-positive learning,(ii) autoencoders, and (iii) hardware telemetry. We demonstrate AutoPerf’s generality and efficacy against 3 types of performance regressions across 10 real performance bugs in 7 benchmark and open-source programs. On average, AutoPerf exhibits 4% profiling overhead and accurately diagnoses more performance bugs than prior state-of-the-art approaches. Thus far, AutoPerf has produced no false negatives.