Automated and Secure System for Machine Learning Lifecycles

Abstract:

With the advances in big data management and artificial intelligence, machine learning-based data analytics has become a crucial process to obtain insights for decision-making. Throughout its lifecycle, three phases, namely exploration, transition, and deployment, are central among the various stages. Each phase is characterized by complex and flexible workflows, yet existing tools and techniques provide limited support for automation and security, resulting in large overheads of manual work. To address these challenges, this thesis proposes a novel system called ASCENT, which stands for Automated and Secure maChine lEarning pipeliNe managemenT, that increases the degree of automation and security of machine learning lifecycles with comparable or better quality of their outcome. Built with the mind of the machine learning pipeline and its evolution, where a pipeline consists of components that flow the raw data to the final model outcome, the system comprises three main modules, covering the three central phases in a lifecycle.

The first module of ASCENT, namely AExplore, targets the early exploration phase by the data scientists. It enhances the automation of this phase by designing an efficient framework to recommend high-quality data preparation pipelines with standard components. Existing works on this problem are limited in either pipeline quality or runtime overhead, due to the large and complicated search space of component combinations. AExplore leverages contextual information to tackle these challenges. It leverages pre-trained embedding models to capture the data semantics, which are then used to guide the selection of pipeline components. After realizing the proposed method with deep reinforcement learning, we evaluate the module against state-of-the-art automated pipeline synthesis solutions. Results on commonly adopted test datasets show that AExplore outperforms all the baselines in both downstream model accuracy and runtime cost. To support user customization, we let AExplore output computational notebooks with executable pipeline code.

The second module of ASCENT, namely AExtract, targets the transition phase from exploration to production, a joint effort of data scientists and engineers. In particular, data scientists commonly use computational notebooks for customized data exploration, while engineers develop deployable pipelines for production management systems. However, the divergence in these two scenarios results in costly communication and labor during the migration of the underlying pipelines in the notebooks. AExtract enhances the automation of this phase by intelligently extracting pipelines from exploratory notebooks. It parses the notebook and identifies the correct execution order, extracts a series of components based on the functional boundaries of data operations, and builds the final pipeline by analyzing data flows. We develop a novel notebook semantic segmentation mechanism with a tree-based encoding-decoding method to extract components from the execution order of notebook code. Quantitative experimental results and user studies show that AExtract can generate high-quality pipelines while incurring low runtime overhead.

The third module of ASCENT, namely SEvolve, targets the final deployment phase of the lifecycle, focusing on the untrusted outsourcing issue of existing machine learning pipeline management systems. To tackle this, we adopt trusted hardware technology and carefully design the module architecture to efficiently protect the integrity and confidentiality of the pipeline management features. Moreover, we identify the cold start overhead as the main bottleneck to latency in fast-evolving pipelines. This overhead is obvious when naively isolating the full pipeline execution in one trusted enclave. We propose a strategy to administer enclaves and trusted runtimes, caching and reusing them for different pipeline components to avoid the initialization cost. After implementing SEvolve on the latest version of Intel Software Guard Extension, evaluation shows that the module greatly reduces the total execution time compared to not reusing, is more efficient than the single-enclave strategy, and incurs a modest performance overhead over insecure baselines.

In the end, we conclude ASCENT with its implementation, and provide several future research directions.