Rumble: Data independence for large, messy datasets
Abstract:
We introduce Rumble, a query execution engine for large, heterogeneous, and nested collections of JSON objects built on top of Apache Spark.
While data sets of this type are more and more wide-spread, most existing tools are built around a tabular data model, creating an impedance mismatch for both the engine and the query interface. In contrast, Rumble uses JSONiq, a standardized language specifically designed for querying JSON documents. The key challenge in the design and implementation of Rumble is mapping the recursive structure of JSON documents and JSONiq queries onto Spark's execution primitives based on tabular data frames. Our solution is to translate a JSONiq expression into a tree of iterators that dynamically switch between local and distributed execution modes depending on the nesting level. By overcoming the impedance mismatch in the engine, Rumble frees the user from solving the same problem for every single query, thus increasing their productivity considerably. As we show in extensive experiments, Rumble is able to scale to large and complex data sets in the terabyte range with a similar or better performance than other engines. The results also illustrate that Codd's concept of data independence makes as much sense for heterogeneous, nested data sets as it does on highly structured tables.
Biodata:
Ghislain Fourny is a senior scientist at ETH Zurich with a focus on databases and game theory. He holds a Master of Science in Computer Science and a Doctorate of Science from ETH Zurich. Ghislain teaches Big Data courses for computer scientists as well as non-computer scientists.
His research interests cover query languages for large-scale, heterogeneous, nested datasets, as well as rebooting game theory with a non-Nashian form of free choice. Ghislain was a member of the W3C XML Query working group from 2011 to 2014 and is a co-designer of the JSONiq query language and of the Rumble engine.