DataCockpit: A Toolkit for Data Lake Navigation and Monitoring Utilizing Quality and Usage Information
IEEE BigData (MLBD) · 2023
★ Patent
Abstract
Modern organizations amass their datasets into centralized repositories called data lakes, affording analytics as needed. The resultant scale and complexity of these data lakes, however, can make data navigation and monitoring challenging for users. We present DataCockpit, a Python toolkit that leverages datasets, usage logs, and associated meta-data to provision data usage and quality characteristics. DataCockpit computes these characteristics for each attribute (e.g., number of times it was queried for subsequent use in downstream applications) and record (e.g., number of non-missing, valid values) and aggregates them at the level of datasets. We develop a visual monitoring tool, powered by DataCockpit, and demonstrate how it can assist data / system administrators as well as end-users to effectively navigate and monitor a data lake. DataCockpit and the monitoring tool are available as open source software for developers to build custom monitoring applications on top of data lakes.
Citation
@inproceedings{narechania2023datacockpit,
author = {Narechania, Arpit and Chakraborty, Surya and Agarwal, Shivam and Sinha, Atanu R and Rossi, Ryan A. and Du, Fan and Hoffswell, Jane and Guo, Shunan and Koh, Eunyee and Endert, Alex and Navathe, Shamkant},
title = {{DataCockpit: A Toolkit for Data Lake Navigation and Monitoring Utilizing Quality and Usage Information}},
year = {2023},
volume={},
number={},
pages={5305-5310},
booktitle = {2023 IEEE International Conference on Big Data (BigData)},
doi = {https://doi.org/10.1109/BigData59044.2023.10386133},
url = {https://doi.org/10.1109/BigData59044.2023.10386133}
}