Nessie Logo

About Nessie

Nessie is a transactional catalog for data lakes that brings Git-like version control semantics to data. It allows users to manage data in data lakes with concepts like commits, branches, and tags, similar to how source code is managed. This enables an always-consistent view of data across all involved datasets and tables, ensuring that changes from batch jobs or experiments are isolated and applied atomically. Nessie eliminates the need for manual tracking of individual data files by referencing existing immutable data files and automatically managing their lifecycle, including garbage collection. This tool is designed for data engineers and organizations working with large data lakes, especially those using tools like Apache Hive or Apache Spark. It simplifies data management by providing a robust versioning system that prevents incomplete changes from being visible to users and allows for safe experimentation and development without impacting production data. Nessie does not copy data but rather tracks metadata, making it efficient for managing vast amounts of data and numerous tables.