Marquez

Marquez is an open-source metadata server that provides real-time collection of information from running jobs and applications, acting as the reference implementation for OpenLineage. It offers a unified visual graph through a web user interface, allowing users to explore complex interdependencies within their data ecosystem, trace lineage, and analyze performance metrics. The platform is designed as a modular, highly scalable, and extensible solution for metadata management. It consists of a Metadata Repository for storing job and dataset metadata, a RESTful Metadata API for interaction, and a Metadata UI for discovery and dependency graph exploration. Marquez emphasizes an immutable data model, tracking versioned jobs and datasets to ensure reproducibility and provide powerful visualizations of data flow. It's ideal for data engineers, data scientists, and anyone managing complex data pipelines who needs to understand data provenance, automate tasks like backfills, and perform root cause analysis.