{"id":13487225,"url":"https://github.com/MarquezProject/marquez","last_synced_at":"2025-03-27T21:32:15.506Z","repository":{"id":37390955,"uuid":"139907613","full_name":"MarquezProject/marquez","owner":"MarquezProject","description":"Collect, aggregate, and visualize a data ecosystem's metadata","archived":false,"fork":false,"pushed_at":"2024-10-28T22:50:13.000Z","size":53848,"stargazers_count":1770,"open_issues_count":182,"forks_count":316,"subscribers_count":47,"default_branch":"main","last_synced_at":"2024-10-29T15:34:09.124Z","etag":null,"topics":["data-dictionary","data-discovery","data-ecosystem-metadata","data-governance","data-lineage","data-ops","data-provenance","marquez","metadata","metadata-service"],"latest_commit_sha":null,"homepage":"https://marquezproject.ai","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MarquezProject.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":"GOVERNANCE.md","roadmap":null,"authors":null,"dei":null}},"created_at":"2018-07-05T22:43:20.000Z","updated_at":"2024-10-29T07:00:48.000Z","dependencies_parsed_at":"2024-03-05T07:46:35.924Z","dependency_job_id":"3789081c-6a85-4f63-afa0-3a322bb62ef7","html_url":"https://github.com/MarquezProject/marquez","commit_stats":null,"previous_names":[],"tags_count":88,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarquezProject%2Fmarquez","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarquezProject%2Fmarquez/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarquezProject%2Fmarquez/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarquezProject%2Fmarquez/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MarquezProject","download_url":"https://codeload.github.com/MarquezProject/marquez/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222321853,"owners_count":16966433,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-dictionary","data-discovery","data-ecosystem-metadata","data-governance","data-lineage","data-ops","data-provenance","marquez","metadata","metadata-service"],"created_at":"2024-07-31T18:00:56.842Z","updated_at":"2025-03-27T21:32:15.495Z","avatar_url":"https://github.com/MarquezProject.png","language":"Java","readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./docs/assets/images/marquez-logo.png\" width=\"500px\" /\u003e\n  \u003ca href=\"https://lfaidata.foundation/projects\"\u003e\n    \u003cimg src=\"./docs/assets/images/lfaidata-project-badge-graduate-black.png\" width=\"125px\" /\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\nMarquez is an open source **metadata service** for the **collection**, **aggregation**, and **visualization** of a data ecosystem's metadata. It maintains the provenance of how datasets are consumed and produced, provides global visibility into job runtime and frequency of dataset access, centralization of dataset lifecycle management, and much more. Marquez was released and open sourced by [WeWork](https://www.wework.com).\n\n## Badges\n\n[![CircleCI](https://circleci.com/gh/MarquezProject/marquez/tree/main.svg?style=shield)](https://circleci.com/gh/MarquezProject/marquez/tree/main)\n[![codecov](https://codecov.io/gh/MarquezProject/marquez/branch/main/graph/badge.svg)](https://codecov.io/gh/MarquezProject/marquez/branch/main)\n[![status](https://img.shields.io/badge/status-active-brightgreen.svg)](#status)\n[![Slack](https://img.shields.io/badge/slack-chat-blue.svg)](https://join.slack.com/t/marquezproject/shared_invite/zt-2iylxasbq-GG_zXNcJdNrhC9uUMr3B7A)\n[![license](https://img.shields.io/badge/license-Apache_2.0-blue.svg)](https://raw.githubusercontent.com/MarquezProject/marquez/main/LICENSE)\n[![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-v2.0%20adopted-ff69b4.svg)](CODE_OF_CONDUCT.md)\n[![maven](https://img.shields.io/maven-central/v/io.github.marquezproject/marquez-api.svg)](https://search.maven.org/search?q=g:io.github.marquezproject)\n[![docker](https://img.shields.io/badge/docker-hub-blue.svg?style=flat)](https://hub.docker.com/r/marquezproject/marquez)\n[![Known Vulnerabilities](https://snyk.io/test/github/MarquezProject/marquez/badge.svg)](https://snyk.io/test/github/MarquezProject/marquez)\n[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/5160/badge)](https://bestpractices.coreinfrastructure.org/projects/5160)\n\n## Status\n\nMarquez is an [LF AI \u0026 Data Foundation](https://lfaidata.foundation/projects/marquez) Graduated project under active development, and we'd love your help!\n\n## Adopters\n\nWant to be added? Send a pull request our way!\n\n* [Astronomer](https://astronomer.io)\n* [Datakin](https://datakin.com)\n* [Northwestern Mutual](https://www.northwesternmutual.com)\n* [Ilum](https://ilum.cloud)\n* [CsvPath](https://www.csvpath.org)\n\n## Try it!\n\n[![Open in Gitpod](https://gitpod.io/button/open-in-gitpod.svg)](https://gitpod.io/#https://github.com/MarquezProject/marquez)\n\n## Quickstart\n\nMarquez provides a simple way to collect and view _dataset_, _job_, and _run_ metadata using [OpenLineage](https://openlineage.io). The easiest way to get up and running is with Docker. From the base of the Marquez repository, run:\n\n### MacOS and Linux users:\n\n```bash\n$ ./docker/up.sh\n```\n\n### Windows users:\n\nBefore cloning Marquez, configure Git to check out files with Unix-style file endings:\n\n```bash\n$ git config --global core.autocrlf false\n```\n\nVerify that Bash and PostgreSQL have been installed and added to the PATH variable (Git Bash is recommended).\n\nStart all services:\n\n```bash\n$ sh ./docker/up.sh\n```\n\n\u003e **Tip:** Use the `--build` flag to build images from source, and/or `--seed` to start Marquez with sample lineage metadata. For a more complete example using the sample metadata, please follow our [quickstart](https://marquezproject.github.io/marquez/quickstart.html) guide.\n\n\u003e **Note:** Port 5000 is now reserved for MacOS. If running locally on MacOS, you can run `./docker/up.sh --api-port 9000` to configure the API to listen on port 9000 instead. Keep in mind that you will need to update the URLs below with the appropriate port number.\n\n**`WEB UI`**\n\nYou can open [http://localhost:3000](http://localhost:3000) to begin exploring the Marquez Web UI. The UI enables you to discover dependencies between jobs and the datasets they produce and consume via the lineage graph, view run metadata of current and previous job runs, and much more!\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./web/docs/demo.gif\"\u003e\n\u003c/p\u003e\n\n**`HTTP API`**\n\nThe Marquez [HTTP API](https://marquezproject.github.io/marquez/openapi.html) listens on port `5000` for all calls and port `5001` for the admin interface. The admin interface exposes helpful endpoints like `/healthcheck` and `/metrics`. To verify the HTTP API server is running and listening on `localhost`, browse to [http://localhost:5001](http://localhost:5001). To begin collecting lineage metadata as OpenLineage events, use the [LineageAPI](https://marquezproject.github.io/marquez/openapi.html#tag/Lineage/paths/~1lineage/post) or an OpenLineage [integration](https://openlineage.io/docs/integrations/about).\n\n\u003e **Note:** By default, the HTTP API does not require any form of authentication or authorization.\n\n**`GRAPHQL`**\n\nTo explore metadata via graphql, browse to [http://localhost:5000/graphql-playground](http://localhost:5000/graphql-playground). The graphql endpoint is currently in _beta_ and is located at [http://localhost:5000/api/v1-beta/graphql](http://localhost:5000/api/v1-beta/graphql).\n\n## Documentation\n\nWe invite everyone to help us improve and keep documentation up to date. Documentation is maintained in this repository and can be found under [`docs/`](https://github.com/MarquezProject/marquez/tree/main/docs).\n\n\u003e **Note:** To begin collecting metadata with Marquez, follow our [quickstart](https://marquezproject.github.io/marquez/quickstart.html) guide. Below you will find the steps to get up and running from source.\n\n## Versions and OpenLineage Compatibility\n\nVersions of Marquez are compatible with OpenLineage unless noted otherwise. We ensure backward compatibility with a newer version of Marquez by recording events with an older OpenLineage specification version. **We strongly recommend understanding how the OpenLineage specification is** [versioned](https://github.com/OpenLineage/OpenLineage/blob/main/spec/Versioning.md) **and published**.\n\n| **Marquez**                                                                                      | **OpenLineage**                                               | **Status**    |\n|--------------------------------------------------------------------------------------------------|---------------------------------------------------------------|---------------|\n| [`UNRELEASED`](https://github.com/MarquezProject/marquez/blob/main/CHANGELOG.md#unreleased)      | [`2-0-2`](https://openlineage.io/spec/2-0-2/OpenLineage.json) | `CURRENT`     |\n| [`0.50.0`](https://github.com/MarquezProject/marquez/blob/main/CHANGELOG.md#0500---2024-10-23)   | [`2-0-2`](https://openlineage.io/spec/2-0-2/OpenLineage.json) | `RECOMMENDED` |\n| [`0.49.0`](https://github.com/MarquezProject/marquez/blob/0.49.0/CHANGELOG.md#0490---2024-08-07) | [`2-0-2`](https://openlineage.io/spec/2-0-2/OpenLineage.json) | `MAINTENANCE` |\n\n\u003e **Note:** The [`openlineage-python`](https://pypi.org/project/openlineage-python) and [`openlineage-java`](https://central.sonatype.com/artifact/io.openlineage/openlineage-java) libraries will a higher version than the OpenLineage [specification](https://github.com/OpenLineage/OpenLineage/tree/main/spec) as they have different version requirements.\n\nWe currently maintain three categories of compatibility: `CURRENT`, `RECOMMENDED`, and `MAINTENANCE`. When a new version of Marquez is released, it's marked as `RECOMMENDED`, while the previous version enters `MAINTENANCE` mode (which gets bug fixes whenever possible). The unreleased version of Marquez is marked `CURRENT` and does not come with any guarantees, but is assumed to remain compatible with OpenLineage, although surprises happen and there maybe rare exceptions.\n\n## Modules\n\nMarquez uses a _multi_-project structure and contains the following modules:\n\n* [`api`](https://github.com/MarquezProject/marquez/tree/main/api): core API used to collect metadata\n* [`web`](https://github.com/MarquezProject/marquez/tree/main/web): web UI used to view metadata\n* [`clients`](https://github.com/MarquezProject/marquez/tree/main/clients): clients that implement the HTTP [API](https://marquezproject.github.io/marquez/openapi.html)\n* [`chart`](https://github.com/MarquezProject/marquez/tree/main/chart): helm chart\n\n\u003e **Note:** The `integrations` module was removed in [`0.21.0`](https://github.com/MarquezProject/marquez/blob/main/CHANGELOG.md#removed), so please use an OpenLineage [integration](https://openlineage.io/integration) to collect lineage events easily.\n\n## Requirements\n\n* [Java 17](https://adoptium.net)\n* [PostgreSQL 14](https://www.postgresql.org/download)\n\n\u003e **Note:** To connect to your running PostgreSQL instance, you will need the standard [`psql`](https://www.postgresql.org/docs/9.6/app-psql.html) tool.\n\n## Building\n\nTo build the entire project run:\n\n```bash\n./gradlew build\n```\n\nThe executable can be found under `api/build/libs/`\n\n## Configuration\n\nTo run Marquez, you will have to define `marquez.yml`. The configuration file is passed to the application and used to specify your database connection. The configuration file creation steps are outlined below.\n\n### Step 1: Create Database\n\nWhen creating your database using [`createdb`](https://www.postgresql.org/docs/12/app-createdb.html), we recommend calling it `marquez`:\n\n```bash\n$ createdb marquez\n```\n\n### Step 2: Create `marquez.yml`\n\nWith your database created, you can now copy [`marquez.example.yml`](https://github.com/MarquezProject/marquez/blob/main/marquez.example.yml):\n\n```bash\n$ cp marquez.example.yml marquez.yml\n```\n\nYou will then need to set the following environment variables (we recommend adding them to your `.bashrc`): `POSTGRES_DB`, `POSTGRES_USER`, and `POSTGRES_PASSWORD`. The environment variables override the equivalent option in the configuration file.\n\nBy default, Marquez uses the following ports:\n\n* TCP port `8080` is available for the HTTP API server.\n* TCP port `8081` is available for the admin interface.\n\n\u003e **Note:** All of the configuration settings in `marquez.yml` can be specified either in the configuration file or in an environment variable.\n\n## Running the [HTTP API](https://github.com/MarquezProject/marquez/blob/main/src/main/java/marquez/MarquezApp.java) Server\n\n```bash\n$ ./gradlew :api:runShadow\n```\nMarquez listens on port `8080` for all API calls and port `8081` for the admin interface. To verify the HTTP API server is running and listening on `localhost`, browse to [http://localhost:8081](http://localhost:8081). We encourage you to familiarize yourself with the [data model](https://marquezproject.github.io/marquez/#data-model) and [APIs](https://marquezproject.github.io/marquez/openapi.html) of Marquez. To run the web UI, please follow the steps outlined [here](https://github.com/MarquezProject/marquez/tree/main/web#development).\n\n\u003e **Note:** By default, the HTTP API does not require any form of authentication or authorization.\n\n## Related Projects\n\n* [`OpenLineage`](https://github.com/OpenLineage/OpenLineage): an open standard for metadata and lineage collection\n\n## Getting Involved\n\n* Website: https://marquezproject.github.io/marquez/\n* Source: https://github.com/MarquezProject/marquez\n* Chat: [MarquezProject Slack](https://join.slack.com/t/marquezproject/shared_invite/zt-2iylxasbq-GG_zXNcJdNrhC9uUMr3B7A)\n* X: [@MarquezProject](https://twitter.com/MarquezProject)\n\n## Contributing\n\nSee [CONTRIBUTING.md](https://github.com/MarquezProject/marquez/blob/main/CONTRIBUTING.md) for more details about how to contribute.\n\n## Reporting a Vulnerability\n\nIf you discover a vulnerability in the project, please open an issue and attach the \"security\" label.\n\n----\nSPDX-License-Identifier: Apache-2.0\nCopyright 2018-2024 contributors to the Marquez project.\n","funding_links":[],"categories":["Java","Data Catalog","Data Management","others","Frameworks and Libraries","Metadata Management","其他_机器学习与深度学习","Uncategorized","GenAI Readiness Features","大数据","Data"],"sub_categories":["Commercial","Uncategorized","Data Quality, Observability \u0026 Governance","Catalog"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMarquezProject%2Fmarquez","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMarquezProject%2Fmarquez","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMarquezProject%2Fmarquez/lists"}