Marquez is a core service for the collection, aggregation, and visualization of a data ecosystem's metadata. It maintains the provenance of how datasets are consumed and produced, provides global visibility into job runtime and frequency of dataset access, centralization of dataset lifecycle management, and much more.
This project is under active development at WeWork and Stitch Fix (in collaboration with many others organizations).
The Marquez design is being actively updated and is open for comments.
- Java 8 or above
- PostgreSQL database
- Gradle 4.9 or above
To build the entire project run:
$ ./gradlew shadowJarThe executable can be found under build/libs/
To run Marquez, you will have to define config.yml (see reference). The configuration file is passed to the application and used to specify your database connection. When creating your database, we recommend calling it marquez. Please copy config.example.yml:
$ cp config.example.yml config.ymlYou will then need to edit it and specify your database information.
Note: As an optional step, you can manually run the database migration with:
$ ./gradlew run --args 'db migrate config.yml'Tip: You can access environment variables from config.yml with ${VAR_NAME}.
Running the Application
$ ./gradlew run --args 'server config.yml'Then browse to the admin interface: http://localhost:8081