https://github.com/mitodl/ol-data-platform

Pipeline definitions for managing data flows to power analytics at MIT Open Learning
https://github.com/mitodl/ol-data-platform

Last synced: 6 days ago
JSON representation

Pipeline definitions for managing data flows to power analytics at MIT Open Learning

• Host: GitHub
• URL: https://github.com/mitodl/ol-data-platform
• Owner: mitodl
• License: bsd-3-clause
• Created: 2020-04-27T20:31:23.000Z (over 4 years ago)
• Default Branch: main
• Last Pushed: 2024-10-29T15:52:27.000Z (about 2 months ago)
• Last Synced: 2024-10-29T15:56:27.359Z (about 2 months ago)
• Language: Python
• Size: 10.5 MB
• Stars: 37
• Watchers: 20
• Forks: 6
• Open Issues: 23
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Overview

This repository holds the business logic for building and managing the data pipelines used to power various data

services at MIT Open Learning. The core framework is Dagster which provides a flexible, and well structured approach to

building data applications.

# Running Dagster Locally via Docker

- Ensure that you have the latest version of Docker installed.

	https://www.docker.com/products/docker-desktop/

- Install docker compose. Check the documentation and requirements for your specific machine.

	https://docs.docker.com/compose/install/

- Ensure you are able to authenticate into GitHub + Vault

	https://github.com/mitodl/ol-data-platform/tree/main

	https://vault-qa.odl.mit.edu/v1/auth/github/login

	`vault login -address=https://vault-qa.odl.mit.edu -method=github`

	https://vault-production.odl.mit.edu/v1/auth/github/login

	`vault login -address=https://vault-production.odl.mit.edu -method=github`

- Export

	`export GITHUB_TOKEN=[YOUR_GITHUB_TOKEN]`

- Call docker compose up

	`docker compose up --build`

- Navigate to localhost:3000 to access the Dagster UI