{"id":17868268,"url":"https://github.com/matmoore/etl","last_synced_at":"2026-07-14T12:02:26.452Z","repository":{"id":137918505,"uuid":"319042472","full_name":"MatMoore/etl","owner":"MatMoore","description":"Data pipelines for rescuing my data from various web services","archived":false,"fork":false,"pushed_at":"2020-12-31T14:14:42.000Z","size":8,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-02T22:30:41.226Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MatMoore.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-06T13:41:47.000Z","updated_at":"2020-12-31T14:14:44.000Z","dependencies_parsed_at":null,"dependency_job_id":"76c38ebd-4298-487e-a897-18ffa81c4d49","html_url":"https://github.com/MatMoore/etl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/MatMoore/etl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatMoore%2Fetl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatMoore%2Fetl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatMoore%2Fetl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatMoore%2Fetl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MatMoore","download_url":"https://codeload.github.com/MatMoore/etl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatMoore%2Fetl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280761265,"owners_count":26386238,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-24T02:00:06.418Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-28T09:57:27.391Z","updated_at":"2025-10-24T07:37:05.903Z","avatar_url":"https://github.com/MatMoore.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ETL\nThis repository contains data pipelines for rescuing my data from various web services.\n\nThese are implemented in Python and Airflow.\n\nGoals:\n\n- I have a backup of my data in case the service goes away or I stop using it\n- I can easily query my data without writing code\n- I can join and aggregate data from different sources\n- I can filter data by the date it was generated\n\n## Local development\n\n### Python setup\nInstall Python 3.8.\n\n```\npython -m venv env\n. env/bin/activate\npip install -r requirements.txt --use-deprecated legacy-resolver\n```\n\n### Airflow setup\n(Optional) [configure airflow](https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html). By default, `~/airflow/airflow.cfg` will be generated when you install airflow.\n\nBy default this will use the SequentialExecutor with SQLite, which is not recommended for production usage. For a more production-like setup using Postgres, change the following settings:\n\n```\nsql_alchemy_conn = postgres://etl:etl@localhost/airflow\nexecutor = LocalExecutor\n```\n\nBy default `dags_folder` will be set to `$AIRFLOW_HOME/dags`. To run these dags, configure it to point to the `src/dags` directory ([either in the airflow config or via environment variables](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dags-folder)).\n\nRun `airflow initdb`\n\n### Database setup\nCreate a postgres database for all the data to go into. E.g. on ubuntu:\n\n`sudo -u postgres createdb \u003cdbname\u003e`\n\nCreate a database user as well:\n\n```\nsudo -u postgres psql\n\u003e create user \u003cdbusername\u003e with password '\u003cdbpassword\u003e';\n\u003e grant all privileges on database dbname to \u003cdbusername\u003e;\n```\n\n### Generate API keys\nGenerate an Airtable API key on the [account page](https://airtable.com/account).\n\n### Configure the local environment\nCreate a `.env` file in the root of the project. Set the following environment variables:\n\n```\nAIRTABLE_API_KEY=####\nAIRFLOW_CONN_POSTGRES_MOVIES=postgresql://\u003cdbusername\u003e:\u003cdbpassword\u003e@localhost/\u003cdbname\u003e\n```\n\n### Debugging commands\n```\n# Run a task in isolation\nairflow test syncing_movie_and_tv_data create_my_ratings 2015-06-01\n\n# Backfill\nairflow backfill syncing_movie_and_tv_data -s 2020-12-06\n```\n\n### Automated tests\nThere is a CI build defined in `.github/workflows/ci.yaml`.\n\nYou can run this build locally using [act](https://github.com/nektos/act).\n\n```\n# Workaround for ubuntu-latest not working\ncd .github/workflows/ubuntu2004\ndocker build .\ndocker image tag sha256:\u003cimage hash\u003e github-ubuntu-20.04\n\nact -P ubuntu-latest=github-ubuntu-20.04\n```\n\n## Pipelines\n\n### TV and Movies\nThis builds a database of TV shows and movies I've watched.\n\nHow this should work:\n\n- Identify shows and movies by IMDB ID and/or https://www.themoviedb.org/\n- Get my ratings from Airtable and/or Letterboxd\n- Store images in S3\n- Copy movie metadata to Airtable\n\n## Licence\n\nAll code is licenced under MIT.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatmoore%2Fetl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatmoore%2Fetl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatmoore%2Fetl/lists"}