{"id":50814542,"url":"https://github.com/chrischow/open_options_chains","last_synced_at":"2026-06-13T08:05:47.759Z","repository":{"id":39798351,"uuid":"457649701","full_name":"chrischow/open_options_chains","owner":"chrischow","description":"A data pipeline solution for collecting options data at scale.","archived":false,"fork":false,"pushed_at":"2022-02-10T05:59:41.000Z","size":10,"stargazers_count":9,"open_issues_count":0,"forks_count":6,"subscribers_count":1,"default_branch":"main","last_synced_at":"2023-03-05T19:08:43.054Z","etag":null,"topics":["airflow","docker","docker-compose","postgresql","python3","tdameritrade"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chrischow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-02-10T05:56:16.000Z","updated_at":"2022-12-13T18:33:25.000Z","dependencies_parsed_at":"2022-08-30T21:51:58.926Z","dependency_job_id":null,"html_url":"https://github.com/chrischow/open_options_chains","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"purl":"pkg:github/chrischow/open_options_chains","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrischow%2Fopen_options_chains","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrischow%2Fopen_options_chains/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrischow%2Fopen_options_chains/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrischow%2Fopen_options_chains/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chrischow","download_url":"https://codeload.github.com/chrischow/open_options_chains/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrischow%2Fopen_options_chains/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34276557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-13T02:00:06.617Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","docker","docker-compose","postgresql","python3","tdameritrade"],"created_at":"2026-06-13T08:05:46.843Z","updated_at":"2026-06-13T08:05:47.715Z","avatar_url":"https://github.com/chrischow.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Open Options Chains\n### *A Data Pipeline Solution for Collecting Options Data at Scale*\nThis repository contains the code to implement my Airflow/PostgreSQL solution for managing a data pipeline for collection options data. I have written a series of posts detailing the setup:\n\n1. [Database Design](https://medium.datadriveninvestor.com/towards-open-options-chains-a-data-pipeline-for-collecting-options-data-at-scale-part-i-f757c156639b)\n2. [Foundational ETL Code](https://medium.datadriveninvestor.com/towards-open-options-chains-part-ii-foundational-etl-code-2bca40251f95)\n3. [Getting Started with Airflow](https://medium.datadriveninvestor.com/towards-open-options-chains-part-iii-getting-started-with-airflow-61c7c154f00c)\n4. [Building the DAG](https://hackernoon.com/how-to-build-a-directed-acyclic-graph-dag-towards-open-options-chains-part-iv)\n5. [Containerising the Pipeline](https://hackernoon.com/towards-open-options-chains-part-v-containerizing-the-pipeline)\n\n## Pre-requisites\n1. Operating system: Linux (I'm using Ubuntu 20.04)\n2. Docker\n3. Docker Compose\n\n## Installation\nIf you have Git installed, clone the repository to your local machine.\n\n```bash\ngit clone https://github.com/chrischow/open_options_chains.git\n```\n\nOtherwise, you may download the code and unzip the contents in a folder of your choice. Then, `cd` into that directory. You'll see:\n\n- `dags`: Contains the DAGs you'll need for\n- `db`: Contains a script to initialise the Postgres database\n- `logs`: Contains logs from your Airflow instance\n- `pgdata`: Contains data from Postgres\n- `scripts`: Contains a script to initialise Airflow\n- `docker-compose.yml`: File that defines the services, networks, and volumes for our app\n\nThen, change the required variables in the `.env` file. Note that when you edit the `POSTGRES_*` variables, you must change the associated variables in `AIRFLOW__CORE__SQL_ALCHEMY_CONN`. Do remember to input your TD Ameritrade (TDA) API key.\n\n```yaml\n# Postgres\nPOSTGRES_USER=airflow\nPOSTGRES_PASSWORD=airflow\nPOSTGRES_DB=airflow\nAPP_DB_USER=openoptions\nAPP_DB_PASS=openoptions\nAPP_DB_NAME=optionsdata\n\n# Airflow\nAPP_AIRFLOW_USERNAME=admin\nAPP_AIRFLOW_PASSWORD=password\nAIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres/airflow\nAIRFLOW__CORE__EXECUTOR=LocalExecutor\n\n# TDA API\nAPI_KEY=\u003cyour TDA API key\u003e\n```\n\n## Initialisation\nNext, we'll initialise Airflow with the command below. This will:\n\n1. Start a PostgreSQL container\n2. Initialise the Airflow Metadata Database and create an Airflow account\n\n```bash\ndocker-compose up airflow-init\n```\n\nThe `airflow-init` container will shut down after the initialisation is complete. The `postgres` container will continue running. You can stop it with `docker-compose stop` or `docker stop \u003ccontainer id\u003e` if you don't need it.\n\n## Usage\n\n### Launch\nLaunch Airflow with the command below. This will:\n\n1. Start a PostgreSQL container if it's not already running\n2. Start the Airflow Scheduler container\n3. Start the Airflow Webserver container\n\n```bash\ndocker-compose up\n```\n\nIn a browser, navigate to [http://localhost:8088](http://localhost:8088) to access the Airflow UI. For the first time login, use the credentials `APP_AIRFLOW_USERNAME` and\n`APP_AIRFLOW_PASSWORD` that you set in the `.env` file.\n\n### Shutdown\nUse the following command:\n\n```bash\ndocker-compose stop\n```\n\n### Extracting Data\n\n#### Export to CSV\nYou can extract the data in CSV format via the command line using the commands below:\n\n```bash\n# Check Postgres container ID\ndocker ps\n\n# Go to folder to store files\ncd \u003cdirectory-to-store-csv-files\u003e\n\n# Extract CSV file\ndocker exec -t \u003cfirst-few-characters-of-docker-container\u003e psql -U airflow -d optionsdata -c \"COPY table_name to STDOUT WITH CSV HEADER\" \u003e \"filename.csv\"\n```\n\n#### Connect to Postgres Instance\nAlternatively, you can connect to the Postgres database running in the container. Simply use the database name and login credentials set in the `.env` file.\n\n```bash\nimport psycopg2 as pg2\n\n# Connect to database\nconn = pg2.connect(host='localhost', database='optionsdata', user='airflow', password='airflow', port='5432')\n```\n\n### Full Reset - Take Caution!\nTo restart the environment:\n\n```bash\n# Completely shut down the app\ndocker-compose down\n\n# Remove contents of mounted volumes\nrm -rf logs/*\nrm -rf pgdata/*\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrischow%2Fopen_options_chains","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchrischow%2Fopen_options_chains","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrischow%2Fopen_options_chains/lists"}