{"id":26747167,"url":"https://github.com/hanchiang/market-data","last_synced_at":"2026-05-16T11:02:55.661Z","repository":{"id":64897770,"uuid":"568336722","full_name":"hanchiang/market-data","owner":"hanchiang","description":"API server and job scheduler for stock options data","archived":false,"fork":false,"pushed_at":"2023-02-28T13:58:45.000Z","size":55,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-03-06T01:08:40.029Z","etag":null,"topics":["fastapi","python3","stock-market"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hanchiang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-11-20T07:45:19.000Z","updated_at":"2023-03-06T01:08:40.030Z","dependencies_parsed_at":"2023-02-18T17:16:26.587Z","dependency_job_id":null,"html_url":"https://github.com/hanchiang/market-data","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanchiang%2Fmarket-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanchiang%2Fmarket-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanchiang%2Fmarket-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanchiang%2Fmarket-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hanchiang","download_url":"https://codeload.github.com/hanchiang/market-data/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245999608,"owners_count":20707573,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fastapi","python3","stock-market"],"created_at":"2025-03-28T09:17:12.023Z","updated_at":"2026-05-16T11:02:55.654Z","avatar_url":"https://github.com/hanchiang.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Introduction\nThis project is a API service that retrieves options data for a ticker and an expiration date.  \nThere are 2 parts to the project:\n* Scheduler:\n  * `src/job/options/scraper.py`: Options data for stocks\n  * `src/job/stocks/scraper.py`: Stock price\n  * TODO: Most active options\n  * TODO: Changes in options open interest\n* API server: Serves options prices(WIP)\n  * `src/server/main.py`\n\n# Set up\n* Install [Python 3.12](https://www.python.org/downloads/)\n  * Supported local interpreter range is `\u003e=3.12,\u003c3.13`\n  * Python 3.13+ is currently not supported because the released `market-data-library` dependency is pinned to Python 3.12 and `asyncpg==0.29.0` does not build cleanly on Python 3.14\n* Copy environment template: `cp .env.example .env`\n* Configure database settings in `.env`\n* Install dependencies with Poetry: `poetry install --no-root`\n  * `market-data-library` is installed from `git@github.com:hanchiang/market_data_api.git` at tag `1.5.0`\n* Confirm which `market-data-library` source the repo currently imports:\n  * `bash scripts/show_market_data_library_source.sh`\n* Switch to the sibling workspace checkout when you need unpublished local library changes:\n  * `bash scripts/use_local_market_data_library.sh`\n* Switch back to the released Git dependency:\n  * `bash scripts/use_git_market_data_library.sh`\n* Set PYTHONPATH: `export PYTHONPATH=$(pwd)`\n* Run migrations before starting the app: `poetry run piccolo migrations forwards market_data_piccolo --trace`\n* Start server: `poetry run uvicorn --reload --app-dir src/server main:app`\n  * Server runs at: `localhost:8000`\n  * API documentation runs at: `localhost:8000/docs`, `localhost:8000/redoc`.\n* Docker builds install from the same locked Poetry dependency graph as local development.\n* Before using the API or scrapers, ensure Postgres is reachable and Piccolo migrations have already been applied with the configured credentials.\n\n# Run Modes\n* Dockerized DB, app running locally:\n  * Start only the database: `docker compose up -d db`\n  * In `.env`, set `POSTGRES_HOST=localhost`\n  * Run migrations locally: `poetry run piccolo migrations forwards market_data_piccolo --trace`\n  * Set `PYTHONPATH`: `export PYTHONPATH=$(pwd)`\n  * Start the app locally: `poetry run uvicorn --reload --app-dir src/server main:app`\n* Fully Dockerized app and DB:\n  * Use the compose workflow in the Docker section below\n\n# Docker\n* Before any Docker build that needs `market-data-library`, create the build secret used by Dockerfiles:\n  * `mkdir -p secret`\n  * `printf '%s' \"$GITHUB_TOKEN_WITH_REPO_ACCESS\" \u003e secret/github_token`\n* Future CI note:\n  * when this repo adds CI or CD workflows that need to install the private `market-data-library` dependency, mirror the backend repo pattern instead of hard-coding a personal token\n  * provide GitHub App credentials such as `MARKET_DATA_LIBRARY_GITHUB_APP_ID` and `MARKET_DATA_LIBRARY_GITHUB_APP_PRIVATE_KEY`\n  * mint a short-lived installation token in CI, then use that token for `poetry install` and any Docker `secret/github_token` build-secret step\n* You can also start the full stack with `docker compose up -d`\n  * Compose will start `db`, run the one-shot `migrate` service, then start `backend`\n* Default local compose path uses the released git-tagged `market-data-library` package via the locked Poetry dependency graph:\n  * `mkdir -p secret`\n  * `printf '%s' \"$GITHUB_TOKEN_WITH_REPO_ACCESS\" \u003e secret/github_token`\n  * `docker compose up -d db`\n  * `docker compose run --rm migrate`\n  * `docker compose up backend`\n  * Inside compose, the backend and migration service connect to Postgres with `POSTGRES_HOST=db`\n* To make the container import the sibling workspace checkout instead, uncomment the documented `../market-data-library` bind mount plus `PYTHONPATH` override in [docker-compose.yml](docker-compose.yml) before recreating the backend container.\n* To build the git-backed dev image directly:\n  * `mkdir -p secret`\n  * `printf '%s' \"$GITHUB_TOKEN_WITH_REPO_ACCESS\" \u003e secret/github_token`\n  * `docker build --secret id=github_token,src=secret/github_token --target dev-git -t market-data:dev-git .`\n  * This validation path requires a GitHub token with access to the private repository.\n\n# Operations\n* Piccolo migration status check:\n  * `poetry run piccolo migrations check`\n* Create a new Piccolo migration:\n  * `poetry run piccolo migrations new market_data_piccolo`\n* Apply migrations locally:\n  * `poetry run piccolo migrations forwards market_data_piccolo --trace`\n* Apply migrations in Docker:\n  * `docker compose run --rm migrate`\n* Roll back migrations locally:\n  * `poetry run piccolo migrations backwards market_data_piccolo --trace`\n* Roll back all migrations locally:\n  * `poetry run piccolo migrations backwards market_data_piccolo --migration_id=all --trace --auto_agree`\n* Show compose service status:\n  * `docker compose ps`\n* Show compose logs:\n  * `docker compose logs --tail 100 backend migrate db`\n* Stop the Dockerized stack:\n  * `docker compose down`\n* Run the options scraper locally:\n  * `poetry run python src/job/options/scraper.py`\n* Run the stocks scraper locally:\n  * `poetry run python src/job/stocks/scraper.py`\n\n# Tech stack\n* Language: Python\n* Framework: FastAPI\n* Database: TimescaleDB\n* ORM: Piccolo\n\n# Features\n2 kinds of data:\n- Aggregated data for a stock ticker(volume(total, put, call), IV, IV rank, Iv %, underlying IV 1 year high)\n- Individual options data(expiration date, strike price, option type) for a stock ticker such as greeks, volatility, average volatility, volume, open interest\n\nView options data for a ticker for an expiration date, strike price, option type, for each day(change - line graph) and total(sum - bar graph).\n* volume/open interest/IV rank/IV % of option type by expiration date and strike price \n* Highest spike in volume/open interest/IV rank/IV % by expiration date and strike price\n\nOther metrics: % change\n\n## Options\n- [ ] View how different dimensions change over time for an expiration date, strike price(+- 20 from current price)\n  - [ ] High priority: Open interest, volume, IV, last, change\n- [ ] Aggregate statistics for an expiration date\n  - [ ] Open interest by strike price\n  - [ ] Volume by strike price\n\n\n## Stocks\n- [ ] View how different dimensions change over time\n  - [ ] bid/ask, size\n  - [ ] volume\n  - [ ] OHLC\n\n\n# Data model\n## Stock ticker table\nColumns:\n* id(PK) - big serial\n* symbol(PK) - varchar\n* name - varchar\n\n## Stock ticker price table\n* id(PK) - big serial\n* symbol(FK stock ticker)\n* date - date\n* open_price - decimal\n* high_price - decimal\n* low_price - decimal\n* close_price - decimal\n* volume - integer\n\n## Options price table\nStore snapshot of options price for a stock ticker, expiration date, strike price, option type for every trade day\n* id(PK) - big serial\n* symbol - varchar\n* base_symbol(FK stock ticker)\n* trade_time - timestamp\n* option_type - varchar\n* strike_price - decimal\n* open_price - decimal\n* high_price - decimal\n* low_price - decimal\n* last_price - decimal\n* moneyness - decimal\n* bid_price - decimal\n* midpoint - decimal\n* ask_price - decimal\n* volume - integer\n* open_interest - integer\n* volatility - decimal\n* expiration_date - date\n* expiration_type - varchar\n* average_volatility - decimal\n* delta - decimal\n* theta - decimal\n* gamma - decimal\n* vega - decimal\n* rho - decimal\n\n## Most active options table\nStore ticker that have the most options activity(open interest, volume, IV)\n* id(PK) - big serial\n* symbol - varchar\n* last_price - decimal\n* price_change - decimal\n* percent_change - decimal\n* options_total_volume - integer\n* options_put_volume_percent - decimal\n* options_call_volume_percent - decimal\n* options_weighted_implied_volatility - decimal\n* options_implied_volatility_rank_1y - decimal\n* options_implied_volatility_percentile_1y - decimal\n* underlying_implied_volatility_high_1y - decimal\n* trade_time - timestamp\n\n## Options open interest change table\nStore options with the largest positive and negative open interest change\n* id(PK) - big serial\n* symbol - varchar\n* base_symbol - varchar\n* last_price - decimal\n* option_type - varchar\n* strike_price - decimal\n* expiration_date - date\n* days_to_expiration - integer\n* bid_price - decimal\n* midpoint - decimal\n* ask_price - decimal\n* last_price - decimal\n* volume - integer\n* open_interest - integer\n* open_interest_change - integer\n* volatility - decimal\n* trade_time - timestamp\n\n# TODO:\n* market data library change refactor status: done\n* background job to remove option_price that have expired\n* job: largest change in open interest, most active options\n* Reporting: number of stock ticker in option_price table, number of rows for each stock ticker\n* Confirm whether options data is indeed a daily snapshot that includes all past trade days\n* Encode response data\n* Test\n* Get expirations date of a stock\n* Need to wait for timescaledb to support MACOS ventura....\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhanchiang%2Fmarket-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhanchiang%2Fmarket-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhanchiang%2Fmarket-data/lists"}