{"id":29458917,"url":"https://github.com/paologazzola/etf-open-analytics","last_synced_at":"2026-01-20T16:26:04.651Z","repository":{"id":304495294,"uuid":"1018933849","full_name":"paologazzola/etf-open-analytics","owner":"paologazzola","description":"ETF Open Analytics is an open-source project for scraping and analyzing historical ETF (Exchange-Traded Funds) data.","archived":false,"fork":false,"pushed_at":"2025-11-09T16:12:41.000Z","size":520,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-09T18:13:14.783Z","etag":null,"topics":["data-scraping","etf","finance","financial-data","github-actions","machine-learning","open-data","portfolio-analysis","postgresql","python","yfinance"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paologazzola.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-13T11:27:59.000Z","updated_at":"2025-10-10T14:03:59.000Z","dependencies_parsed_at":"2025-08-02T15:18:20.254Z","dependency_job_id":null,"html_url":"https://github.com/paologazzola/etf-open-analytics","commit_stats":null,"previous_names":["paologazzola/etf-open-analytics"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/paologazzola/etf-open-analytics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paologazzola%2Fetf-open-analytics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paologazzola%2Fetf-open-analytics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paologazzola%2Fetf-open-analytics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paologazzola%2Fetf-open-analytics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paologazzola","download_url":"https://codeload.github.com/paologazzola/etf-open-analytics/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paologazzola%2Fetf-open-analytics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28607021,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-20T16:10:39.856Z","status":"ssl_error","status_checked_at":"2026-01-20T16:10:39.493Z","response_time":117,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-scraping","etf","finance","financial-data","github-actions","machine-learning","open-data","portfolio-analysis","postgresql","python","yfinance"],"created_at":"2025-07-14T00:10:03.315Z","updated_at":"2026-01-20T16:26:04.640Z","avatar_url":"https://github.com/paologazzola.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ETF Open Analytics\n\n## Table of Contents\n\n- [ETF Scraper](#etf-scraper)\n- [ETF Predictor](#etf-predictor)\n- [Portfolio Evaluator](#portfolio-evaluator)\n- [License](#license)\n- [Disclaimer on Data Usage](#disclaimer-on-data-usage)\n\nETF Open Analytics is an open-source project for scraping and analyzing historical ETF (Exchange-Traded Funds) data.\n\nIt uses:\n- **Python** for data collection  \n- **PostgreSQL** for structured data storage  \n- **GitHub Actions** to schedule and automate daily updates  \n\nThis is a **proof-of-concept project**: the dataset currently includes only a subset of ETFs and partial historical data.  \nHowever, the architecture is designed to be extensible, and it can be expanded to include more complete data and support deeper, more accurate portfolio analysis, backtesting, and machine learning applications.\n\n## ETF Scraper\n\nThe `etf_scraper` module is responsible for downloading and storing historical ETF price data.\n\n### Features\n\n- Retrieves daily historical prices using [Yahoo Finance](https://finance.yahoo.com/) via `yfinance`\n- Stores data into a PostgreSQL database (e.g. [Render](https://render.com))\n- Keeps track of the latest saved price per ETF\n- Automatically updates missing days on each run\n- Uses GitHub Actions to run daily via scheduled job\n\n### Database Tables\n\n- `etf`: contains ETF metadata (ISIN, description, region, replication method, etc.)\n- `etf_price`: contains historical daily prices for each ETF\n\n### Usage\n\n```bash\n# Install dependencies\npip install -r etf_scraper/requirements.txt\n\n# Run the updater manually\npython etf_scraper/main.py\n```\n\nMake sure to create a .env file with the following variables:\n```env\nDATABASE_URL=postgresql://\u003cuser\u003e:\u003cpassword\u003e@\u003chost\u003e/\u003cdb\u003e\nTICKER_FIELD=yahoo_symbol\n```\n\n### GitHub Action\nThe workflow .github/workflows/fetch.yml runs the scraper daily using:\n```yaml\non:\n  schedule:\n    - cron: '0 6 * * *'  # Every day at 6:00 UTC\n```\n\nTo use it:\n- Add `DATABASE_URL` as a GitHub secret\n- Add `TICKER_FIELD` as a GitHub environment variable\n\n## ETF Predictor\n\nThe `etf_predictor` module provides tools for forecasting future ETF prices and trends using historical data.\n\n### Features\n\n- Applies machine learning models to predict future ETF prices\n- Supports backtesting with historical data\n- Allows evaluation of prediction accuracy\n- Designed for extensibility with custom models and strategies\n\n### Usage\n\n```bash\n# Install dependencies\npip install -r etf_predictor/requirements.txt\n\n# Run predictor webapp\nuvicorn main:app --host 0.0.0.0 --port 8000\n```\n\nMake sure to create a `.env` file with any required variables for your portfolio evaluation module.\n\n## Portfolio Evaluator\n\nThe `portfolio_evaluator` module provides advanced tools for analyzing, simulating, and optimizing ETF portfolios using historical data and machine learning techniques.\n\n### Features\n\n- Calculates key portfolio metrics such as returns, volatility, Sharpe ratio, and maximum drawdown\n- Simulates portfolio performance over time using historical ETF prices\n- Supports scenario analysis and stress testing to evaluate portfolio robustness\n- Applies machine learning models to optimize portfolio allocation based on risk and return objectives\n- Performs feature selection and dimensionality reduction to identify the most relevant ETFs for a given strategy\n- Enables backtesting of portfolio strategies and evaluation of their predictive performance\n- Designed for extensibility with custom evaluation metrics and optimization algorithms\n\n### Usage\n\n```bash\n# Install dependencies\npip install -r portfolio_evaluator/requirements.txt\n\n# Run portofolio_evaluator webapp\nuvicorn main:app --host 0.0.0.0 --port 8000\n\n# Update or retrain the portfolio evaluation model (run once)\npython portfolio_evaluator/run_once.py\n```\n\nMake sure to create a `.env` file with any required variables for the portfolio\n\n## License\nThis project is licensed under the [Creative Commons Attribution-NonCommercial 4.0 International License](https://creativecommons.org/licenses/by-nc/4.0/).\n\nYou are free to use, modify, and distribute this code for **non-commercial purposes only**.  \nCommercial use of this software or the data it processes is **strictly prohibited**.\n\n## Disclaimer on Data Usage\nThis project uses data fetched from Yahoo Finance (via `yfinance`) for educational and non-commercial research purposes only.  \nMake sure your usage complies with [Yahoo’s Terms of Service](https://legal.yahoo.com/ie/en/yahoo/terms/otos/index.html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaologazzola%2Fetf-open-analytics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpaologazzola%2Fetf-open-analytics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaologazzola%2Fetf-open-analytics/lists"}