{"id":13468872,"url":"https://github.com/theOGognf/finagg","last_synced_at":"2025-03-26T05:31:21.015Z","repository":{"id":147846755,"uuid":"541211616","full_name":"theOGognf/finagg","owner":"theOGognf","description":"A Python package for aggregating and normalizing historical data from popular and free financial APIs.","archived":false,"fork":false,"pushed_at":"2025-02-08T15:32:34.000Z","size":7164,"stargazers_count":467,"open_issues_count":0,"forks_count":23,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-03-24T02:02:04.069Z","etag":null,"topics":["finance"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/theOGognf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-25T15:11:34.000Z","updated_at":"2025-03-23T08:23:37.000Z","dependencies_parsed_at":"2024-01-07T04:42:16.221Z","dependency_job_id":"664f2776-649d-4f82-bc43-8919b1e8943b","html_url":"https://github.com/theOGognf/finagg","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theOGognf%2Ffinagg","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theOGognf%2Ffinagg/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theOGognf%2Ffinagg/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theOGognf%2Ffinagg/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/theOGognf","download_url":"https://codeload.github.com/theOGognf/finagg/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245597324,"owners_count":20641865,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["finance"],"created_at":"2024-07-31T15:01:20.901Z","updated_at":"2025-03-26T05:31:20.972Z","avatar_url":"https://github.com/theOGognf.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":["Data Sources"],"readme":"# finagg: Financial Aggregation for Python\n\n![PyPI Downloads](https://img.shields.io/pypi/dm/finagg)\n![PyPI Version](https://img.shields.io/pypi/v/finagg)\n![Python Versions](https://img.shields.io/pypi/pyversions/finagg)\n\n**finagg** is a Python package that provides implementations of popular and free\nfinancial APIs, tools for aggregating historical data from those APIs into SQL\ndatabases, and tools for transforming aggregated data into features useful for\nanalysis and AI/ML.\n\n* **Documentation**: https://theogognf.github.io/finagg/\n* **PyPI**: https://pypi.org/project/finagg/\n* **Repository**: https://github.com/theOGognf/finagg\n\n# Quick Start\n\n## Installation\n\nInstall with pip for the latest stable version.\n\n```console\npip install finagg\n```\n\nInstall from GitHub for the latest unstable version.\n\n```console\ngit clone https://github.com/theOGognf/finagg.git\npip install ./finagg/\n```\n\nOptionally install the recommended datasets (economic data, company\nfinancials, stock histories, etc.) from 3rd party APIs into a local SQL\ndatabase.\n\n```console\nfinagg install -ss economic -ts indices -z -r\n```\n\nThe installation will point you where to get free API keys for each API that\nrequires one and will write those API keys to a local ``.env`` file for storage.\nRun ``finagg install --help`` for more installation options and details.\n\n## Basic Usage\n\nThese are just **finagg** usage samples. See the [documentation][4] for all the\nsupported APIs and features.\n\n### Explore the APIs directly\n\n*These methods require internet access and API keys/user agent declarations.*\n\nGet Bureau of Economic Analysis (BEA) data.\n\n```pycon\n\u003e\u003e\u003e finagg.bea.api.gdp_by_industry.get(year=[2019]).head(5)\n   table_id freq  year quarter industry                         industry_description ...\n0         1    Q  2019       1       11  Agriculture, forestry, fishing, and hunting ...\n1         1    Q  2019       1    111CA                                        Farms ...\n2         1    Q  2019       1    113FF    Forestry, fishing, and related activities ...\n3         1    Q  2019       1       21                                       Mining ...\n4         1    Q  2019       1      211                       Oil and gas extraction ...\n```\n\nGet Federal Reserve Economic Data (FRED).\n\n```pycon\n\u003e\u003e\u003e finagg.fred.api.series.observations.get(\n...   \"CPIAUCNS\",\n...   realtime_start=0,\n...   realtime_end=-1,\n...   output_type=4\n... ).head(5)\n  realtime_start realtime_end        date  value series_id\n0     1949-04-22   1953-02-26  1949-03-01  169.5  CPIAUCNS\n1     1949-05-23   1953-02-26  1949-04-01  169.7  CPIAUCNS\n2     1949-06-24   1953-02-26  1949-05-01  169.2  CPIAUCNS\n3     1949-07-22   1953-02-26  1949-06-01  169.6  CPIAUCNS\n4     1949-08-26   1953-02-26  1949-07-01  168.5  CPIAUCNS\n```\n\nGet Securities and Exchange Commission (SEC) filings.\n\n```pycon\n\u003e\u003e\u003e finagg.sec.api.company_facts.get(ticker=\"AAPL\").head(5)\n          end        value                  accn    fy  fp    form       filed ...\n0  2009-06-27  895816758.0  0001193125-09-153165  2009  Q3    10-Q  2009-07-22 ...\n1  2009-10-16  900678473.0  0001193125-09-214859  2009  FY    10-K  2009-10-27 ...\n2  2009-10-16  900678473.0  0001193125-10-012091  2009  FY  10-K/A  2010-01-25 ...\n3  2010-01-15  906794589.0  0001193125-10-012085  2010  Q1    10-Q  2010-01-25 ...\n4  2010-04-09  909938383.0  0001193125-10-088957  2010  Q2    10-Q  2010-04-21 ...\n```\n\n### Use installed raw data for exploring the most popular features\n\n*These methods require internet access, API keys/user agent declarations, and\ndownloading and installing raw data through the* ``finagg install`` *or*\n``finagg \u003capi/subpackage\u003e install`` *commands.*\n\nGet the most popular FRED features all in one dataframe.\n\n```pycon\n\u003e\u003e\u003e finagg.fred.feat.economic.from_raw().head(5)\n            CIVPART  LOG_CHANGE(CPIAUCNS)  LOG_CHANGE(CSUSHPINSA)  FEDFUNDS ...\ndate                                                                        ...\n2014-10-06     62.8                   0.0                     0.0      0.09 ...\n2014-10-08     62.8                   0.0                     0.0      0.09 ...\n2014-10-13     62.8                   0.0                     0.0      0.09 ...\n2014-10-15     62.8                   0.0                     0.0      0.09 ...\n2014-10-20     62.8                   0.0                     0.0      0.09 ...\n```\n\nGet quarterly report features from SEC data.\n\n```pycon\n\u003e\u003e\u003e finagg.sec.feat.quarterly.from_raw(\"AAPL\").head(5)\n                    LOG_CHANGE(Assets)  LOG_CHANGE(AssetsCurrent) ...\nfy   fp filed                                                     ...\n2010 Q1 2010-01-25            0.182629                  -0.023676 ...\n     Q2 2010-04-21            0.000000                   0.000000 ...\n     Q3 2010-07-21            0.000000                   0.000000 ...\n2011 Q1 2011-01-19            0.459174                   0.278241 ...\n     Q2 2011-04-21            0.000000                   0.000000 ...\n```\n\nGet an aggregation of quarterly and daily features for a particular ticker.\n\n```pycon\n\u003e\u003e\u003e finagg.fundam.feat.fundam.from_raw(\"AAPL\").head(5)\n            PriceBookRatio  PriceEarningsRatio\ndate\n2010-01-25        0.175061            2.423509\n2010-01-26        0.178035            2.464678\n2010-01-27        0.178813            2.475448\n2010-01-28        0.177154            2.452471\n2010-01-29        0.173825            2.406396\n```\n\n### Use installed features for exploring refined aggregations of raw data\n\n*These methods require installing refined data through the* ``finagg install``\n*or* ``finagg \u003capi/subpackage\u003e install`` *commands.*\n\nGet a ticker's industry's averaged quarterly report features.\n\n```pycon\n\u003e\u003e\u003e finagg.sec.feat.quarterly.industry.from_refined(ticker=\"AAPL\").head(5)\n                                 mean                           ...\nname               AssetCoverageRatio BookRatio DebtEquityRatio ...\nfy   fp filed                                                   ...\n2014 Q1 2014-05-15          10.731301  9.448954        0.158318 ...\n     Q2 2014-08-14          10.731301  9.448954        0.158318 ...\n     Q3 2014-11-14          10.731301  9.448954        0.158318 ...\n2015 Q1 2015-05-15          16.738972  9.269250        0.294238 ...\n     Q2 2015-08-13          16.738972  9.269250        0.294238 ...\n```\n\nGet a ticker's industry-averaged quarterly report features.\n\n```pycon\n\u003e\u003e\u003e finagg.sec.feat.quarterly.normalized.from_refined(\"AAPL\").head(5)\n                    NORM(LOG_CHANGE(Assets))  NORM(LOG_CHANGE(AssetsCurrent)) ...\nfy   fp filed                                                                 ...\n2010 Q2 2010-04-21                  0.000000                         0.000000 ...\n     Q3 2010-07-21                  0.000000                         0.000000 ...\n2011 Q1 2011-01-19                  0.978816                         0.074032 ...\n     Q2 2011-04-21                  0.000000                         0.000000 ...\n     Q3 2011-07-20                 -0.353553                        -0.353553 ...\n```\n\nGet tickers sorted by an industry-averaged quarterly report feature.\n\n```pycon\n\u003e\u003e\u003e finagg.sec.feat.quarterly.normalized.get_tickers_sorted_by(\n...   \"NORM(EarningsPerShareBasic)\",\n...   year=2019\n... )[:5]\n['XRAY', 'TSLA', 'SYY', 'WHR', 'KMB']\n```\n\nGet tickers sorted by an industry-averaged fundamental feature.\n\n```pycon\n\u003e\u003e\u003e finagg.fundam.feat.fundam.normalized.get_tickers_sorted_by(\n...   \"NORM(PriceEarningsRatio)\",\n...   date=\"2019-01-04\"\n... )[:5]\n['AMD', 'TRGP', 'HPE', 'CZR', 'TSLA']\n```\n\n# Configuration\n\n## API Keys and User Agents\n\nAPI keys and user agent declarations are required for most of the APIs.\nYou can set environment variables to expose your API keys and user agents\nto **finagg**, or you can pass your API keys and user agents to the implemented\nAPIs programmatically. The following environment variables are used for\nconfiguring API keys and user agents:\n\n* ``BEA_API_KEY`` is for the Bureau of Economic Analysis's API key. You can get\n  a free API key from the [BEA API site][3].\n* ``FRED_API_KEY`` is for the Federal Reserve Economic Data API key. You can get\n  a free API key from the [FRED API site][8].\n* ``INDICES_API_USER_AGENT`` is for scraping popular indices' compositions from\n  Wikipedia and should be equivalent to a browser's user agent declaration.\n  This defaults to a hardcoded value, but it may not always work.\n* ``SEC_API_USER_AGENT`` is for the Securities and Exchange Commission's API. This\n  should be of the format ``FIRST_NAME LAST_NAME E_MAIL``.\n\n## Data Locations\n\n**finagg**'s root path, HTTP cache path, and database path are all configurable\nthrough environment variables. By default, all data related to **finagg** is put\nin a ``./findata`` directory relative to a root directory. You can change these\nlocations by modifying the respective environment variables:\n\n* ``FINAGG_ROOT_PATH`` points to the parent directory of the ``./findata`` directory.\n  Defaults to your current working directory.\n* ``FINAGG_HTTP_CACHE_PATH`` points to the HTTP requests cache SQLite storage.\n  Defaults to ``./findata/http_cache.sqlite``.\n* ``FINAGG_DATABASE_URL`` points to the **finagg** data storage. Defaults to\n  ``./findata/finagg.sqlite``.\n\n## Other\n\nYou can change some **finagg** behavior with other environment variables:\n\n* ``FINAGG_DISABLE_HTTP_CACHE``: Set this to ``\"1\"`` or ``\"True\"`` to disable the\n  HTTP requests cache. Instead of a cachable session, a default, uncached user\n  session will be used for all requests.\n\n# Dependencies\n\n* [pandas][11] for fast, flexible, and expressive representations of relational data.\n* [requests][12] for HTTP requests to 3rd party APIs.\n* [requests-cache][13] for caching HTTP requests to avoid getting throttled by 3rd\n  party API servers.\n* [SQLAlchemy][17] for a SQL Python interface.\n* [yfinance][18] for historical stock data from Yahoo! Finance.\n\n# API References\n\n* The [BEA API][1] and the [BEA API key registration link][2].\n* The [FRED API][6] and the [FRED API key registration link][7].\n* The [SEC API][14].\n\n# Related Projects\n\n* [FinRL][5] is a collection of financial reinforcement learning environments\n  and tools.\n* [fredapi][9] is an implementation of the FRED API.\n* [OpenBBTerminal][10] is an open-source version of the Bloomberg Terminal.\n* [sec-edgar][15] is an implementation of a file-based SEC EDGAR parser.\n* [sec-edgar-api][16] is an implementation of the SEC EDGAR REST API.\n\n# Frequently Asked Questions\n\n## Where should I start?\n\nAggregate some data, create some analysis notebooks, or create some RL\nenvironments using the implemented data features and SQL tables. This\nproject was originally created to make RL environments for financial\napplications but has since focused its purpose to just aggregating financial\ndata and features. That being said, all the implemented features are\ndefined in such a way to make it very easy to develop financial AI/ML,\nso we encourage you to do just that!\n\n## Why aren't features being installed for a specific ticker or economic data series?\n\nImplemented APIs may be relatively new and simply may not provide data for a\nparticular ticker or economic data series. For example, earnings per share may\nnot be accessible for all companies through the SEC EDGAR API. In some cases,\nAPIs may raise an HTTP error, causing installations to skip the ticker or\nseries. Additionally, not all tickers and economic data series contain\nsufficient data for feature normalization. If a ticker or series only has one\ndata point, that data point could be dropped when computing a feature (such as\npercent change), causing no data to be installed.\n\n## What Python versions are supported?\n\nPython 3.10 and up are supported. We don't plan on supporting lower versions\nbecause 3.10 introduces some nice quality of life updates that are used\nthroughout the package.\n\n## What operating systems are supported?\n\nThe package is developed and tested on both Linux and Windows, but we recommend\nusing Linux or WSL in practice. The package performs a good amount of I/O and\ninterprocess operations that could result in a noticeable performance\ndegradation on Windows.\n\n[1]: https://apps.bea.gov/api/signup/\n[2]: https://apps.bea.gov/API/signup/\n[3]: https://apps.bea.gov/API/signup/\n[4]: https://theogognf.github.io/finagg/\n[5]: https://github.com/AI4Finance-Foundation/FinRL\n[6]: https://fred.stlouisfed.org/docs/api/fred/\n[7]: https://fredaccount.stlouisfed.org/login/secure/\n[8]: https://fredaccount.stlouisfed.org/login/secure/\n[9]: https://github.com/mortada/fredapi\n[10]: https://github.com/OpenBB-finance/OpenBBTerminal\n[11]: https://pandas.pydata.org/\n[12]: https://requests.readthedocs.io/en/latest/\n[13]: https://requests-cache.readthedocs.io/en/stable/\n[14]: https://www.sec.gov/edgar/sec-api-documentation\n[15]: https://github.com/sec-edgar/sec-edgar\n[16]: https://github.com/jadchaar/sec-edgar-api\n[17]: https://www.sqlalchemy.org/\n[18]: https://github.com/ranaroussi/yfinance\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FtheOGognf%2Ffinagg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FtheOGognf%2Ffinagg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FtheOGognf%2Ffinagg/lists"}