{"id":13586089,"url":"https://github.com/jbms/finance-dl","last_synced_at":"2025-04-07T15:02:03.122Z","repository":{"id":40380360,"uuid":"151639678","full_name":"jbms/finance-dl","owner":"jbms","description":"Tools for automatically downloading/scraping personal financial data.","archived":false,"fork":false,"pushed_at":"2024-10-07T04:27:53.000Z","size":266,"stargazers_count":303,"open_issues_count":11,"forks_count":39,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-03-31T14:07:23.984Z","etag":null,"topics":["beancount"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jbms.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-04T21:44:22.000Z","updated_at":"2025-03-18T09:11:42.000Z","dependencies_parsed_at":"2023-02-08T09:01:37.566Z","dependency_job_id":"507d8a8d-aa91-43ba-80c2-63d4e012123b","html_url":"https://github.com/jbms/finance-dl","commit_stats":{"total_commits":145,"total_committers":19,"mean_commits":7.631578947368421,"dds":0.5931034482758621,"last_synced_commit":"4b8e28a29b8f0faf5ab3457b5cded2079e73f3fd"},"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbms%2Ffinance-dl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbms%2Ffinance-dl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbms%2Ffinance-dl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbms%2Ffinance-dl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jbms","download_url":"https://codeload.github.com/jbms/finance-dl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247675597,"owners_count":20977376,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beancount"],"created_at":"2024-08-01T15:05:19.153Z","updated_at":"2025-04-07T15:02:03.098Z","avatar_url":"https://github.com/jbms.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"Python package for scraping personal financial data from financial\ninstitutions.\n\n[![License: GPL v2](https://img.shields.io/badge/License-GPL%20v2-blue.svg)](LICENSE)\n[![PyPI](https://img.shields.io/pypi/v/finance-dl)](https://pypi.org/project/finance-dl)\n[![Build](https://github.com/jbms/finance-dl/workflows/Build/badge.svg)](https://github.com/jbms/finance-dl/actions?query=workflow%3ABuild)\n\nThis package may be useful on its own, but is specifically designed to be\nused with\n[beancount-import](https://github.com/jbms/beancount-import).\n\nSupported data sources\n==\n\n- [finance_dl.ofx](finance_dl/ofx.py): uses\n  [ofxclient](https://github.com/captin411/ofxclient) to download data\n  using the OFX protocol.\n- [finance_dl.mint](finance_dl/mint.py): uses\n  [mintapi](https://github.com/mrooney/mintapi) to download data from\n  the Mint.com website.\n- [finance_dl.venmo](finance_dl/venmo.py): downloads transaction and\n  balance information from the Venmo.com website\n- [finance_dl.paypal](finance_dl/paypal.py): downloads transactions\n  from the Paypal.com website\n- [finance_dl.amazon](finance_dl/amazon.py): downloads order invoices\n  from the Amazon website\n- [finance_dl.healthequity](finance_dl/healthequity.py): downloads\n  transaction history and balance information from the HealthEquity\n  website.\n- [finance_dl.google_purchases](finance_dl/google_purchases.py):\n  downloads purchases that Google has heuristically extracted from\n  Gmail messages.\n- [finance_dl.stockplanconnect](finance_dl/stockplanconnect.py):\n  downloads PDF documents (including release and trade confirmations)\n  from the Morgan Stanley Stockplanconnect website.\n- [finance_dl.pge](finance_dl/pge.py): downloads Pacific Gas \u0026\n  Electric (PG\u0026E) PDF bills.\n- [finance_dl.comcast](finance_dl/comcast.py): downloads Comcast PDF\n  bills.\n- [finance_dl.ebmud](finance_dl/ebmud.py): downloads East Bay\n  Municipal Utility District (EBMUD) water bills.\n- [finance_dl.anthem](finance_dl/anthem.py): downloads Anthem\n  BlueCross insurance claim statements.\n- [finance_dl.waveapps](finance_dl/waveapps.py): downloads receipt\n  images and extracted transaction data from\n  [Wave](https://waveapps.com), which is a free receipt-scanning\n  website/mobile app.\n- [finance_dl.ultipro_google](finance_dl/ultipro_google.py): downloads\n  Google employee payroll statements in PDF format from Ultipro.\n- [finance_dl.usbank](finance_dl/usbank.py): downloads data from US Bank credit cards in OFX format.\n- [finance_dl.radiusbank](finance_dl/radiusbank.py): downloads data from Radius Bank in QFX format.\n- [finance_dl.schwab](finance_dl/schwab.py): downloads data from Schwab Brokerage accounts in CSV format.\n- [finance_dl.gemini](finance_dl/gemini.py): downloads trades, transfers and balances from Gemini crypto exchange using REST API, stores in a custom CSV format.\n\nSetup\n==\n\nTo install the most recent published package from PyPi, simply type:\n\n```shell\npip install finance-dl\n```\n\nTo install from a clone of the repository, type:\n\n```shell\npip install .\n```\n\nor for development:\n\n```shell\npip install -e .\n```\n\nConfiguration\n==\n\nCreate a configuration file called something like `finance_dl_config.py`.\nFor a complete example of this file and some documentation, \nsee [example_finance_dl_config.py](example_finance_dl_config.py).\n\nRefer to the documentation of the individual scraper modules for\nfurther details.\n\nBasic Usage\n==\n\nYou can run a scraping configuration named `myconfig` as follows:\n\n    python -m finance_dl.cli --config-module example_finance_dl_config --config myconfig\n\nThe configuration `myconfig` refers to a function named\n`CONFIG_myconfig` in the configuration module.\n\nMake sure that your configuration module is accessible in your Python\n`sys.path`.  Since `sys.path` includes the current directory by\ndefault, you can simply run this command from the directory that\ncontains your configuration module.\n\nBy default, the scrapers run fully automatically, and the ones based\non `selenium` and `chromedriver` run in headless mode.  If the initial\nattempt for a `selenium`-based scraper fails, it is automatically\nretried again with the browser window visible.  This allows you to\nmanually complete the login process and enter any multi-factor\nauthentication code that is required.\n\nTo debug a scraper, you can run it in interactive mode by specifying\nthe `-i` command-line argument.  This runs an interactive IPython\nshell that lets you manually invoke parts of the scraping process.\n\nAutomatic Usage\n==\n\nTo run multiple configurations at once, and keep track of when each\nconfiguration was last updated, you can use the `finance_dl.update`\ntool.\n\nTo display the update status, first create a `logs` directory and run:\n\n    python -m finance_dl.update --config-module example_finance_dl_config --log-dir logs status\n\nInitially, this will indicate that none of the configurations have\nbeen updated.  To update a single configuration `myconfig`, run:\n\n    python -m finance_dl.update --config-module example_finance_dl_config --log-dir logs update myconfig\n\nWith a single configuration specified, this does the same thing as the\n`finance_dl.cli` tool, except that the log messages are written to\n`logs/myconfig.txt` and a `logs/myconfig.lastupdate` file is created\nif it is successful.\n\nIf multiple configurations are specified, as in:\n\n    python -m finance_dl.update --config-module example_finance_dl_config --log-dir logs update myconfig1 myconfig2\n\nthen all specified configurations are run in parallel.\n\nTo update all configurations, run:\n\n    python -m finance_dl.update --config-module example_finance_dl_config --log-dir logs update --all\n\nNote on Chromedriver Versioning\n==\n\nChromedriver and Chrome are very tightly coupled; their versions need to\nmatch. `finance_dl` uses Chromedriver from the `chromedriver_binary` Python\npackage (not your system's installed Chromedriver binary). However,\nChromedriver, by default, uses your system's installed version of Chrome.\nDepending on how you manage the two installations on your system, this\ncombination may frequently end up causing `finance_dl` to fail with messages\nlike\n\n    selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 97\n    Current browser version is 96.0.4664.45 with binary path /usr/bin/google-chrome\n\nIn this event, you have a few options:\n\n1. Explicitly manage your version of the `chromedriver_binary` Python package\n   to match your installed version of Chrome;\n1. Explicitly manage your installed version of Chrome to match your version of\n   the `chromedriver_binary` Python package; or\n1. Install the version of Chrome matching your version of\n   `chromedriver_binary` somewhere other than your system's default Chrome\n   version, and set the environment variable `CHROMEDRIVER_CHROME_BINARY` to\n   point to it. (You can do this from within your finance_dl config script,\n   e.g. with a line like `os.environ[\"CHROMEDRIVER_CHROME_BINARY\"] = \"/usr/bin/google-chrome-beta\"`).\n\nLicense\n==\n\nCopyright (C) 2014-2018 Jeremy Maitin-Shepard.\n\nDistributed under the GNU General Public License, Version 2.0 only.\nSee [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjbms%2Ffinance-dl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjbms%2Ffinance-dl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjbms%2Ffinance-dl/lists"}