{"id":22112409,"url":"https://github.com/lukeredpath/hsbcscraper","last_synced_at":"2025-10-30T05:11:59.846Z","repository":{"id":6712763,"uuid":"7958424","full_name":"lukeredpath/hsbcscraper","owner":"lukeredpath","description":"Web scraper for downloading statements from HSBC Business Banking","archived":false,"fork":false,"pushed_at":"2013-02-08T12:27:53.000Z","size":103,"stargazers_count":15,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-07-25T11:30:03.433Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lukeredpath.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-02-01T13:13:59.000Z","updated_at":"2023-07-14T23:36:21.000Z","dependencies_parsed_at":"2022-08-26T04:02:02.322Z","dependency_job_id":null,"html_url":"https://github.com/lukeredpath/hsbcscraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lukeredpath/hsbcscraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukeredpath%2Fhsbcscraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukeredpath%2Fhsbcscraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukeredpath%2Fhsbcscraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukeredpath%2Fhsbcscraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lukeredpath","download_url":"https://codeload.github.com/lukeredpath/hsbcscraper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukeredpath%2Fhsbcscraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281748716,"owners_count":26554822,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-30T02:00:06.501Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-01T10:57:51.118Z","updated_at":"2025-10-30T05:11:59.816Z","avatar_url":"https://github.com/lukeredpath.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HSBC Business Banking Scraper\n\nA simple scraping mechanism, built on top of Mechanize, with a scraper for the HSBC Business banking website.\n\nAll of the dependencies are managed by Bundler. Run `bundle install` to get up and running.\n\nThe scraper architecture is quite simple, and is inspired by the Rack middleware API.\n\nThe `Scraper` class handles the pipeline - use the `build` method to build a pipeline of different scrapers. See `hsbc.rb` for an example.\n\nScraper modules for logging in and out and downloading statements from HSBC business banking are provided.\n\nThese scripts obviously contain no credentials - it is recommended you store these in something like the OSX keychain, which is supported natively in the provided scripts. Here's an example:\n\n```ruby\ndownload_handler = -\u003e(downloaded_file) {\n  downloaded_file.save('/Users/luke/Desktop/statement.qif')\n}\n\n# username, password and bank account number are stored in the OSX keychain\nkeychain_item = Keychain.generic_passwords.where(:service =\u003e 'hsbc-business').first\n\nraise \"Couldn't find keychain item!\" unless keychain_item\n\nstatement_date_range = [HSBC.last_statement_date, Date.today]\n\nscraper = Scraper.build(HSBC::BANKING_URL) do\n  use HSBC::Login.new(keychain_item.account, keychain_item.password, HSBC::InteractiveShellKeyfob.new)\n  use HSBC::DownloadStatement.new(keychain_item.comment, statement_date_range, \u0026download_handler)\n  use HSBC::Logout\nend\n\nscraper.run\n```\n\nThe `HSBC::DownloadStatement` scraper takes a block to handle the downloaded statement (which is an instance of Mechanize::File). \n\nThe above snippet will try to find the HSBC credentials from your OSX keychain with the service name of 'hsbc-business'. It will exit if it doesn't find them. See the `HSBC::Login` class for more details on how these credentials are used.\n\nIt will then read the date of the last downloaded statement from `~/.hsbc` and use that to fetch a statement from that date until the current date.\n\nYou will not be able to run these scripts using some automated method like cron - you will need to be present to run the scripts as it will prompt you from a code from your security fob.\n\n## Importing statement into FreeAgent\n\nIf you also happen to be using [FreeAgent](http://freeagent.com) to manage your books, you can automatically import your statement into FreeAgent using their API.\n\nThe `FreeAgent::Authenticator` class takes care of OAuth2 authentication. It expects a \"credentials\" object that responds to `account` and `password`. A method, `FreeAgent.credentials_from_keychain` is provided that can retrieve these credentials from the OSX keychain.\n\nSee `upload_statement_to_freeagent.rb` for more details. You will need to modify the `HSBC_BANK_ACCOUNT_ID` constant to match the ID of your HSBC account in FreeAgent.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukeredpath%2Fhsbcscraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flukeredpath%2Fhsbcscraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukeredpath%2Fhsbcscraper/lists"}