{"id":15722128,"url":"https://github.com/eidorb/portfolio","last_synced_at":"2025-05-13T03:43:54.290Z","repository":{"id":236896984,"uuid":"500762094","full_name":"eidorb/portfolio","owner":"eidorb","description":"Serverless investment portfolio management","archived":false,"fork":false,"pushed_at":"2025-04-26T07:43:37.000Z","size":488,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-26T08:31:23.724Z","etag":null,"topics":["aws-cdk","datasette","git-scraping","mangum"],"latest_commit_sha":null,"homepage":"https://eidorb.github.io/portfolio/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eidorb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-07T08:53:05.000Z","updated_at":"2025-04-26T07:43:41.000Z","dependencies_parsed_at":"2024-04-29T06:32:12.522Z","dependency_job_id":"ed503d32-e8e6-4690-825e-1814cb261977","html_url":"https://github.com/eidorb/portfolio","commit_stats":{"total_commits":191,"total_committers":2,"mean_commits":95.5,"dds":"0.041884816753926746","last_synced_commit":"fdfff2780a9937dd13324c612292a469dee395fb"},"previous_names":["eidorb/portfolio"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eidorb%2Fportfolio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eidorb%2Fportfolio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eidorb%2Fportfolio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eidorb%2Fportfolio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eidorb","download_url":"https://codeload.github.com/eidorb/portfolio/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253870822,"owners_count":21976610,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-cdk","datasette","git-scraping","mangum"],"created_at":"2024-10-03T22:04:11.438Z","updated_at":"2025-05-13T03:43:54.268Z","avatar_url":"https://github.com/eidorb.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Portfolio\n\nSee [*Serverless investment portfolio management*](https://brodie.id.au/blog/serverless-investment-management-portfolio.html) for an introduction to this project.\n\n\n## Summary\n\nThis project assists with investment portfolio management.\nIt retrieves account balances from financial institutions and deploys a web\napplication summarising the actions required to [rebalance](https://www.bogleheads.org/wiki/Rebalancing)\nthe portfolio.\n\nYou can take a look at a demo deployment [here](https://portfolio-demo.brodie.id.au).\n\nThe project is comprised of several pieces working together:\n- A GitHub Actions workflow periodically runs [Python code](portfolio) to retrieve account balances.\n- Balances and asset prices are stored in a plain text [Beancount](https://beancount.github.io) ledger.\n- [AWS Cloud Development Kit](https://docs.aws.amazon.com/cdk/v2/guide/home.html) is used to define the cloud application in [code](cdk).\n- [Datasette](https://datasette.io) transforms data in the Beancount ledger into an interactive website.\n- [Mangum](https://mangum.fastapiexpert.com) allows the Datasette web application to run on AWS Lambda, resulting in a practically zero-cost deployment.\n\n\n## Contents\n\n- [Summary](#summary)\n- [Contents](#contents)\n- [How-to guides](#how-to-guides)\n  - [How to set up a local development environment](#how-to-set-up-a-local-development-environment)\n  - [How to activate Mamba environment](#how-to-activate-mamba-environment)\n  - [How to update Node.js](#how-to-update-nodejs)\n  - [How to update AWS CDK Toolkit](#how-to-update-aws-cdk-toolkit)\n  - [How to update AWS Construct Library](#how-to-update-aws-construct-library)\n  - [How to update Python dependencies](#how-to-update-python-dependencies)\n  - [How to update Lambda function Python dependencies](#how-to-update-lambda-function-python-dependencies)\n  - [How to serve Datasette locally](#how-to-serve-datasette-locally)\n  - [How to rotate personal access tokens](#how-to-rotate-personal-access-tokens)\n    - [portfolio/actions/write](#portfolioactionswrite)\n    - [portfolio-ledger/contents/write](#portfolio-ledgercontentswrite)\n  - [How to update ubank device credentials](#how-to-update-ubank-device-credentials)\n- [Explanation](#explanation)\n  - [Storing secrets](#storing-secrets)\n  - [Datasette authentication](#datasette-authentication)\n  - [Embedding a dashboard in Datasette's index page](#embedding-a-dashboard-in-datasettes-index-page)\n  - [Lambda function performance](#lambda-function-performance)\n  - [GitHub Actions workflows](#github-actions-workflows)\n    - [test](#test)\n    - [update](#update)\n    - [deploy](#deploy)\n    - [dispatch (in portfolio-ledger)](#dispatch-in-portfolio-ledger)\n  - [Authenticating GitHub Actions workflows with AWS](#authenticating-github-actions-workflows-with-aws)\n  - [Routing GitHub Actions traffic via a Tailscale exit node](#routing-github-actions-traffic-via-a-tailscale-exit-node)\n- [Reference](#reference)\n  - [GitHub personal access tokens](#github-personal-access-tokens)\n  - [GitHub Actions secrets](#github-actions-secrets)\n  - [AWS Parameter Store parameters](#aws-parameter-store-parameters)\n\n\n## How-to guides\n\n### How to set up a local development environment\n\nCreate the `portfolio` Mamba environment defined in `environment.yml`:\n\n```bash\nmicromamba create --file environment.yml --yes\n```\n\nActivate the `portfolio` environment:\n\n```bash\nmicromamba activate portfolio\n```\n\nInstall Node.js dependencies:\n\n```bash\nnpm ci\n```\n\nInstall Python dependencies:\n\n```bash\npoetry install\n```\n\nInstall Lambda function Python dependencies:\n\n```bash\ncd cdk/function\npoetry install\ncd -\n```\n\n\n### How to activate Mamba environment\n\nActivate the `portfolio` environment with the following command:\n\n```bash\nmicromamba activate portfolio\n```\n\n### How to update Node.js\n\nPin the `nodejs` dependency in [environment.yml](environment.yml) to the active LTS version listed on [this page](https://nodejs.org/en/about/previous-releases).\n\n\n### How to update AWS CDK Toolkit\n\nInstall the latest version of AWS CDK Toolkit:\n\n```bash\nnpm install aws-cdk\n```\n\n\n### How to update AWS Construct Library\n\nUpdate *just* the AWS Construct Library (`aws-cdk-lib`):\n\n```bash\npoetry update aws-cdk-lib\n```\n\n\n### How to update Python dependencies\n\nUpdate Python dependencies to their latest versions (according to version constraints in `pyproject.toml`):\n\n```bash\npoetry update\n```\n\n\n### How to update Lambda function Python dependencies\n\nUpdate Lambda function dependencies defined in separate directory:\n\n```bash\ncd cdk/function\npoetry update\ncd -\n```\n\n\n### How to serve Datasette locally\n\nBuild an SQLite database from a Beancount ledger using commands almost identical to those in the [deploy](.github/workflows/deploy.yml) workflow:\n\n```bash\nmicromamba activate portfolio\nbean-sql ../portfolio-ledger/portfolio.beancount cdk/function/portfolio.db\nsqlite3 cdk/function/portfolio.db \u003c ../portfolio-ledger/target_allocation.sql\nsqlite3 cdk/function/portfolio.db \u003c tables.sql\n```\n\nRun the Datasette web application locally using the following command.\nChanges to `metadata.yml` will restart the web application, which is useful when developing dashboards.\nGitHub authentication is not configured as opposed to the production application deployed to AWS.\n\n```bash\ndatasette cdk/function/portfolio.db --reload --metadata cdk/function/metadata.yaml\n```\n\n\n### How to rotate personal access tokens\n\nComplete the following steps after an expiry notification is received.\n\n\n#### portfolio/actions/write\n\n- [Regenerate the token](https://github.com/settings/personal-access-tokens/3189417).\n- Update portfolio-ledger's [TOKEN](https://github.com/eidorb/portfolio-ledger/settings/secrets/actions/TOKEN) repository secret.\n\n\n#### portfolio-ledger/contents/write\n\n- [Regenerate the token](https://github.com/settings/personal-access-tokens/3189407).\n- Update portfolio's [TOKEN](https://github.com/eidorb/portfolio/settings/secrets/actions/TOKEN) repository secret.\n\n\n### How to update ubank device credentials\n\nubank device credentials are stored in AWS Parameter Store.\nEnrol a new device and update the parameter with the following commands:\n```console\n$  python -m ubank name@domain.com --output device.json\nEnter ubank password:\nEnter security code sent to 04xxxxx789: 123456\n$ aws ssm put-parameter \\\n    --name \"/portfolio/ubank-device\" \\\n    --value \"$(\u003c device.json)\" \\\n    --type SecureString \\\n    --overwrite \\\n    --region us-east-1 \\\n    --no-cli-pager\n{\n    \"Version\": 19,\n    \"Tier\": \"Standard\"\n}\n$ rm device.json\n```\n\nCheck the parameter's value with the following command:\n```console\n$ aws ssm get-parameter \\\n    --name \"/portfolio/ubank-device\" \\\n    --with-decryption \\\n    --region us-east-1 \\\n    --output text \\\n    --query 'Parameter.Value' \\\n    --no-cli-pager\n{\n  \"hardware_id\": \"d5c79ef7-8d6a-4feb-b129-a7f54440a348\",\n  \"device_id\": \"85ce55d4-4175-4016-bc37-1b563c680763\",\n  ...\n}\n```\n\n\n## Explanation\n\n### Storing secrets\n\nCredentials used to authenticate with financial institutions are stored in [this repository](portfolio/secrets.py) 😱.\nThis makes it a breeze to develop and test things locally.\n\nEncryption and decryption is handled using [git-crypt](https://www.agwa.name/projects/git-crypt/).\nGitHub Actions workflows decrypt the secrets file when required.\n\n\n### Datasette authentication\n\nThis application serves financial information over the internet.\nThe Datasette plugin [datasette-auth-github](https://datasette.io/plugins/datasette-auth-github)\nis used to control access to the application.\n\nThe OAuth application [Portfolio Datasette](https://github.com/settings/applications/1931808)\nwas registered on GitHub, with *Authorization callback URL* set to `https://portfolio.brodie.id.au/-/github-auth-callback`.\n\nThe [Lambda function](cdk/function/index.py) configures the plugin's OAuth client\nID and secret settings with values retrieved from Parameter Store.\n\nAccess is restricted to my GitHub user ID.\nForbidden requests are redirected to the GitHub auth page using the [datasette-redirect-forbidden](https://datasette.io/plugins/datasette-redirect-forbidden)\nplugin.\n\n\n### Embedding a dashboard in Datasette's index page\n\nPortfolio information is summarised in dashboard charts using the [datasette-dashboards](https://datasette.io/plugins/datasette-dashboards) plugin.\nThis information should be visible on the index page, rather than having to navigate to the dashboard page.\n\ndatasette-dashboards documentation suggests using `\u003ciframe\u003e` elements to embed dashboards and charts in HTML.\nHowever, it was difficult to achieve a responsive layout without scrollbars using this approach.\nInstead, a subset of elements from the dashboard page are included in the index page's HTML.\nDatasette's index page is customised using the `description_html` [metadata](https://docs.datasette.io/en/stable/metadata.html) property.\n\nThe [Datasette CLI](https://docs.datasette.io/en/stable/cli-reference.html#datasette-get) and `extract-dashboard.py` script is used to extract HTML elements from the dashboard page.\n\nThe following command extracts dashboard HTML elements to the clipboard for easy pasting into `metadata.yaml`:\n\n```bash\ndatasette serve \\\n  --get https://portfolio.brodie.id.au/-/dashboards/portfolio \\\n  --metadata cdk/function/metadata.yaml \\\n  cdk/function/portfolio.db | \\\npython extract-dashboard.py | \\\npbcopy\n```\n\nThis command does the same for the demo application:\n\n```bash\ndatasette serve \\\n  --get https://portfolio-demo.brodie.id.au/-/dashboards/portfolio \\\n  --metadata cdk/function/metadata.yaml \\\n  cdk/function/portfolio.db | \\\npython extract-dashboard.py | \\\npbcopy\n```\n\n\n### Lambda function performance\n\nLambda allocates CPU proportional to the amount of memory configured. Request durations of multiple seconds were observed with the default setting of 128 MB. Occasional timeouts occurred when the default timeout of 3 seconds was exceeded.\n\nSetting memory size to 1024 MB resulted in much shorter durations: 100 ms or less. Costs should be comparable or even reduced as we're using more expensive compute but for less time.\n\n\n### GitHub Actions workflows\n\nThis section describes the workflows used by this project.\n\n```mermaid\nflowchart LR\n    dispatch --\u003e deploy\n    subgraph portfolio\n        s2([Schedule]) --\u003e test[\u003cpre\u003etest\u003c/pre\u003e workflow]\n        s([Schedule]) --\u003e update[\u003cpre\u003eupdate\u003c/pre\u003e workflow]\n        p([Push]) --\u003e deploy[\u003cpre\u003edeploy\u003c/pre\u003e workflow]\n    end\n    subgraph \"portfolio-ledger (private)\"\n        p2([Push]) --\u003e dispatch[\u003cpre\u003edispatch\u003c/pre\u003e workflow]\n    end\n```\n\n\n#### [test](.github/workflows/test.yml)\n\nFinancial institution's websites/APIs are subject to change.\nThis workflow runs [pytest tests](tests/) fortnightly.\nA failed test workflow indicates that something on the financial institution's end has changed and the code needs to be fixed.\n\n\n#### [update](.github/workflows/update.yml)\n\nThis workflow updates the Beancount ledger in [portfolio-ledger](https://github.com/eidorb/portfolio-ledger)\nwith the latest balances and asset prices.\n\nIt is scheduled to run approximately every 10 days.\n\n\n#### [deploy](.github/workflows/deploy.yml)\n\nThis workflow converts a Beancount ledger contained in the private [portfolio-ledger](https://github.com/eidorb/portfolio-ledger)\nrepository to an SQLite database and then deploys the CDK application to AWS.\n\nIt is triggered when changes are pushed to this repository or the [portfolio-ledger](https://github.com/eidorb/portfolio-ledger)\nrepository.\n\n\n#### dispatch (in [portfolio-ledger](https://github.com/eidorb/portfolio-ledger))\n\nThis workflow triggers the [deploy](.github/workflows/deploy.yml) workflow when\nchanges are pushed to [portfolio-ledger](https://github.com/eidorb/portfolio-ledger),\nregardless of whether the changes were manual or automatic.\n\n\n### Authenticating GitHub Actions workflows with AWS\n\nMy [AWS account](https://github.com/eidorb/aws) was [configured](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services)\nto trust GitHub's OpenID Connect (OIDC) provider.\nThis allows workflows to deploy to AWS without using long-lived credentials.\n\n\n### Routing GitHub Actions traffic via a Tailscale exit node\n\n\u003e [!NOTE]\n\u003e This hack is no longer required after reverse engineering SelfWealth's mobile API.\n\u003e I'll keep this information here, because it may be required again in the future.\n\nSome financial institution's websites behave differently when accessed from the\nGitHub Actions network, likely due to overly sensitive anti-bot protection.\nCode that would successfully retrieve a balance when run on a computer at home would fail when run on GitHub Actions.\n\nTo work around this, the [deploy](.github/workflows/deploy.yml) and [test](.github/workflows/test.yml)\nworkflows connect to a Tailscale network and route traffic via an exit node at home.\n\n\n## Reference\n\n### GitHub personal access tokens\n\n| Name                                                                                          | Description                                                                                                                                                                                                                                                                            |\n| --------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| [portfolio-ledger/contents/write](https://github.com/settings/personal-access-tokens/3189407) | Grants `contents:write` access to [portfolio-ledger](https://github.com/eidorb/portfolio-ledger) repository. Used by [update](.github/workflows/update.yml) workflow to update Beancount ledger, and by [deploy](.github/workflows/deploy.yml) workflow to check out portfolio-ledger. |\n| [portfolio/actions/write](https://github.com/settings/personal-access-tokens/3189417)         | Grants `actions:write` access to this repository. Used by [portfolio-ledger](https://github.com/eidorb/portfolio-ledger)'s dispatch workflow to trigger this repository's [deploy](.github/workflows/deploy.yml) workflow.                                                             |\n\n\n### GitHub Actions secrets\n\nThe following secrets were created in the [repository](https://github.com/eidorb/portfolio/settings/secrets/actions):\n\n| Name                                                                                                                        | Description                                                                                                           |\n| --------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |\n| [GIT_CRYPT_KEY](https://github.com/eidorb/portfolio/settings/secrets/actions/GIT_CRYPT_KEY)                                 | Used by [git-crypt](https://www.agwa.name/projects/git-crypt/) to decrypt secret repository files.                    |\n| [TAILSCALE_OAUTH_CLIENT_SECRET](https://github.com/eidorb/portfolio/settings/secrets/actions/TAILSCALE_OAUTH_CLIENT_SECRET) | Used in update and test workflows to connect to a [Tailscale network](https://tailscale.com/kb/1111/ephemeral-nodes). |\n| [TOKEN](https://github.com/eidorb/portfolio/settings/secrets/actions/TOKEN)                                                 | [portfolio-ledger/contents/write](https://github.com/settings/personal-access-tokens/3189407) personal access token.  |\n\n\n### AWS Parameter Store parameters\n\nThe Lambda function retrieves the following parameters from Parameter Store on startup:\n\n| Name                            | Description                             |\n| ------------------------------- | --------------------------------------- |\n| /portfolio/datasette-secret     | Key used to sign Datasette cookies.     |\n| /portfolio/github-client-id     | GitHub OAuth application client ID.     |\n| /portfolio/github-client-secret | GitHub OAuth application client secret. |\n| /portfolio/ubank-device         | Enrolled ubank device credentials.      |\n\nParameters are stored in the `us-east-1` region.\n\nList parameters associated with this project with the following command:\n\n```bash\naws ssm describe-parameters --region us-east-1 --parameter-filters \"Key=tag:project,Values=portfolio\" --query 'Parameters[*].[Name,Type]' --output text --no-cli-pager\n\n/portfolio/datasette-secret     SecureString\n/portfolio/github-client-id     String\n/portfolio/github-client-secret SecureString\n/portfolio/ubank-device SecureString\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feidorb%2Fportfolio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feidorb%2Fportfolio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feidorb%2Fportfolio/lists"}