{"id":17116473,"url":"https://github.com/dakimura/bot","last_synced_at":"2025-03-24T00:52:34.825Z","repository":{"id":137757555,"uuid":"544619582","full_name":"dakimura/bot","owner":"dakimura","description":null,"archived":false,"fork":false,"pushed_at":"2022-10-23T06:45:06.000Z","size":3028,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-29T07:27:03.949Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dakimura.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-02T23:08:04.000Z","updated_at":"2022-11-09T06:36:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"db151a4a-cbc2-4d9a-9c46-789f4e63e729","html_url":"https://github.com/dakimura/bot","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dakimura%2Fbot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dakimura%2Fbot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dakimura%2Fbot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dakimura%2Fbot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dakimura","download_url":"https://codeload.github.com/dakimura/bot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245191622,"owners_count":20575248,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T17:48:56.705Z","updated_at":"2025-03-24T00:52:34.804Z","avatar_url":"https://github.com/dakimura.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bot environment setup\n\nIn order to operate an automated trading bot, it is necessary to prepare a mechanism that \nallows the bot to automatically run without the operator having to start and stop it manually.\nThis is one of the high hurdles for those who want to build their own bot but are not experts in infrastructure or networking. \nEven if they know effective machine-learning algorithms, they may find it difficult to put them into actual system operation.\n\nIn this repository, We explain how to build a simple environment for running a bot, for free.\nEven if you do not understand some network/infrastructure/cloud computing terms,\nwe aim for you to be able to start bot operation without paying cost for the initial server by following the steps.\n\nOf course, we are aware that this is not enough to operate a full-fledged, production-scale service.\nDatabases, log collection, a mechanism to send alerts when the server is down, CD/CI, and so on would be necessary.\nHowever, we hope this procedure will be helpful for those who are taking their first steps in automatic trading.\n\n# Environment created by this instruction\n\n![](img/system_arch.png)\nhttps://docs.google.com/presentation/d/1SyGSNEX8N3V8m4CkujqgTyiVN-Eeu1zUN4UCOVIs5Ss/edit?usp=sharing\n\n## Create(Fork) a Github repository for the bot\n\nTo store the code for the bot and network/infra configuration,\na [Github](https://github.com/) repository is needed.\nI created dakimura/bot repository.\nYou can start from forking this repo.\nIf you don't want to make the repository public, \nyou can create an empty repository and copy-paste the contents of this repository to it.\n\n## Sign up for Google Cloud Platform\n\nTo run the bot for 24/7, you need a server on which to run the bot program.\nHere we will use Google Cloud Platform (GCP). Of course, other rental servers or cloud services such as AWS and Azure are acceptable, \nbut As of October 2022, GCP has a free tier that allows you to keep running a small(e2-micro) instance for free for as long as you want,\nup to the maximum amount each month. Other services also offer free allowances, \nbut AFAIK GCP is the only one that keeps the free allowance for a long time.\n\nSign up for Google Cloud Platform from [here](https://www.google.com/cloud)\n\n### Create a project on Google Cloud Platform\n\nTo create a resource such as an Google Compute Engine instance, \nyou must first create a \"project\" on Google Cloud Platform.\nOpen the Google Cloud console https://cloud.google.com' \u003e cloud-console,\nsign in and Create a new project. I have created a project named `trade-bot`.\n\n### Install gcloud (Google Cloud SDK command line interface)\n\nPlease follow the steps:\nhttps://cloud.google.com/sdk/docs/install\n\nMake sure the `gcloud` command is available from Terminal.\n\n### Sign in Google Cloud and create a service account\nA service account is a special type of Google account intended to represent \na non-human user that needs to authenticate and be authorized to access data in Google APIs.\n\nIn the steps that follow, we will build our server on the Google Cloud Platform using a service called Terraform.\nTo automatically build a server using Terraform on your Google Cloud Platform account, \n you need to pass the service account information to Terraform.\n\nLet's use the `gcloud auth login` command to first make sure that the gcloud command is properly installed.\nYou have succeeded if the screen [You are now authenticated with the gcloud CLI!](https://cloud.google.com/sdk/auth_success) \nappears in your browser after logging in to your Google account.\n\nLet's create a service account by referring to https://cloud.google.com/iam/docs/creating-managing-service-accounts.\n\nConfirm the GCP project you created and set the project. \nPlease replace the project name(trade-bot-364400) below with your own,\n```bash\n $ gcloud projects list\nPROJECT_ID              NAME                  PROJECT_NUMBER\ntrade-bot-364400        trade-bot             406377578255\n\n$ gcloud config set project trade-bot-364400\nUpdated property [core/project].\n```\n\nand create a service account by the following command.\n```bash\n$ gcloud iam service-accounts create trade-bot \\\n--description=\"ServiceAccount for my trading bot\" \\\n--display-name=\"trade-bot\"\n\nCreated service account [trade-bot].\n```\n\nGrant the `roles/owner` role (privilege) to the service account by the following command.\nThis is a very strong privilege, please be very careful when handling the service account.\nPlease replace the project name(trade-bot-364400) below with your own,\n\n```bash\n$ gcloud projects add-iam-policy-binding trade-bot-364400 \\\n--member=\"serviceAccount:trade-bot@trade-bot-364400.iam.gserviceaccount.com\" \\\n--role=roles/owner\n\nUpdated IAM policy for project [trade-bot-364400].\nbindings:\n- members:\n  - serviceAccount:trade-bot@trade-bot-364400.iam.gserviceaccount.com\n  - user:example@example.com\n  role: roles/owner\netag: XXXXXXXXXXXX\nversion: 1\n```\n\nTo pass the service account information, create a key and download it to your localhost.\nhttps://cloud.google.com/iam/docs/creating-managing-service-account-keys#iam-service-account-keys-create-gcloud\n\n(Please replace the local file path, service account name, and project ID to your own)\n```bash\n$ gcloud iam service-accounts keys create /Users/dakimura/Desktop/sa-trade-bot-private-key.json \\\n    --iam-account=trade-bot@trade-bot-364400.iam.gserviceaccount.com\n    \ncreated key [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] of type [json] as [/Users/dakimura/Desktop/sa-trade-bot-private-key.json] for [trade-bot@trade-bot-364400.iam.gserviceaccount.com]\n```\nDO NOT share the downloaded private key file with anyone.\n\nAlso, you cannot download the private key file again. If you accidentally lose the file or share the file with someone,\nplease immediately delete it by the following command, and create another one:\n```bash\n$gcloud iam service-accounts keys delete /Users/dakimura/Desktop/sa-trade-bot-private-key.json --iam-account=trade-bot@trade-bot-364400.iam.gserviceaccount.com\n```\n\n## Sign up for Terraform Cloud\nTo configure the bot server automatically, we use Terraform Cloud.\nPlease register an account with [Terraform Cloud](https://app.terraform.io/public/signup/account).\nand confirm your e-mail address after the registration.\n\n### Create a workspace in Terraform Cloud\nCreate a workspace by following the steps:\n\nhttps://www.terraform.io/cloud-docs/workspaces/creating#create-a-workspace.\n\nPlease [Login to Terraform Cloud](https://app.terraform.io/app).\nWhen your infrastructure config is updated, it should be automatically applied to the infrastructure.\nTerraform cloud can update the infrastructure when your code on GitHub is updated.\nLet's integrate Github when creating a new workspace.\n\nWorkspaces -\u003e \"+ New Workspace\" -\u003e \"Version Control Workflow\" -\u003e \"Connect to a different VCS\"\n-\u003e Click the down-arrow button next to \"Github\" -\u003e Click \"GitHub.com(Custom)\"\n-\u003e Setup Provider by following the instruction.\n\n![](img/setup_provider.png)\n\n-\u003e Click \"Connect and Continue\"\n\nAfter successfully integrating Terraform cloud app to Github,\nYou'll see the following screen:\n![](img/tf_github_integrated.png)\n\nLet's go back to the screen to create a workspace, and now you would be able to see the button with the integrated GitHub app.\n![](img/after_github_integ.png)\nClick it and choose this repository, and create a workspace.\n![](img/choose_repo.png)\n\nYou would be asked to set PROJECT_ID and GOOGLE_CREDENTIALS variables but skip it for now\nand move to the next step.\n\n### Register GCP Project ID \u0026 Service account key to Terraform Cloud as Environmental Variables\nAfter creating a workspace, we register the GCP Project ID and downloaded GCP service account key to Terraform cloud.\nGo to the workspace you created.\n![](img/workspaces.png)\n\nClick \"Varibale\" from the left pane, Choose \"Add Variable\", and check \"Environmental Variable\".\n\n![](img/reg_env_var_tf_cloud.png)\nSet Key=\"GOOGLE_CREDENTIALS\", Value=the content of the service account file you created.\n\nEnvironmental Variable does not accept new lines, so please remove the value by the following command before entering.\n(If)\n```bash\n$ brew install jq\n$ jq -c \u003c /Users/dakimura/Desktop/sa-trade-bot-private-key.json\n```\nPlease check on `Sensitive`, and save the variable. Otherwise, the service account key value will be visible \nto other users who can see your workspace.\n\n\nIn the same way, add \n\nKey=\"PROJECT_ID\", Value=trade-bot-364400\"\n\n(Replace the project ID with your own)\n![](img/set_variable.png)\n\nClick \"Save Variable\".\n\n![](img/variables.png)\nNow you have 2 environmental variables.\n\n### Run Terraform (apply)\nGo to your workspace page -\u003e Click \"Runs\" -\u003e Click \"Actions\" -\u003e \"Start new run\"\nAfter Planning and Cost Estimation is done, click \"Confirm \u0026 Apply\"\n-\u003e Click \"Confirm Plan\"\n![](img/tf_confirm_apply.png)\n\nPlease follow the error message if you get the following:\n![](img/apply_error.png)\nGo to the [API dashboard](https://console.cloud.google.com/apis/dashboard) \nfor your project and search the APIs that are not enabled -\u003e enable API\n\n\nOnce the Run is succeeded, you will see the following:\n![](img/run_succeeded.png)\n\n### Enable Auto Apply\nIt's time-consuming to manually confirm each apply, let's enable Auto Apply for Terraform Cloud.\nGo to Workspace page on Terraform Cloud -\u003e Setting -\u003e General \n-\u003e check \"Auto apply\" under Apply Method section.\n![](img/apply_method.png)\nClick \"Save settings\".\n\n## Setup Github Actions (CD/CI)\nIt is time-consuming to deploy the application every time you update it.\nBy integrating [Github Actions](https://docs.github.com/en/actions) with your repository,\nyou can automatically deploy the application by just pushing the code to the main branch.\nIn this section, we set it up using the [workflow file](./github/workflows/main.yml)\n\n### Get Alpaca Key ID and API Secret\nIn the bot application, Alpaca platform is used to get data and trade by default.\nBefore deploying the application to your server,\nlet's [sign up Alpaca for free from here](https://alpaca.markets/).\n\nAlpaca has [Paper Trading mode](https://alpaca.markets/docs/trading/paper-trading/).\nLet's start the bot with the simulation mode.\nAfter sign in to Alpaca, You can see your API Keys on the [paper trade dashboard](https://app.alpaca.markets/paper/dashboard/overview)\n\n![](img/paper_trade_api_keys.png)\n\nPlease create your API key/secret for the paper trades.\n\n### Set API key/secret as your github repository secrets\nTo ask Github Actions to deploy your apps, we need to pass the Alpaca API Key/Secret\nto Github.\nPlease go to your Github Repository -\u003e Settings -\u003e Secrets -\u003e Actions\n\n![](../../../../../Desktop/actions_secret.png)\n\nCreate `ALPACA_API_KEY_ID` and `ALPACA_API_SECRET_KEY` secrets on your repository.\n\n### Modify the workflow file\nGithub automatically reads `./github/workflows/main.yml` and triggers the workflow.\nOpen the file and change the GCP project name with your own on the following line\n```\n          service_account: 'github-actions@trade-bot-364400.iam.gserviceaccount.com'\n```\n\nThen, push the change to the remote.\nGithub automatically つぎここから\n\n## Run the bot application\n\n### SSH and run\n\nNow you have a server instance.\nLet's SSH-login and run the bot.\n\nYou can SSH-login to the instance by using the following command:\n(Replace the project name and instance name with your own.)\n```bash\n$ gcloud compute --project \"trade-bot-364400\" ssh --zone \"us-central1-c\" \"bot-instance\"\n\nWarning: Permanently added 'compute.1***5' (***) to the list of known hosts.\nLinux bot-instance ***-cloud-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64\n\nThe programs included with the Debian GNU/Linux system are free software;\nthe exact distribution terms for each program are described in the\nindividual files in /usr/share/doc/*/copyright.\n\nDebian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent\npermitted by applicable law.\ndakimura@bot-instance:~$ sudo -s\nroot@bot-instance:/home/dakimura#\nroot@bot-instance:/home/dakimura# apt install pip\n```\nFor now, let's copy-paste the source files and requirements.txt, \n\nand directly run the bot.\n\n```bash\n$ pip install -r requirements.txt\n# run as a background process\n$ nohup streamlit run --server.port=80 dashboard.py \u003e\u003edashboard.log 2\u003e\u00261 \u0026\n$ cat dashboard.log\nnohup: ignoring input\n\nCollecting usage statistics. To deactivate, set browser.gatherUsageStats to False.\n\n\n  You can now view your Streamlit app in your browser.\n\n  Network URL: http://***:80\n  External URL: http://***:80\n```\n\nYou can see `External URL: http://***:80`\nat the end of the log. Let's access the URL.\n\n![](img/bot_dashboard.png)\nCongrats!\nNow you are running your bot, and you can see the dashboard via the Internet.\n\n## Continuous Delivery using Github Actions\nIt's time-consuming to ssh-login and pull the latest code and restart the application.\nLet's automate it using [Github Actions](https://github.co.jp/features/actions).\n\nGo to Terraform Cloud -\u003e Workspace -\u003e Set Terraform Variable:\nKey=REPO_NAME, Value=dakimura/bot (replace with your own repo name)\n\nGo to the [dashboard of your GCP project](https://console.cloud.google.com/home/dashboard) and check the project name and number.\n![](img/gcp_dashboard.png)\nCopy `.github/` directory of this repository to yours, open `main.yml`(=terraform script), and change the following lines to your project name and number.\n\n```bash\nworkload_identity_provider: 'projects/{YOUR PROJECT ID}/locations/global/workloadIdentityPools/gh-oidc-pool/providers/github-actions'\nservice_account: 'github-actions@{YOUR PROJECT NUMBE}.iam.gserviceaccount.com'\n```\n\n-\u003e commit and push the change.\nWorkload Identity will be configured for the github-actions service account for your GCP project.\nFor now, you don't have to understand what Workload Identity and Service Account are, please think it as a permission to let Github Actions to edit your GCP resources.\n\n## FAQ\n\n### What are Terraform and Terraform cloud, and why are they necessary?\n\nTerraform is an open-source infrastructure as code(IaC) software tool that\nenables you to safely and predictably create, change, and improve infrastructure.\nHere we are going to build a server for the bot, but probably you don't want to care about:\n- Does the server cost a lot?\n- Is the server opening a proper port?\n- Can I delete the server when I want to?\n\nYou don't have to worry because all those kind of settings are stored in the text files (.tf files) on this repository.\nYou can reproduce the infrastructure anytime without learning much about Google Cloud.\n\nIn short, Terraform will manage the state of your server and network settings.\nTo store the state, Terraform needs to be integrated with your Github repository and GCP account.\nTerraform Cloud does it, and you don't have to care where the Terraform commands are run.\n\nAs of October 2022, Terraform Cloud is free to use for up to 5 users with unlimited number of workspaces. \nhttps://www.hashicorp.com/products/terraform/pricing\n\nOf course, you can build a server and network infrastructure at GCP console(=Web UI), \nso Terraform is not a must if you know how to do it.\n\n### Your python code uses Streamlit. Why don't you use Streamlit Cloud?\n\nhttps://discuss.streamlit.io/t/does-streamlit-has-a-maximum-run-time-for-functions/22452/3\nStreamlit is intended for web applications such as dashboard feature, and it has processing timeouts.\nSo we determined that it is not suitable for bot applications that persistently retrieve data and make trades without any action from the user.\n\n### Install Terraform command line interface\n\nChoose \"Try an example configuration\" at https://app.terraform.io/app/getting-started\nand install Terraform command by following the instruction.\nhttps://app.terraform.io/app/getting-started/example\n\nI'm using mac, so installed it by the following 2 commands:\n```bash\n$ brew tap hashicorp/tap\n$ brew install hashicorp/tap/terraform\n\n$ terraform --version\nTerraform v1.3.1\non darwin_arm64\n```\n\n\n### Don't we use Docker?\nActually, we should use Docker.\nBut it's time-consuming to register the docker image to Google Container Registry and \nconfigure the permission to pull the image from the GCE instance, so we think it as a nice-to-have but not must-have thing.\n\nAlso, because we are using a tiny([e2-micro](https://cloud.google.com/compute/vm-instance-pricing?hl=ja#e2_sharedcore_machine_types)) instance,\nso we cannot use much resource to the Docker daemon.\n\nBut, again, we should use Docker ideally.\n\n### Don't we register the bot as a systemctl service?\nFor simplicity, we didn't explain how to do it.\nYou can do so if you know how to do it.\n\n### Don't we use Workload Identity Federation for Terraform Cloud?\nApparently Workload Identity Federation is not supported by Terraform Cloud yet.\nhttps://discuss.hashicorp.com/t/workload-identity-federation-to-integrate-terraform-cloud-with-gcp/40334","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdakimura%2Fbot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdakimura%2Fbot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdakimura%2Fbot/lists"}