{"id":17300419,"url":"https://github.com/atomantic/stock_signals","last_synced_at":"2025-10-17T20:46:29.492Z","repository":{"id":42360191,"uuid":"142647062","full_name":"atomantic/stock_signals","owner":"atomantic","description":"💹    my own stock data collection engine, saving a bunch of data for a Google Spreadsheet process. UPDATE: this is no longer in use or maintained but could be academically interesting so I'll leave it here.","archived":false,"fork":false,"pushed_at":"2022-12-07T23:27:21.000Z","size":11145,"stargazers_count":7,"open_issues_count":6,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-28T01:48:10.874Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/atomantic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-28T04:36:20.000Z","updated_at":"2022-01-03T19:16:58.000Z","dependencies_parsed_at":"2022-09-21T17:51:41.325Z","dependency_job_id":null,"html_url":"https://github.com/atomantic/stock_signals","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atomantic%2Fstock_signals","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atomantic%2Fstock_signals/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atomantic%2Fstock_signals/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atomantic%2Fstock_signals/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/atomantic","download_url":"https://codeload.github.com/atomantic/stock_signals/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248881302,"owners_count":21176827,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T11:28:18.772Z","updated_at":"2025-10-17T20:46:29.389Z","avatar_url":"https://github.com/atomantic.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003e NOTICE: I am not a financial or trading advisor. This tool comes without any warrenty or assurances. None of the stock tickers listed or the results found within this codebase indicate a recommendation to buy or sell assets. Use at your own risk.\n\n# Technical Signal Caching Engine\n\nI use this project to get signals from TradingView, cache them on myjson, and ingest them into a Google Sheets to alert me of whether I should buy or sell particular holdings (based on whether my current position in those assets is underwater or in profit).\n\nMy general starting strategy is something like this (but has been evolving):\n\n- If the meta indicator (summary of summaries over time periods) says `Buy`\n    - If my cost-basis is lower than current price: `buy`\n    - if my cost-basis is higher than the current price, `buy` only at regular intervals (1 share per week/month/etc), else `hold` if I've already bought recently\n- If the meta indicator says `Sell`\n    - If my cost-basis is lower than current price: `hold`\n    - if my cost-basis is higher than the current price: `sell`\n- Neutral = `hold`\n- `Strong Sell` and `Strong Buy` signals just increase the color coding darkness in the Google Sheet, which informs me of the quanitity I might wish to buy. I might sell half on a `Sell` and sell all on a `Strong Sell`.\n\nAdditionally, these indicators are still only prompting suggestions that require human evaluation. I consider other factors such as insider trading (e.g. insiders buying a bunch of stock in a mining company: https://www.secform4.com/insider-trading/1600470.htm) before making a decision. Sometimes, I still open TradingView to manually eyeball the RSI, Stochastic RSI, Ultimate Oscillator, Ichi Cloud, etc.\n\nI might post the Google Sheets template logic later...\n\n# Local Setup\n```\n# run setup\nnpm run setup\n```\n## Local Run\n```\nnpm start\n```\n![running](img/running.png)\n\n# Data\nFor all logs and db files, the following legend maps TradingView Summary Signal status to numeric values:\n\n- `Strong Buy` = 2\n- `Buy` = 1\n- `Neutral` = 0\n- `Sell` = -1\n- `Strong Sell` = -2\n\n## Results\n\n`data/latest.json` is a snapshot of the latest runs for each ticker. It is only the most recent run without historical context. This is used by the Google Sheet to determine the current state. This set maintains the last different state each summary was in:\n```javascript\n{\n    \"tickers\": {\n      \"NASDAQ-EKSO\": {\n      \"ma\": [\n        [[0, 1]], // last value, 4 hour Hull MA (numeric indicator)\n        [[0, 1]], // 1 day Hull MA\n        [[0, 1]], // 1 week Hull MA\n        [[0, 1]]  // 1 month Hull MA\n      ],\n      \"osc\": [\n        [ // 4 hour oscillators\n          [54.83], // rsi\n          [0.98], // stoch rsi\n          [32.13], // stoch\n          [34.48] // ultimate osc\n          [1], // macd numeric directional indicator\n        ],\n        [ // 1 day oscillators\n          [60.53],\n          [64.05],\n          [0.26],\n          [57.64],\n          [1]\n        ],\n        [ // 1 week oscillators\n          [65.11],\n          [73.07],\n          [0.18],\n          [95.43],\n          [1]\n        ],\n        [ // 1 month oscillators\n          [46.35],\n          [27.77],\n          [-1.22],\n          [93.2],\n          [1]\n        ]\n      ],\n        \"price\": 2.72,\n        \"time\": 1534604640049, // unix timestamp\n        \"sum\": [1,2,1,1], // summary values for periods (4 hour, 1 day, 1 week, 1 month)\n        \"prev\": [2,1,2,1], // last different values in those windows\n        \"meta\": 1, // current meta summary signal accross time periods\n        \"from\": 2 // last different summary\n      }\n    },\n    \"time\": 1534611504984,\n    \"last\": \"NASDAQ-CIBR\"\n  }\n```\n\n## Signals Files\n\nAdditionally, we save every single hit to a rolling log file, which we can use for backtesting strategies, and to see if the current signal is either moving up or down (direction is important). We use the last known state to know if the signal is either retreating or moving forward.\n\nThe rolling logs are capped at 100MB and are saved as minimal csv files using numeric indicators to create smaller files.\n\nThe column headers for the files are the following:\n\n```\nframe.columns = [\"ticker\",\"price\",\"change\",\"time\",\"meta_signal\",\"meta_previous\",\n                \"4h\",\"1d\",\"1w\",\"1m\",\n                \"4hp\",\"1dp\",\"1wp\",\"1mp\",\n                \"4h_rsi\",\"4h_stochrsi\",\"4h_stoch\",\"4h_ult\",\"4h_macd\",\"4h_hull\",\"4h_ma\",\n                \"1d_rsi\",\"1d_stochrsi\",\"1d_stoch\",\"1d_ult\",\"1d_macd\",\"1d_hull\",\"1d_ma\",\n                \"1w_rsi\",\"1w_stochrsi\",\"1w_stoch\",\"1w_ult\",\"1w_macd\",\"1w_hull\",\"1w_ma\",\n                \"1m_rsi\",\"1m_stochrsi\",\"1m_stoch\",\"1m_ult\",\"1m_macd\",\"1m_hull\",\"1m_ma\",\n                \"4h_divergence\",\"1d_divergence\",\"1w_divergence\",\"1m_divergence\"\n                ]\n```\n\nThe signal database creates 8KB per full run.\nAny results that exactly match the previous result (nothing changed in price or signals), will not be saved to the rolling DB file. This means that generally all day Saturday and Sunday, there will be no new db activity since the markets are closed. It's probably best to not run this on the weekends or late at night when it won't add any value.\nEach run takes ~1 hour.\nThis should create ~50MB per year worth of data\n\n\n## Cleaning Errors\n\n### Jupyter Notebook\nAs a quick interactive, rapid-dev-feedback practice, I usually start python data ETL tasks in Jupyter to quickly visualize it (also allows using matplotlib UI). My process for sanitizing the data currently is in a Jupyter notebook:\n```\ncd data;\ndocker run -it -v $(pwd):/home/jovyan --rm -p 8888:8888 jupyter/scipy-notebook\n```\nThis outputs `./data/signals.csv`, which is the combined, sanitized database file of unique data on each ticker over time. This can then be thrown into BigQuery/DynamoDB or some other system to run ML, draw charts, or run other queries.\n\n## Sync to Google Cloud Storage\n```\ngsutil rsync -d data gs://stock-tickers/data\n```\n\n# BigQuery ML\n\n## Load into BigQuery\n```\n# from cloud storage\nbq mk ticker_tracker\nbq load --source_format=CSV --location=US ticker_tracker.signals gs://stock-tickers/data/signals.csv \\\n  ticker:string,price:float,change:float,time:integer,\\\n  meta_signal:integer,meta_previous:integer,\\\n  h:integer,d:integer,w:integer,m:integer,\\\n  hp:integer,dp:integer,wp:integer,mp:integer,\\\n  h_rsi:float, d_rsi:float, w_rsi:float, m_rsi:float,\\\n  h_stochrsi:float, d_stochrsi:float, w_stochrsi:float, m_stochrsi:float,\\\n  h_stoch:float, d_stoch:float, w_stoch:float, m_stoch:float,\\\n  h_ult:float, d_ult:float, w_ult:float, m_ult:float,\\\n  h_macd:integer, d_macd:integer, w_macd:integer, m_macd:integer,\\\n  h_hull:integer, d_hull:integer, w_hull:integer, m_hull:integer,\\\n  h_ma:integer, d_ma:integer, w_ma:integer, m_ma:integer,\\\n  move:integer,\\\n  exchange:string,name:string\n```\n\n## Linear Regression Model\n\ncreate model\n```\nCREATE MODEL\n  `signals.model_aapl_price`\nOPTIONS\n  ( model_type='linear_reg',\n    ls_init_learn_rate=.15,\n    l1_reg=1,\n    max_iterations=5,\n    data_split_method='seq',\n    data_split_eval_fraction=0.3,\n    data_split_col='time' ) AS\nSELECT\n  meta_signal,\n  meta_previous,\n  h,d,w,m,\n  hp,dp,wp,mp,move,\n  h_stochrsi, d_stochrsi, w_stochrsi, m_stochrsi,\n  h_stoch, d_stoch, w_stoch, m_stoch,\n  h_ult, d_ult, w_ult, m_ult,\n  h_macd, d_macd, w_macd, m_macd,\n  h_hull, d_hull, w_hull, m_hull,\n  h_ma, d_ma, w_ma, m_ma,\n  time,\n  price\nFROM\n  `ticker_tracker.signals`\nWHERE\n  name = \"AAPL\"\n```\n\npredict\n```\nSELECT\n  *\nFROM\n  ML.PREDICT(MODEL `signals.model_aapl_price`,\n    (\n    SELECT\n      price,\n      time,\n      meta_signal,\n      meta_previous,\n      h,d,w,m,\n      hp,dp,wp,mp,move,\n    FROM\n      `ticker_tracker.signals`\n    WHERE\n      name = \"AAPL\"))\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fatomantic%2Fstock_signals","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fatomantic%2Fstock_signals","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fatomantic%2Fstock_signals/lists"}