{"id":15662912,"url":"https://github.com/johnhof/gdax-candle-extractor","last_synced_at":"2025-05-06T01:54:45.873Z","repository":{"id":148538097,"uuid":"93798273","full_name":"johnhof/gdax-candle-extractor","owner":"johnhof","description":" Simple historical candlestick extraction from the GDAX API","archived":false,"fork":false,"pushed_at":"2017-11-27T23:26:21.000Z","size":30,"stargazers_count":19,"open_issues_count":4,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-06T01:54:40.903Z","etag":null,"topics":["bitcoin","candlestick","candlestick-chart","ethereum","gdax"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/johnhof.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-08T22:55:24.000Z","updated_at":"2023-12-05T19:10:47.000Z","dependencies_parsed_at":"2023-05-20T11:00:31.219Z","dependency_job_id":null,"html_url":"https://github.com/johnhof/gdax-candle-extractor","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnhof%2Fgdax-candle-extractor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnhof%2Fgdax-candle-extractor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnhof%2Fgdax-candle-extractor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnhof%2Fgdax-candle-extractor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/johnhof","download_url":"https://codeload.github.com/johnhof/gdax-candle-extractor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252606926,"owners_count":21775413,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitcoin","candlestick","candlestick-chart","ethereum","gdax"],"created_at":"2024-10-03T13:34:53.923Z","updated_at":"2025-05-06T01:54:45.822Z","avatar_url":"https://github.com/johnhof.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GDAX Candlestick Extractor\n\nThis package is designed to simplify historical [candlestick](http://www.investopedia.com/terms/c/candlestick.asp) data extraction from the GDAX API, for the purposes of analysis and  model training.\n\nThe extractor itself is abstracted to allow use in existing golang codebases in addition to the command line util.\n\nThis utility supports outputting data to the following collection tools:\n  * stdout\n  * CSV file\n  * JSON file\n  * Newline delimited JSON file\n  * Elasticsearch index\n\n`go get github.com/johnhof/gdax-candle-extractor`\n\n## Command line usage\n\n`$ gdax-candle-extractor --key=KEY --secret=SECRET --passphrase=PASSPHRASE --product=PRODUCT [\u003cflags\u003e]`\n\n**Get candlestick data for each hour since the beginning of 01/01/17, and pipe it to a csv file**\n\n`$ gdax-candle-extractor -start=2017-01-01T00:00:09+00:00 -granularity=3600 -out-csv -out-csv-file=./data.csv`\n\n## Docker usage\n\nEither \n\n`git clone git@github.com:johnhof/gdax-candle-extractor.git \u0026\u0026 docker build gdax-candle-extractor -t johnhof/gdax-candle-extractor`\n\nOr\n\n`docker pull johnhof/gdax-candle-extractor`\n\n```bash\ndocker run \\\n-e GDAX_API_KEY=foo_key \\\n-e GDAX_API_SECRET=foo_secret \\\n-e GDAX_API_PASSPHRASE=foo_phrase \\\n-e GDAX_EXTRACTOR_PRODUCT=ETH-USD \\\n-e GDAX_EXTRACTOR_VERBOSE=true \\\njohnhof/gdax-candle-extractor \n\n```\n\n### Options\n\nThe following options are the result of `--help`. The text is modified to include environment var alternatives which will override the defaults, but not command line params.\n\n```bash\n      --help                                                                context-sensitive help (also try --help-long and --help-man).\n  -v, --verbose,          GDAX_EXTRACTOR_VERBOSE                            verbose logging\n  -k, --key,              GDAX_API_KEY=KEY                                  GDAX API key\n  -s, --secret,           GDAX_API_SECRET=SECRET                            GDAX API secret\n  -p, --passphrase,       GDAX_API_PASSPHRASE=PASSPHRASE                    GDAX API passphrase\n      --product,          GDAX_EXTRACTOR_PRODUCT=PRODUCT                    Product ID to extract [BTC-USD, ETH-USD, LTC-USD]\n  -G, --granularity,      GDAX_EXTRACTOR_GRANULARITY=86400                  Granularity in seconds of blocks in the candlestick data\n  -b, --buffer-size,      GDAX_EXTRACTOR_BUFFER_SIZE=100                    Size of candlestick buffer waiting for collection\n  -S, --start,            GDAX_EXTRACTOR_START=\"2017-10-31T00:11:58-07:00\"  Start time as RFC3339\n  -E, --end,              GDAX_EXTRACTOR_END=\"2017-11-06T23:11:58-08:00\"    End time in as RFC3339\n      --out-stdout,       GDAX_EXTRACTOR_OUT_STDOUT                         Write output to stdout. Used by default if no other output is specified\n      --out-csv,          GDAX_EXTRACTOR_OUT_CSV                            Write output to CSV file\n      --out-csv-file,     GDAX_EXTRACTOR_OUT_CSV_FILE=\"out.csv\"             Set the file to write to\n      --out-json,         GDAX_EXTRACTOR_OUT_JSON                           Write output to JSON file\n      --out-json-file,    GDAX_EXTRACTOR_OUT_JSON_FILE=\"out.json\"           Set the file to write to\n      --out-nd-json,      GDAX_EXTRACTOR_OUT_ND_JSON                        Write output to new line delimited JSON file\n      --out-nd-json-file, GDAX_EXTRACTOR_OUT_ND_JSON_FILE=\"out.ndjson\"      Set the file to write to\n      --out-es,           GDAX_EXTRACTOR_OUT_ES                             Index output to elasticsearch\n      --out-es-index,     GDAX_EXTRACTOR_OUT_ES_INDEX=\"candlestick\"         Elasticsearch index to use for output\n      --out-es-host,      GDAX_EXTRACTOR_OUT_ES_HOST=\"localhost\"            Set the elasticsearch host to write to\n      --out-es-port,      GDAX_EXTRACTOR_OUT_ES_PORT=\"9200\"                 Set the elasticsearch port to write to\n      --out-es-secure,    GDAX_EXTRACTOR_SECURE                             Set the elasticsearch requests to use https\n      --version                                                             Show application version.\n\n```\n\n## Programmatic usage\n\nThe source is comprised of two discrete steps\n- Extraction\n  - Create an extractor\n  - Run the extractor\n    - Begins extraction using the provided config\n    - Pushes candlestick data onto the candlestick channel\n    - Pushes retrieval errors onto the error channel\n\n- Collection\n  - Either\n    - Build your own logic reading Candlestick's directly from the channel\n    - Use the provided collector\n      - Either\n        - Build a custom receiver and add it to the collector\n        - Use one of the provided receivers\n\n```go\npackage main\n\nimport (\n\tmyReceivers \"./custom-receivers\"\n\t\"github.com/johnhof/gdax-candle-extractor/extractor\"\n\t\"github.com/johnhof/gdax-candle-extractor/receivers\"\n}\n\nfunc main() {\n\t// Create the extractor\n\textract := extractor.New(\u0026extractor.ExtractorConfig{\n\t\tKey:        \"SuperSecretGDAXKey\",\n\t\tSecret:     \"SuperSecretGDAXSecret\",\n\t\tPassphrase: \"SuperSecretGDAXPassphrase\",\n\t\tExtraction: \u0026extractor.ExtractionConfig{\n\t\t\tProduct:     \"BTC-USD\", // Bitcoin price in US dollars\n\t\t\tGranularity: 5, // candlesticks split by  5 second chunks\n\t\t\tStart:       time.Now().Sub(24*time.Hour), // from yesterday\n\t\t\tEnd:         time.Now(), // until now\n\t\t},\n\t})\n\n\t// Start extracting\n\terr := extract.Extract()\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\t// Create a collector to simplify candlestick collection from the extractor\n\tc := extractor.NewCollector(\u0026extractor.CollectorConfig{\n\t\tExtractor: extract,\n\t})\n\n\t// Create a CSV receiver to write to\n\tcsv, err := receivers.NewCSV(\"out.csv\")\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tc.Add(csv)\n\n\t// Add your custom receiver which implements extractor.Receiver\n\tc.Add(\u0026myReceivers.Foo{})\n\n\t// Start pulling data from the extractor channels, and forwarding it to all receivers\n\t// Errors will be printed to stoud unless a handler is set\n\tc.Collect()\n}\n```\n\n## Authors\n\n* [John Hofrichter](github.com/johnhof)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnhof%2Fgdax-candle-extractor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjohnhof%2Fgdax-candle-extractor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnhof%2Fgdax-candle-extractor/lists"}