{"id":23405208,"url":"https://github.com/bonitoo-io/influxdb-csv-import","last_synced_at":"2025-06-14T04:37:00.376Z","repository":{"id":57519438,"uuid":"247665466","full_name":"bonitoo-io/influxdb-csv-import","owner":"bonitoo-io","description":"CSV import to INFLUXDB","archived":false,"fork":false,"pushed_at":"2020-04-07T07:43:45.000Z","size":188,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-14T18:00:04.205Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bonitoo-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-03-16T09:50:12.000Z","updated_at":"2020-04-07T07:43:47.000Z","dependencies_parsed_at":"2022-08-30T04:21:58.423Z","dependency_job_id":null,"html_url":"https://github.com/bonitoo-io/influxdb-csv-import","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bonitoo-io%2Finfluxdb-csv-import","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bonitoo-io%2Finfluxdb-csv-import/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bonitoo-io%2Finfluxdb-csv-import/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bonitoo-io%2Finfluxdb-csv-import/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bonitoo-io","download_url":"https://codeload.github.com/bonitoo-io/influxdb-csv-import/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247941713,"owners_count":21022037,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-22T13:18:56.696Z","updated_at":"2025-04-08T22:48:57.466Z","avatar_url":"https://github.com/bonitoo-io.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# influxdb-csv-import\nCSV data written to influx\n\n\n## DONE: STEP 1 - merged to influxdata/influxdb\nhttps://github.com/influxdata/influxdb/issues/17003 introduces a new CSV format for existing _influx write_ command.  CSV data on input are transformed to line protocol with the help of CSV annotations.\n### CSV Annotations\n* https://v2.docs.influxdata.com/v2.0/reference/syntax/annotated-csv/#annotations\n   * all of them are supported\n* additionally\n   * *#datatype* annotation is enhanced with:\n      * to mark non-field data: \n          * _measurement_, _tag_, _time_\n          * _ignore_ simply ignores the column\n          * _time_ is also supported, it is an alias for the existing _dateTime_\n          * _dateTime_ is either as a long (int64) number or RFC3339 format, this type is always used to represent time of measurement; or you can specify\n             * _dateTime:RFC3339_ for RFC3339 time\n             * _dateTime:number_ to expect a long number\n      * a _field_ data type can be used to let detect field's data type\n      * default datatype for a column is =field= unless _field column is present (ignored then)\n      * there can be at most 1 _dateTime_ column\n   * the following columns will have an extra semantics: _measurement, _time, _field, _value\n      * _measurement:  name of measurement\n      * _time: time of measurement\n      * _field: column that contains field name\n      * _value: column that contains field value\n* new `influx write` flags\n```sh\n  -f, --file string        The path to the file to import\n      --format string      Input format, either lp (Line Protocol) or csv (Comma Separated Values). Defaults to lp unless '.csv' extension\n```  \n### DRY RUN\nRun \"write dryrun\" command to write lines to stdout instead of InfluxDB. This \"dryrun\" command helps with validation and troubleshooting of CSV data.\n\n### Example 1 - Flux Query Result\n*influx write dryrun --file doc/examples/fluxQueryResult.csv*\n\n```bash\n#group,false,false,true,true,false,false,true,true,true,true\n#datatype,string,long,dateTime:RFC3339,dateTime:RFC3339,dateTime:RFC3339,double,string,string,string,string\n#default,_result,,,,,,,,,\n,result,table,_start,_stop,_time,_value,_field,_measurement,cpu,host\n,,0,2020-02-25T22:17:54.068926364Z,2020-02-25T22:22:54.068926364Z,2020-02-25T22:17:57Z,0,time_steal,cpu,cpu1,rsavage.prod\n,,0,2020-02-25T22:17:54.068926364Z,2020-02-25T22:22:54.068926364Z,2020-02-25T22:18:07Z,0,time_steal,cpu,cpu1,rsavage.prod\n\n#group,false,false,true,true,false,false,true,true,true,true\n#datatype,string,long,dateTime:RFC3339,dateTime:RFC3339,dateTime:RFC3339,double,string,string,string,string\n#default,_result,,,,,,,,,\n,result,table,_start,_stop,_time,_value,_field,_measurement,cpu,host\n,,1,2020-02-25T22:17:54.068926364Z,2020-02-25T22:22:54.068926364Z,2020-02-25T22:18:01Z,2.7263631815907954,usage_user,cpu,cpu-total,tahoecity.prod\n,,1,2020-02-25T22:17:54.068926364Z,2020-02-25T22:22:54.068926364Z,2020-02-25T22:18:11Z,2.247752247752248,usage_user,cpu,cpu-total,tahoecity.prod\n```\nline protocol data:\n```\ncpu,cpu=cpu1,host=rsavage.prod time_steal=0 1582669077000000000\ncpu,cpu=cpu1,host=rsavage.prod time_steal=0 1582669087000000000\ncpu,cpu=cpu-total,host=tahoecity.prod usage_user=2.7263631815907954 1582669081000000000\ncpu,cpu=cpu-total,host=tahoecity.prod usage_user=2.247752247752248 1582669091000000000\n```\n### Example 2 - Simple Annotated CSV file\n*influx write dryrun --file doc/examples/annotatedSimple.csv*\n\n```bash\n#datatype measurement,tag,tag,double,double,ignored,dateTime:number\nm,cpu,host,time_steal,usage_user,nothing,time\ncpu,cpu1,rsavage.prod,0,2.7,a,1482669077000000000\ncpu,cpu1,rsavage.prod,0,2.2,b,1482669087000000000\n```\n\nline protocol data: \n```\ncpu,cpu=cpu1,host=rsavage.prod time_steal=0,usage_user=2.7 1482669077000000000\ncpu,cpu=cpu1,host=rsavage.prod time_steal=0,usage_user=2.2 1482669087000000000\n```\n\n### Example 3 - Annotated CSV file with Data Types\n*influx write dryrun --file doc/examples/annotatedDatatype.csv*\n\n```bash\n#datatype measurement,tag,string,double,boolean,long,unsignedLong,duration,dateTime\n#default test,annotatedDatatypes,,,,,,\nm,name,s,d,b,l,ul,dur,time\n,,str1,1.0,true,1,1,1ms,1\n,,str2,2.0,false,2,2,2us,2020-01-11T10:10:10Z\n```\n\nline protocol data: \n```\ntest,name=annotatedDatatypes s=\"str1\",d=1,b=true,l=1i,ul=1u,dur=1000000i 1\ntest,name=annotatedDatatypes s=\"str2\",d=2,b=false,l=2i,ul=2u,dur=2000i 1578737410000000000\n```\n\n## STEP 2\nThese examples are related to https://github.com/influxdata/influxdb/issues/17004\n\n### Extended: Example 2 - Simple CSV file\n*influx write dryrun --file doc/examples/annotatedSimple.csv*\n\n```bash\n#datatype measurement,tag,tag,double,double,ignored,dateTime:number\nm,cpu,host,time_steal,usage_user,nothing,time\ncpu,cpu1,rsavage.prod,0,2.7,a,1482669077000000000\ncpu,cpu1,rsavage.prod,0,2.2,b,1482669087000000000\n```\n\nline protocol data: \n```\ncpu,cpu=cpu1,host=rsavage.prod time_steal=0,usage_user=2.7 1482669077000000000\ncpu,cpu=cpu1,host=rsavage.prod time_steal=0,usage_user=2.2 1482669087000000000\n```\n\nData type can be supplied in the column name, the CSV can be shortened to:\n\n```\nm|measurement,cpu|tag,host|tag,time_steal|double,usage_user|double,nothing|ignored,time|dateTime:number\ncpu,cpu1,rsavage.prod,0,2.7,a,1482669077000000000\ncpu,cpu1,rsavage.prod,0,2.2,b,1482669087000000000\n```\n*influx write dryrun --file doc/examples/labelsWithDataTypes_labels.csv*\n\n### Extended: Example 3 - Data Types with default values\n*influx write dryrun --file doc/examples/annotatedDatatype.csv*\n\n```bash\n#datatype measurement,tag,string,double,boolean,long,unsignedLong,duration,dateTime\n#default test,annotatedDatatypes,,,,,,\nm,name,s,d,b,l,ul,dur,time\n,,str1,1.0,true,1,1,1ms,1\n,,str2,2.0,false,2,2,2us,2020-01-11T10:10:10Z\n```\n\nline protocol data: \n```\ntest,name=annotatedDatatypes s=\"str1\",d=1,b=true,l=1i,ul=1u,dur=1000000i 1\ntest,name=annotatedDatatypes s=\"str2\",d=2,b=false,l=2i,ul=2u,dur=2000i 1578737410000000000\n```\n\nDefault value can be supplied in the column label after data type, the CSV could be also:\n\n```\nm|measurement|test,name|tag|annotatedDatatypes,s|string,d|double,b|boolean,l|long,ul|unsignedLong,dur|duration,time|dateTime\n,,str1,1.0,true,1,1,1ms,1\n,,str2,2.0,false,2,2,2us,2020-01-11T10:10:10Z\n```\n*influx write dryrun --file doc/examples/annotatedDatatype_labels.csv*\n\n### Example 4 - Advanced usage\n*influx write dryrun --file doc/examples/datetypeFormats_labels.csv*\n\n```\n#constant measurement,test\n#constant tag,name,datetypeFormats\n#timezone -0500\nt|dateTime:2006-01-02|1970-01-02,\"d|double:,. \",\"b|boolean:y,Y:n,N|y\"\n1970-01-01,\"123.456,78\",\n,\"123 456,78\",Y\n```\n   - measurement and extra tags is defined using the `#constant` annotation\n   - timezone for dateTime is to `-0500` (EST)\n   - `t` column is of `dateTime` data type of format is `2006-01-02`, default value is _January 2nd 1970_\n   - `d` column is of `double` data type with `,` as a fraction delimiter and `. ` as ignored separators that  used to visually separate large numbers into groups\n   - `b` column os of `boolean` data type that considers `y` or `Y` truthy, `n` or `N` falsy and empty column values as truthy \n\n\nline protocol data:\n```\ntest,name=datetypeFormats d=123456.78,b=true 18000000000000\ntest,name=datetypeFormats d=123456.78,b=true 104400000000000\n```\n\nYou can prepend and/or remove first lines from input data using command line options. You can also \noverride column name and define extra annotations that drive data processing. For example:\n\n*influx write dryrun --skipHeader=4 --header \"#constant measurement,test2\" --header \"t|dateTime:2006-01-02,_|ignored,s|string|unknown\" --file doc/examples/datetypeFormats_labels.csv*\n    - removes the first 4 lines from input and prepends the following that define\n       - measurement of the data\n       - column header row with \n          - custom dateTime format without a default value\n          - string column with a default value\n\nline protocol data:\n```\ntest2 s=\"unknown\" 0\ntest2 s=\"Y\"\n```\n### Example 5 - Custom column separator\n*influx write dryrun --file doc/examples/columnSeparator.csv*\n```\nsep=;\nm|measurement;available|boolean:y,Y:|n;dt|dateTime:number\ntest;nil;1\ntest;N;2\ntest;\";\";3\ntest;;4\ntest;Y;5\n```\n   - the first line can define a column separator character for next lines, here: `;`\n   - other lines use this separator, `available|boolean:y,Y` does not need to be wrapped in double quotes\n\nline protocol data:\n```\ntest available=false 1\ntest available=false 2\ntest available=false 3\ntest available=false 4\ntest available=true 5\n```\n### Example 6 - CSV conversion troubleshooting\n`influx write dryrun` helps with troubleshooting together with the following options\n   - `--debug` \n      - prints out internal representation of command line arguments including the default values for the flags that drive CSV processing\n      - prints out metadata about CSV columns that used for processing\n   - `--skipRowOnError`\n      - if a row cannot be parsed, it is ignored and the parsing error is printed to to stderr\n      - any error stops the processing without this option\n\nHaving csv input:\n```\nm|measurement,usage_user|double\ncpu,2.7\ncpu,nil\ncpu,\n,2.9\n```\n\n*influx write dryrun --skipRowOnError --file doc/examples/troubleshooting.csv 2\u003e/dev/null*\n```\ncpu usage_user=2.7\n```\n\n*influx write dryrun --skipRowOnError --file doc/examples/troubleshooting.csv 1\u003e/dev/null*\n```\n2020/04/03 15:14:51 line 3: column 'usage_user': strconv.ParseFloat: parsing \"nil\": invalid syntax\n2020/04/03 15:14:51 line 4: no field data found\n2020/04/03 15:14:51 line 5: column 'm': no measurement supplied\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbonitoo-io%2Finfluxdb-csv-import","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbonitoo-io%2Finfluxdb-csv-import","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbonitoo-io%2Finfluxdb-csv-import/lists"}