{"id":22180671,"url":"https://github.com/jftuga/date_gap_finder","last_synced_at":"2025-03-24T18:46:12.605Z","repository":{"id":57641376,"uuid":"359847511","full_name":"jftuga/date_gap_finder","owner":"jftuga","description":"detect date gaps in a CSV file","archived":false,"fork":false,"pushed_at":"2021-05-14T13:55:14.000Z","size":383,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-29T23:29:56.134Z","etag":null,"topics":["command-line","csv-files","gap","golang"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jftuga.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-20T14:35:55.000Z","updated_at":"2021-11-09T08:25:06.000Z","dependencies_parsed_at":"2022-09-05T07:41:44.187Z","dependency_job_id":null,"html_url":"https://github.com/jftuga/date_gap_finder","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jftuga%2Fdate_gap_finder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jftuga%2Fdate_gap_finder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jftuga%2Fdate_gap_finder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jftuga%2Fdate_gap_finder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jftuga","download_url":"https://codeload.github.com/jftuga/date_gap_finder/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245332067,"owners_count":20598099,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["command-line","csv-files","gap","golang"],"created_at":"2024-12-02T09:19:04.766Z","updated_at":"2025-03-24T18:46:12.584Z","avatar_url":"https://github.com/jftuga.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# date_gap_finder\n\n## Description\n`date_gap_finder` searches for missing dates with in CSV files and optionally inserts new CSV entries for those missing dates.\n\nBinaries for Windows, Mac, and Linux are available on the [releases](https://github.com/jftuga/date_gap_finder/releases) page.\n\n## Motivation\nI have many automated tasks that will append data to individual CSV files once per day.  However, if for some reason that \ntask does not start, there is a possibility that I will not be notified of any errors.  Each of these tasks are monitored via a \n`Grafana Dashboard` using the [grafana-csv-datasource](https://github.com/marcusolsson/grafana-csv-datasource) data source\nplugin.  This allows me to quickly view the status of all automated tasks.  When a task has missed a day, `date_gap_finder` will\ninsert CSV data in such a way that I will be notified that a task was not run.\n\nThis image displays the before and after of using `date_gap_finder`.  The **Photo Import** job runs every weekday.  Notice that `Monday, April 26` is missing from the *Before* image on the left.  This could be easily missed since it occurs on a Monday *(and the job isn't run on the weekends)*.  The *After* image on the right shows what can be displayed when using `date_gap_finder` to insert missing CSV data.  It is now much easier to detect a date gap.\n\n![Grafana Before and After](dgf_before_after.png)\n\n## Usage\n```\ndate_gap_finder searches for missing dates with in CSV files and optionally inserts CSV entries for those missing dates.\n\nUsage:\n  date_gap_finder [command]\n\nAvailable Commands:\n  help        Help about any command\n  insert      insert missing CSV entries\n  search      search CSV files for missing dates\n  stats       Show CSV file statistics\n\nFlags:\n  -a, --amount int         a maximum, numeric duration (default -1)\n  -B, --begin string       only consider dates on or after this date\n  -c, --column int         CSV column number (starts at zero)\n  -D, --debug int          enable verbose debugging, set to 999 or 9999\n  -d, --delimiter string   CSV delimiter (default \",\")\n  -E, --end string         only consider dates on or before this date\n  -f, --format string      use this date/time format; See https://github.com/nleeper/goment#format\n  -h, --help               help for date_gap_finder\n  -n, --noheader           set if CSV file does not have header line\n  -p, --padding string     add time to range before considering a gap between two dates (default \"1s\")\n  -S, --skipDays string    skip comma-delimited set of fully spelled out days\n  -s, --skipWeekends       allow gaps on weekends when set\n  -t, --tab                use tab character as CSV delimiter\n  -u, --unit string        unit of time, such as: days, hours, minutes\n  -v, --version            version for date_gap_finder\n```\n\n___\n\n## Example 1\n\nThis example file is called `e.csv`, which *should* get updated once per day. On the `15th`, it got updated twice. There are no entries for the `17th` and `18th`.\n\n| Date | Errors | Warnings\n|------|--------|----------\n| 2021-04-15 06:55:01 | 0 | 23\n| 2021-04-15 08:30:26 | 1 | 22\n| 2021-04-16 06:55:01 | 0 | 23\n| 2021-04-19 06:55:01 | 2 | 21\n\n__\n\n## search for date gaps\n* Search for gaps occurring more than `1 day` apart\n\n```\n$ date_gap_finder search -a 1 -u days e.csv\n2021-04-17 06:55:01\n2021-04-18 06:55:01\n```\n\n## search for date gaps, but exclude weekends\n* Use -s to skip weekends\n```\nPS C:\\\u003e .\\date_gap_finder.exe search -a 1 -u days -s .\\e.csv\n(no results, because the 17th and 18th occur on a Saturday and Sunday)\n```\n\n## search for date gaps, but exclude on certain days\n* Use -S to create a comma-separated list of days to skip\n```\n$ date_gap_finder search -a 1 -u days -S Sunday,Saturday e.csv\n(no results, because the 17th and 18th occur on a Saturday and Sunday)\n\n$ date_gap_finder search -a 1 -u days -S Sunday e.csv\n2021-04-17 06:55:01\n```\n\n## insert rows for date gaps\n* Adds column data for missing dates\n* **CSV columns start 0**\n* * `-r 1,1` - in column `1`, insert a value of `-1`\n* * `-r 2,999` - in column `2`, insert a value of `999`\n* * Note that multiple `-r` switches can be used\n```\nPS C:\\\u003e .\\date_gap_finder.exe insert -a 1 -u days -r 1,-1 -r 2,999 .\\e.csv\nDate,Errors,Warnings\n2021-04-15 06:55:01,0,23\n2021-04-15 08:30:26,0,23\n2021-04-16 06:55:01,0,23\n2021-04-17 06:55:01,-1,999\n2021-04-18 06:55:01,-1,999\n2021-04-19 06:55:01,0,23\n```\n\n## insert rows for date gaps with the same value\n* Similar to above, but use one `-R` switch instead of two `-r` switches\n* * `-R 567` - excluding the `non-date` column, insert values of `567`\n* * Note that `-R` and `-r` can be simultaneously used, with `-r` having precedence over `-R`\n```\n$ date_gap_finder insert -a 1 -u days -R 567 e.csv\nDate,Errors,Warnings\n2021-04-15 06:55:01,0,23\n2021-04-15 08:30:26,0,23\n2021-04-16 06:55:01,0,23\n2021-04-17 06:55:01,567,567\n2021-04-18 06:55:01,567,567\n2021-04-19 06:55:01,0,23\n```\n\n## insert rows for date gaps and overwrite original file\n* Use `-O` to overwrite the original file\n* * a *date-versioned* backup file will be created containing the original file\n* Use `-R` to add `0` to all missing columns of all missing rows\n* Note that a backup file similar to this has been created: `e--20210506.162151.bak`\n* * The naming convention is `filename--YYYYMMDD.HHMMSS.bak`\n```\nPS C:\u003e .\\date_gap_finder.exe insert -a 1 -u days -R 0 -O .\\e.csv\nPS C:\u003e cat e.csv\nDate,Errors,Warnings\n2021-04-15 06:55:01,0,23\n2021-04-15 08:30:26,0,23\n2021-04-16 06:55:01,0,23\n2021-04-17 06:55:01,0,0\n2021-04-18 06:55:01,0,0\n2021-04-19 06:55:01,0,23\n```\n___\n\n## Example 2\n\nThis example file is called `f.csv`, which *should* get updated once per day. However, each entry is off by a few seconds.  You can use the `-p` switch to correct for this.  It will pad time before and after the correct time. It is missing `5` days: `12-14` and `17-18`.\n\n| Date | Total | \n|------|-------|\n| 2021-03-10 18:40:01 | 317\n| 2021-03-11 18:40:01 | 249\n| 2021-03-15 18:40:04 | 287\n| 2021-03-16 18:40:03 | 320\n| 2021-03-19 18:40:06 | 102\n\n__\n\n## search for date gaps and use time padding\n* If a `6 second` time padding is not used, then `8` rows will be returned.  This is most likely not an accurate result since the times are only off by a few seconds. By using `-p 6s` an accurate list of missing rows is returned.\n* Padding can end in `s` for seconds, `m` for minutes, or `h` for hours. `Days` are not supported.\n* Note that `24 hours` is used instead of `1 day`.\n* **It is better to use time padding (-p) rather than using a longer time amount (-a)**\n```\n$ date_gap_finder search -a 24 -u hours -p 6s f.csv\n2021-03-12 18:40:01\n2021-03-13 18:40:01\n2021-03-14 18:40:01\n2021-03-17 18:40:01\n2021-03-18 18:40:01\n```\n___\n\n## Example 3\n\nThis example file is called `g.csv`.  The `date` column is in `1` instead of the normal column `0`.  This file is also delimited by the `tab` character instead of the `comma` character. It has a date gap consisting of `04-15` through `04-18`.\n\n| Processed | Date |\n|-----------|------|\n| 5125 | 2021-04-12\n| 5197 | 2021-04-13\n| 5206 | 2021-04-14\n| 5222 | 2021-04-19\n\n## search for gaps when date is not the first column and use a different column delimiter\n* Use `-d \"\\t\"` to define the column delimiter\n* * A short cut for the `tab` character is to simply use `-t` instead of `-d \"\\t\"`\n* Use `-c` to denote column `1` instead of the default of column `0`\n* Note that `1440 minutes` is equal to `1 day`\n```\nPS C:\\\u003e .\\date_gap_finder.exe search -a 1440 -u minutes -d \"\\t\" -c 1 .\\g.csv\n2021-04-15\n2021-04-16\n2021-04-17\n2021-04-18\n```\n\n## insert data for date-gapped missing rows\n* Similar switches to the above example, except that it is using the `insert` verb instead of `search`\n* `-r 0,9999` - in column `0`, insert a value of `9990`\n```\n$ date_gap_finder insert -a 1440 -u minutes -t -c 1 -r 0,9999 g.csv\nProcessed\tDate\n5125\t2021-04-12\n5197\t2021-04-13\n5206\t2021-04-14\n9999\t2021-04-15\n9999\t2021-04-16\n9999\t2021-04-17\n9999\t2021-04-18\n5222\t2021-04-19\n```\n___\n\n## Example 4\n\nWhen using the `insert` verb along with the `-O` switch (which is used for overwriting the CSV file), a backup file will be created in the form of:\n* `filename--YYYYMMDD.HHMMSS.bak`\n\nThis backup file will be identical to the original CSV file. The number of backup files will **continuously grow** unless the `-m` switch is used.  For example, using `-m 3` will limit the number of `.bak` files to `3`.\n```\n$ date_gap_finder insert -a 1440 -u minutes -t -c 1 -r 0,9999 -m 3 g.csv\n```\n___\n\n## Example 5\n`weekly.csv` is a CSV file with new entries occurring about once per week. The `mode` is the duration that occurs most often in a CSV file.\n```\nPS C:\u003e .\\date_gap_finder.exe stats weekly.csv\n+------------------------------+-----------------------+\n| file                         | weekly.csv            |\n| records                      | 57                    |\n| columns                      | 6                     |\n| date/time layout             | 06-Mar-2020           |\n| average time between entries | 1 week 6 hours        |\n| total duration               | 1 year 5 weeks 6 days |\n| mode                         | 1 week (168h0m0s)     |\n| mode count                   | 54                    |\n+------------------------------+-----------------------+\n```\n\n\n___\n## License\n* [MIT License](LICENSE)\n\n## Acknowledgments\n* https://github.com/spf13/cobra\n* https://github.com/nleeper/goment\n* https://github.com/araddon/dateparse\n* https://github.com/pivotal/go-ape\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjftuga%2Fdate_gap_finder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjftuga%2Fdate_gap_finder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjftuga%2Fdate_gap_finder/lists"}