{"id":28690623,"url":"https://github.com/thunderbird/rt-tb-regular-expression-scanner","last_synced_at":"2025-06-14T06:05:46.907Z","repository":{"id":260852281,"uuid":"882524051","full_name":"thunderbird/rt-tb-regular-expression-scanner","owner":"thunderbird","description":"Experimental Thunderbird Support regular expression scanner","archived":false,"fork":false,"pushed_at":"2024-11-18T00:26:08.000Z","size":23438,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-11-18T01:25:51.707Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thunderbird.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-03T02:13:13.000Z","updated_at":"2024-11-18T00:26:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"2b7de539-823c-477d-b3df-b2256c0028cd","html_url":"https://github.com/thunderbird/rt-tb-regular-expression-scanner","commit_stats":null,"previous_names":["thunderbird/rt-tb-regular-expression-scanner"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/thunderbird/rt-tb-regular-expression-scanner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thunderbird%2Frt-tb-regular-expression-scanner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thunderbird%2Frt-tb-regular-expression-scanner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thunderbird%2Frt-tb-regular-expression-scanner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thunderbird%2Frt-tb-regular-expression-scanner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thunderbird","download_url":"https://codeload.github.com/thunderbird/rt-tb-regular-expression-scanner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thunderbird%2Frt-tb-regular-expression-scanner/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259768612,"owners_count":22908230,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-14T06:05:43.592Z","updated_at":"2025-06-14T06:05:46.897Z","avatar_url":"https://github.com/thunderbird.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# rt-tb-regular-expression-scanner\nExperimental Thunderbird Support regular expression scanner\n## 1. 2024-11-09 Add Windows version e.g. q['windows'] = 7 or q['windows'] = 10\n```bash\n./scan-tb-question-answer-file.rb \\\nlink_epoch_time_yyyy_mm_dd_iso_week_2023-2024-yearly-thunderbird-questions.csv \\\n2023-2024-yearly-thunderbird-answers.csv\n```\n## 7. 2024-11-03 Plot 2024 Week 24 to 2024 Week 43\n* 2024-week-24-2024-week43-plot-2023-2024-tb-oauth-microsoft-email-gmail-antivirus-mentions.R and PLOTS/2024-week-24-2024-week43-plot-2023-2024-tb-oauth-microsoft-email-gmail-antivirus-mentions.png\n## 6. 2024-11-03 Plot 2024 Week 1 to 2024 Week 43\n* 2024-week-1-2024-week43-plot-2023-2024-tb-oauth-microsoft-email-gmail-antivirus-mentions.R and PLOTS/2024-week-1-2024-week43-plot-2023-2024-tb-oauth-microsoft-email-gmail-antivirus-mentions.png\n## 5. 2024-11-03 Plot 2023 Week 14 to 2024 Week 43\n* 2023-week-14-2024-week43-plot-2023-2024-tb-oauth-microsoft-email-gmail-antivirus-mentions.R and PLOTS/2023-week-14-2024-week43-plot-2023-2024-tb-oauth-microsoft-email-gmail-antivirus-mentions.png\n## 4. 2024-11-03 concat all the files with regex mentions: anti-virus, microsoft email, gmail, oauth\n```bash\nmlr --csv join -f 2023-2024-thunderbird-oauth-mentions.csv -j iso_week \\\nthen join -f 2023-2024-thunderbird-microsoft-email-mentions.csv -j iso_week \\\nthen join -f 2023-2024-thunderbird-gmail-email-mentions.csv -j iso_week \\\n2023-2024-thunderbird-antivirus-mentions.csv \\\n\u003e 2023-2024-tb-oauth-microsoft-email-gmail-antivirus-mentions.csv\n```\n## 3. 2024-11-03 add anti-virus\n```bash\nmlr --csv --from regex-matches-link_epoch_time_yyyy_mm_dd_iso_week_2023-2024-yearly-thunderbird-questions.csv \\\nfilter '${av:unknown} == \"0\"' \\\nthen cut -f link,iso_week,yyyy_mm_dd \\\nthen sort -f iso_week \\\nthen count -g iso_week -o \"antivirus mentions\" \u003e 2023-2024-thunderbird-antivirus-mentions.csv\n```\n## 2. 2024-11-03 add microsoft email and gmail \n\n**microsoft email:**\n\n```bash\nmlr --csv --from regex-matches-link_epoch_time_yyyy_mm_dd_iso_week_2023-2024-yearly-thunderbird-questions.csv \\\nfilter '${m:microsoftemail} == \"1\"' \\\nthen cut -f link,iso_week,yyyy_mm_dd \\\nthen sort -f iso_week \\\nthen count -g iso_week -o \"ms email mentions\" \u003e 2023-2024-thunderbird-microsoft-email-mentions.csv\n```\n\n**gmail email:**\n```bash\nmlr --csv --from regex-matches-link_epoch_time_yyyy_mm_dd_iso_week_2023-2024-yearly-thunderbird-questions.csv \\\nfilter '${m:gmail} == \"1\"' \\\nthen cut -f link,iso_week,yyyy_mm_dd \\\nthen sort -f iso_week \\\nthen count -g iso_week -o \"gmail mentions\" \u003e 2023-2024-thunderbird-gmail-email-mentions.csv\n```\n\n## 1. 2024-11-03 Rename the count field to \"OAuth mentions\"\n\n```bash\nmlr --csv --from regex-matches-link_epoch_time_yyyy_mm_dd_iso_week_2023-2024-yearly-thunderbird-questions.csv \\\nfilter '${oa:oauth} == \"1\"' \\\nthen cut -f link,iso_week,yyyy_mm_dd \\\nthen sort -f iso_week \\\nthen count -g iso_week -o \"OAuth mentions\" \\\n\u003e 2023-2024-thunderbird-oauth-mentions.csv\n```\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary markdown=\"span\"\u003eoutput:\u003c/summary\u003e\n\n```   \niso_week,OAuth mentions\n2023-13,1\n2023-14,1\n2023-15,3\n2023-16,8\n2023-17,3\n2023-18,2\n2023-19,5\n2023-20,5\n2023-21,9\n2023-22,7\n2023-23,12\n2023-24,9\n2023-25,2\n2023-26,8\n2023-27,6\n2023-28,8\n2023-29,19\n2023-30,7\n2023-31,2\n2023-32,10\n2023-33,3\n2023-34,4\n2023-35,4\n2023-36,5\n2023-37,12\n2023-38,2\n2023-39,7\n2023-40,17\n2023-41,7\n2023-42,7\n2023-43,7\n2023-44,5\n2023-45,4\n2023-46,6\n2023-47,6\n2023-48,8\n2023-49,4\n2023-50,6\n2023-51,1\n2023-52,4\n2024-01,4\n2024-02,1\n2024-03,6\n2024-04,9\n2024-05,6\n2024-06,7\n2024-07,6\n2024-08,4\n2024-09,6\n2024-10,3\n2024-11,6\n2024-12,2\n2024-13,1\n2024-14,3\n2024-15,5\n2024-16,12\n2024-17,6\n2024-18,4\n2024-19,3\n2024-20,7\n2024-21,3\n2024-22,9\n2024-23,7\n2024-24,8\n2024-25,12\n2024-26,13\n2024-27,18\n2024-28,11\n2024-29,21\n2024-30,19\n2024-31,11\n2024-32,18\n2024-33,9\n2024-34,16\n2024-35,4\n2024-36,5\n2024-37,25\n2024-38,21\n2024-39,24\n2024-40,27\n2024-41,11\n2024-42,14\n2024-43,8\n2024-44,6\n```\n\u003c/details\u003e\n\n## 1. 2024-11-02 Get the daily question and answer files\n1. git pull of latest questions and answers: `pushd /Users/roland/Documents/GIT/github-action-thunderbird-aaq ; git pull; popd`\n2. create 2023 and 2024 answer CSV:\n   ```bash\n   mlr --csv sort -f id /Users/roland/Documents/GIT/github-action-thunderbird-aaq/202*/*thunderbird-answers-for-questions-desktop.csv \\\n   \u003e 2023-2024-yearly-thunderbird-answers.csv\n   ```\n1. create 2023 and 2024 question CSV:\n   ```bash\n   mlr --csv sort -f id /Users/roland/Documents/GIT/github-action-thunderbird-aaq/202*/*thunderbird-creator-answers-desktop-all-locales.csv \\\n   \u003e 2023-2024-yearly-thunderbird-questions.csv\n   ```\n1. add yyyy_mm_dd, iso_week and link field to question CSV\n   ```bash\n   mlr --csv put '$created_epoch = strptime($created, \"%Y-%m-%d %H:%M:%S %z\")' \\\n   then put -f make-question-link.mlr \\\n   then put '$iso_week = strftime($created_epoch, \"%V\")' \\\n   then  put '$yyyy_mm_dd = strftime($created_epoch, \"%Y-%m-%d\")' \\\n   then sort -f id \\\n   2023-2024-yearly-thunderbird-questions.csv \\\n   \u003e link_epoch_time_yyyy_mm_dd_iso_week_2023-2024-yearly-thunderbird-questions.csv\n   ```\n## 2. 2024-11-02 Add: oauth, hotmail/office365/o365, gmail/googlemail etc synthetic boolean columns\n```bash\n./scan-tb-question-answer-file.rb \\\nlink_epoch_time_yyyy_mm_dd_iso_week_2023-2024-yearly-thunderbird-questions.csv \\\n2023-2024-yearly-thunderbird-answers.csv\n```\n## 3. 2024-11-02 BUG! The iso_weeks don't have the year! So iso_week is wrong! and the output below is wrong\n```bash\nmlr --csv --from regex-matches-link_epoch_time_yyyy_mm_dd_iso_week_2023-2024-yearly-thunderbird-questions.csv \\\nfilter '${oa:oauth} == \"1\"' \\\nthen cut -f link,iso_week,yyyy_mm_dd \\\nthen sort -f iso_week \\\nthen count-distinct -f iso_week\n```\n\noutput:\n\n```\niso_week,OAuth mentions\n2023-13,1\n2023-14,1\n2023-15,3\n2023-16,8\n2023-17,3\n2023-18,2\n2023-19,5\n2023-20,5\n2023-21,9\n2023-22,7\n2023-23,12\n2023-24,9\n2023-25,2\n2023-26,8\n2023-27,6\n2023-28,8\n2023-29,19\n2023-30,7\n2023-31,2\n2023-32,10\n2023-33,3\n2023-34,4\n2023-35,4\n2023-36,5\n2023-37,12\n2023-38,2\n2023-39,7\n2023-40,17\n2023-41,7\n2023-42,7\n2023-43,7\n2023-44,5\n2023-45,4\n2023-46,6\n2023-47,6\n2023-48,8\n2023-49,4\n2023-50,6\n2023-51,1\n2023-52,4\n2024-01,4\n2024-02,1\n2024-03,6\n2024-04,9\n2024-05,6\n2024-06,7\n2024-07,6\n2024-08,4\n2024-09,6\n2024-10,3\n2024-11,6\n2024-12,2\n2024-13,1\n2024-14,3\n2024-15,5\n2024-16,12\n2024-17,6\n2024-18,4\n2024-19,3\n2024-20,7\n2024-21,3\n2024-22,9\n2024-23,7\n2024-24,8\n2024-25,12\n2024-26,13\n2024-27,18\n2024-28,11\n2024-29,21\n2024-30,19\n2024-31,11\n2024-32,18\n2024-33,9\n2024-34,16\n2024-35,4\n2024-36,5\n2024-37,25\n2024-38,21\n2024-39,24\n2024-40,27\n2024-41,11\n2024-42,14\n2024-43,8\n2024-44,6\n```\n## 4. 2024-11-02 Fix bug in iso_week\n```bash\nmlr --csv put '$created_epoch = strptime($created, \"%Y-%m-%d %H:%M:%S %z\")' \\\nthen put -f make-question-link.mlr \\\nthen put '$iso_week = strftime($created_epoch, \"%Y-%V\")' \\\nthen  put '$yyyy_mm_dd = strftime($created_epoch, \"%Y-%m-%d\")' \\\nthen sort -f id \\\n2023-2024-yearly-thunderbird-questions.csv \\\n\u003e link_epoch_time_yyyy_mm_dd_iso_week_2023-2024-yearly-thunderbird-questions.csv\n```\n\n## 5.  2024-11-02 With iso_week in the form yyyy-iso_week Add: oauth, hotmail/office365/o365, gmail/googlemail etc synthetic boolean columns\n```bash\n./scan-tb-question-answer-file.rb \\\nlink_epoch_time_yyyy_mm_dd_iso_week_2023-2024-yearly-thunderbird-questions.csv \\\n2023-2024-yearly-thunderbird-answers.csv\n```\n## 6. 2024-11-02 OAuth complaints have  gone up since week 28\n```bash\nmlr --csv --from regex-matches-link_epoch_time_yyyy_mm_dd_iso_week_2023-2024-yearly-thunderbird-questions.csv \\\nfilter '${oa:oauth} == \"1\"' \\\nthen cut -f link,iso_week,yyyy_mm_dd \\\nthen sort -f iso_week \\\nthen count-distinct -f iso_week\n```\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary markdown=\"span\"\u003eoutput:\u003c/summary\u003e\n   \n```\niso_week,count\n2023-13,1\n2023-14,1\n2023-15,3\n2023-16,8\n2023-17,3\n2023-18,2\n2023-19,5\n2023-20,5\n2023-21,9\n2023-22,7\n2023-23,12\n2023-24,9\n2023-25,2\n2023-26,8\n2023-27,6\n2023-28,8\n2023-29,19\n2023-30,7\n2023-31,2\n2023-32,10\n2023-33,3\n2023-34,4\n2023-35,4\n2023-36,5\n2023-37,12\n2023-38,2\n2023-39,7\n2023-40,17\n2023-41,7\n2023-42,7\n2023-43,7\n2023-44,5\n2023-45,4\n2023-46,6\n2023-47,6\n2023-48,8\n2023-49,4\n2023-50,6\n2023-51,1\n2023-52,4\n2024-01,4\n2024-02,1\n2024-03,6\n2024-04,9\n2024-05,6\n2024-06,7\n2024-07,6\n2024-08,4\n2024-09,6\n2024-10,3\n2024-11,6\n2024-12,2\n2024-13,1\n2024-14,3\n2024-15,5\n2024-16,12\n2024-17,6\n2024-18,4\n2024-19,3\n2024-20,7\n2024-21,3\n2024-22,9\n2024-23,7\n2024-24,8\n2024-25,12\n2024-26,13\n2024-27,18\n2024-28,11\n2024-29,21\n2024-30,19\n2024-31,11\n2024-32,18\n2024-33,9\n2024-34,16\n2024-35,4\n2024-36,5\n2024-37,25\n2024-38,21\n2024-39,24\n2024-40,27\n2024-41,11\n2024-42,14\n2024-43,8\n2024-44,6\n```\n\u003c/details\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunderbird%2Frt-tb-regular-expression-scanner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthunderbird%2Frt-tb-regular-expression-scanner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunderbird%2Frt-tb-regular-expression-scanner/lists"}