{"id":38135925,"url":"https://github.com/adsr303/awking","last_synced_at":"2026-01-16T22:41:31.197Z","repository":{"id":57413141,"uuid":"146129003","full_name":"adsr303/awking","owner":"adsr303","description":"Make it easier to use Python as an AWK replacement.","archived":false,"fork":false,"pushed_at":"2021-12-09T17:38:58.000Z","size":26,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-09-20T05:30:53.172Z","etag":null,"topics":["awk-replacement","filtering","iterators","text-processing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/adsr303.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-25T20:54:57.000Z","updated_at":"2021-12-09T17:39:02.000Z","dependencies_parsed_at":"2022-09-26T17:11:17.250Z","dependency_job_id":null,"html_url":"https://github.com/adsr303/awking","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/adsr303/awking","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adsr303%2Fawking","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adsr303%2Fawking/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adsr303%2Fawking/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adsr303%2Fawking/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/adsr303","download_url":"https://codeload.github.com/adsr303/awking/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adsr303%2Fawking/sbom","scorecard":{"id":168225,"data":{"date":"2025-08-11","repo":{"name":"github.com/adsr303/awking","commit":"d7e2466cf49d7910a4e8369b5dc20ae1017c440d"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-16T15:33:06.211Z","repository_id":57413141,"created_at":"2025-08-16T15:33:06.211Z","updated_at":"2025-08-16T15:33:06.211Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28485638,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T11:59:17.896Z","status":"ssl_error","status_checked_at":"2026-01-16T11:55:55.838Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["awk-replacement","filtering","iterators","text-processing"],"created_at":"2026-01-16T22:41:31.130Z","updated_at":"2026-01-16T22:41:31.187Z","avatar_url":"https://github.com/adsr303.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# awking\n\nMake it easier to use Python as an AWK replacement.\n\n## Basic usage\n\n### Extracting groups of lines\n\n```python\nfrom awking import RangeGrouper\n\nlines = '''\ntext 1\ntext 2\ngroup start 1\ntext 3\ngroup end 1\ntext 4\ngroup start 2\ntext 5\ngroup end 2\ntext 6\n'''.splitlines()\n\nfor group in RangeGrouper('start', 'end', lines):\n    print(list(group))\n```\n\nThis will output:\n\n```text\n['group start 1', 'text 3', 'group end 1']\n['group start 2', 'text 5', 'group end 2']\n```\n\n### Extracting fixed-width fields\n\n```python\nfrom awking import records\n\nps_aux = '''\nUSER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND\nroot         1  0.0  0.0  51120  2796 ?        Ss   Dec22   0:09 /usr/lib/systemd/systemd --system --deserialize 22\nroot         2  0.0  0.0      0     0 ?        S    Dec22   0:00 [kthreadd]\nroot         3  0.0  0.0      0     0 ?        S    Dec22   0:04 [ksoftirqd/0]\nroot         5  0.0  0.0      0     0 ?        S\u003c   Dec22   0:00 [kworker/0:0H]\nroot         7  0.0  0.0      0     0 ?        S    Dec22   0:15 [migration/0]\nroot         8  0.0  0.0      0     0 ?        S    Dec22   0:00 [rcu_bh]\nroot         9  0.0  0.0      0     0 ?        S    Dec22   2:47 [rcu_sched]\nsaml      3015  0.0  0.0 117756   596 pts/2    Ss   Dec22   0:00 bash\nsaml      3093  0.9  4.1 1539436 330796 ?      Sl   Dec22  70:16 /usr/lib64/thunderbird/thunderbird\nsaml      3873  0.0  0.1 1482432 8628 ?        Sl   Dec22   0:02 gvim -f\nroot      5675  0.0  0.0 124096   412 ?        Ss   Dec22   0:02 /usr/sbin/crond -n\nroot      5777  0.0  0.0  51132  1068 ?        Ss   Dec22   0:08 /usr/sbin/wpa_supplicant -u -f /var/log/wpa_supplica\nsaml      5987  0.7  1.5 1237740 119876 ?      Sl   Dec26  14:05 /opt/google/chrome/chrome --type=renderer --lang=en-\nroot      6115  0.0  0.0      0     0 ?        S    Dec27   0:06 [kworker/0:2]\n'''\n\nfor user, _, command in records(ps_aux.splitlines(), widths=[7, 58, ...]):\n    print(user, command)\n```\n\nThis will output:\n\n```text\nUSER    COMMAND\nroot    /usr/lib/systemd/systemd --system --deserialize 22\nroot    [kthreadd]\nroot    [ksoftirqd/0]\nroot    [kworker/0:0H]\nroot    [migration/0]\nroot    [rcu_bh]\nroot    [rcu_sched]\nsaml    bash\nsaml    /usr/lib64/thunderbird/thunderbird\nsaml    gvim -f\nroot    /usr/sbin/crond -n\nroot    /usr/sbin/wpa_supplicant -u -f /var/log/wpa_supplica\nsaml    /opt/google/chrome/chrome --type=renderer --lang=en-\nroot    [kworker/0:2]\n```\n\n## The problem\n\nDid you ever have to scan a log file for XMLs? How hard was it for you to\nextract a set of multi-line XMLs into separate files?\n\nYou can use `re.findall` or `re.finditer` but you need to read the entire log\nfile into a string first. You can also use an AWK script like this one:\n\n```awk\n#!/usr/bin/awk -f\n\n/^Payload: \u003c([-_a-zA-Z0-9]+:)?Request/ {\n    ofname = \"request_\" (++index) \".xml\"\n    sub(/^Payload: /, \"\")\n}\n\n/\u003c([-_a-zA-Z0-9]+:)?Request/, /\u003c\\/([-_a-zA-Z0-9]+:)?Request/ {\n    print \u003e ofname\n}\n\n/\u003c\\/([-_a-zA-Z0-9]+:)?Request/ {\n    if (ofname) {\n        close(ofname)\n        ofname = \"\"\n    }\n}\n```\n\nThis works, and quite well. (Despite this being a Python module I encourage you\nto learn AWK if you don't already know it.)\n\nBut what if you want to build this kind of stuff into your Python application?\nWhat if your input is not lines in a file but a different type of objects?\n\n### Python equivalent using `awking`\n\nThe `RangeGrouper` class groups elements from the input iterable based on\npredicates for the start and end element. This is a bit like Perl's range\noperator or AWK's range pattern, except that your ranges get grouped into\n`START..END` iterables.\n\nAn equivalent of the above AWK script might look like this:\n\n```python\nfrom awking import RangeGrouper\nimport re\nimport sys\n\ng = RangeGrouper(r'^Payload: \u003c([-_a-zA-Z0-9]+:)?Request',\n                 r'\u003c/([-_a-zA-Z0-9]+:)?Request', sys.stdin)\nfor index, request in enumerate(g, 1):\n    with open(f'request_{index}.xml', 'w') as f:\n        for line in request:\n            line = re.sub(r'^Payload: ', '', line)  # Not optimal\n            print(line, file=f, end='')\n```\n\nThe predicates may be regular expressions, either as `re.compile()` objects or\nstrings; or they may be any callables that accept a single argument and return\na true/false value.\n\n## Caveats\n\nThe grouping algorithm reads the input iterable lazily. You can still run out\nof memory if you keep references to previous groups without consuming them.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadsr303%2Fawking","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadsr303%2Fawking","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadsr303%2Fawking/lists"}