{"id":19832067,"url":"https://github.com/ail-project/ail-feeder-gharchive","last_synced_at":"2025-07-11T20:36:25.123Z","repository":{"id":45165541,"uuid":"424660123","full_name":"ail-project/ail-feeder-gharchive","owner":"ail-project","description":"AIL feeder for GitHub archive - gharchive.org","archived":false,"fork":false,"pushed_at":"2023-08-16T08:36:51.000Z","size":65,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-11-12T11:42:52.599Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ail-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-04T16:14:38.000Z","updated_at":"2024-11-09T11:35:00.000Z","dependencies_parsed_at":"2024-11-12T11:36:28.696Z","dependency_job_id":"799a63f4-e574-49da-bb66-a3a4e4356607","html_url":"https://github.com/ail-project/ail-feeder-gharchive","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2Fail-feeder-gharchive","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2Fail-feeder-gharchive/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2Fail-feeder-gharchive/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2Fail-feeder-gharchive/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ail-project","download_url":"https://codeload.github.com/ail-project/ail-feeder-gharchive/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233466586,"owners_count":18680564,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T11:36:19.033Z","updated_at":"2025-01-11T10:18:38.358Z","avatar_url":"https://github.com/ail-project.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AIL - feeder from GHArchive\n\nThis AIL feeder is a generic software to extract informations from [GHArchive](https://www.gharchive.org/), collect and feed AIL via AIL ReST API.\n\n\n\n# Usage\n\n~~~shell\ndacru@dacru:~/git/ail-feeder-gharchive/bin$ python3 gharchive_feeder.py --help  \nusage: gharchive_feeder.py [-h] [-d] [-v] -a ARCHIVENAME [--nocache] [-u USERS [USERS ...]] \n\t\t\t\t\t\t   [-fu FILEUSERS] [-o ORG [ORG ...]] [-fo FILEORG] \n\t\t\t\t\t\t   [-w WORDS [WORDS ...]] [-c] [-fw FILEWORD]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -d                    debug\n  -v                    verbose, more display\n  -a ARCHIVENAME, --archiveName ARCHIVENAME\n                        date of the GHArchive to Download, YYYY-MM-DD-H, YYYY-MM-DD-{H..H}, YYYY-MM-{DD..DD}-{H..H}, YYYY-MM-{DD..DD}-H\n  --nocache             disable store of archive\n  -u USERS [USERS ...], --users USERS [USERS ...]\n                        search username\n  -fu FILEUSERS, --fileusers FILEUSERS\n                        file containing list of username\n  -o ORG [ORG ...], --org ORG [ORG ...]\n                        search organisation\n  -fo FILEORG, --fileorg FILEORG\n                        file containing list of organisations\n  -w WORDS [WORDS ...], --words WORDS [WORDS ...]\n                        list of words to search. '-w update bot' will search both words with an AND. '-w \"update bot\"' will search for the string\n  -c, --case            active case for --words option\n  -fw FILEWORD, --fileword FILEWORD\n                        file containing list of words for commit message\n  --git_vuln_finder     Call git-vuln-finder module. Apply patterns on commit message to find vulnerability. This option pass over all other one.\n\n~~~\n\n\n\n\n\n# Example of use\n\n1. Download the archive for: 2 am, 1 October 2021\n\n~~~\ndacru@dacru:~/git/ail-feeder-gharchive/bin$ python3 gharchive_feeder.py -a 2021-10-01-2\n~~~\n\n\n\n2. Download the archive for: 15 pm, 2 October 2021. Search for \"password\"  and for \"removed\" in commit message\n\n~~~\ndacru@dacru:~/git/ail-feeder-gharchive/bin$ python3 gharchive_feeder.py -a 2021-10-01-2 -w password removed\n~~~\n\n\n\n3. Download the archive for: 15 pm, 2 October 2021. Search for \"password removed\" in commit message\n\n~~~\ndacru@dacru:~/git/ail-feeder-gharchive/bin$ python3 gharchive_feeder.py -a 2021-10-01-2 -w \"password removed\"\n~~~\n\n\n\n4. Download the archive for: 15 pm, 2 October 2021. Search for \"password\"  and for \"removed\" in commit message for the org CIRCL\n\n~~~\ndacru@dacru:~/git/ail-feeder-gharchive/bin$ python3 gharchive_feeder.py -a 2021-10-01-2 -w password removed -o CIRCL\n~~~\n\n\n\n5. Download the archive for: 15 pm, 2 October 2021. Search for \"password\"  and for \"removed\" in commit message for the user DavidCruciani\n\n~~~\ndacru@dacru:~/git/ail-feeder-gharchive/bin$ python3 gharchive_feeder.py -a 2021-10-01-2 -w password removed -u DavidCruciani\n~~~\n\n\n\n6. Download the archive for: 15 pm, 2 October 2021. Use special git-vuln-finder module\n\n~~~\ndacru@dacru:~/git/ail-feeder-gharchive/bin$ python3 gharchive_feeder.py -a 2021-10-01-2 --git-vuln-finder\n~~~\n\n\n\n\n\n# JSON output format to AIL\n\n- `source` is the name of the AIL feeder module\n- `source-uuid` is the UUID of the feeder (unique per feeder)\n- `data` is commit message or path informations\n- `meta` is the generic field where feeder can add the metadata collected\n\n\n\nUsing the AIL API, `data` will be compress in gzip format and encode with base64 procedure. Then a new field will created, `data-sha256` who will be the result of sha256 on data after treatment.\n\n\n\n\n\n# (main) Requirements\n\n- [PyAIL](https://github.com/ail-project/PyAIL)\n- [redis](https://github.com/redis/redis-py)\n- [git-vuln-finder](https://github.com/cve-search/git-vuln-finder)\n\n\n\n## ail_feeder_gharchive\n\n#### Commit part\n\n`data` will contain commit message of a PushEvent\n\n~~~json\n{\n    \"data\": \"Bump to 0.0.4\",\n    \"default-encoding\": \"UTF-8\",\n    \"meta\": {\n        \"id\": \"3304d136-ccef-4cee-9ec3-169022547eff\",\n        \"github:id_event\": \"18249112571\",\n        \"github:repo_id\": \"408646046\",\n        \"github:repo_name\": \"edumoreira1506/cig-factories\",\n        \"github:repo_node_id\": \"R_kgDOGFtxng\",\n        \"github:repo_owner\": \"edumoreira1506\",\n        \"github:repo_owner_id\": \"49662698\",\n        \"github:repo_owner_node_id\": \"MDQ6VXNlcjQ5NjYyNjk4\",\n        \"github:push_id\": \"8062525290\",\n        \"github:commit_id\": \"bd6ea0f6acf85ce548d0e9a11629aa5d8a99de59\",\n        \"github:commit_node_id\": \"C_kwDOGFtxntoAKGJkNmVhMGY2YWNmODVjZTU0OGQwZTlhMTE2MjlhYTVkOGE5OWRlNTk\",\n        \"github:commit_url\": \"https://api.github.com/repos/edumoreira1506/cig-factories/commits/bd6ea0f6acf85ce548d0e9a11629aa5d8a99de59\",\n        \"github:pusher_email\": \"00ceee5b1c012899ffa1231a9566ffe1440c25ee@eduardoem.com.br\",\n        \"github:pusher\": \"Eduardo Moreira\",\n        \"github:pusher_id\": \"49662698\",\n        \"github:pusher_node_id\": \"MDQ6VXNlcjQ5NjYyNjk4\",\n        \"github:datestamp\": \"2021-10-02\",\n        \"github:timestamp\": \"00:00:01\",\n        \"github:timezone\": \"UTC\"\n    },\n    \"source\": \"gharchive:commit\",\n    \"source-uuid\": \"80172ead-7023-496c-a4be-6ee280d8fbcf\"\n}\n~~~\n\n\n\n#### Patch part\n\n`data` will contain patch informations of a commit\n\n~~~json\n{\n    \"data\": \"@@ -1,6 +1,6 @@\\n {\\n \\t\\\"name\\\": \\\"@cig-platform/factories\\\",\\n-\\t\\\"version\\\": \\\"0.0.3\\\",\\n+\\t\\\"version\\\": \\\"0.0.4\\\",\\n \\t\\\"description\\\": \\\"\\\",\\n \\t\\\"main\\\": \\\"build/index.js\\\",\\n \\t\\\"types\\\": \\\"build/index.d.ts\\\",\",\n    \"default-encoding\": \"UTF-8\",\n    \"meta\": {\n        \"github:id_event\": \"18249112571\",\n        \"github:repo_id\": \"408646046\",\n        \"github:repo_name\": \"edumoreira1506/cig-factories\",\n        \"github:repo_node_id\": \"R_kgDOGFtxng\",\n        \"github:repo_owner\": \"edumoreira1506\",\n        \"github:repo_owner_id\": \"49662698\",\n        \"github:repo_owner_node_id\": \"MDQ6VXNlcjQ5NjYyNjk4\",\n        \"github:push_id\": \"8062525290\",\n        \"github:commit_id\": \"bd6ea0f6acf85ce548d0e9a11629aa5d8a99de59\",\n        \"github:commit_node_id\": \"C_kwDOGFtxntoAKGJkNmVhMGY2YWNmODVjZTU0OGQwZTlhMTE2MjlhYTVkOGE5OWRlNTk\",\n        \"github:commit_url\": \"https://api.github.com/repos/edumoreira1506/cig-factories/commits/bd6ea0f6acf85ce548d0e9a11629aa5d8a99de59\",\n        \"github:pusher_email\": \"00ceee5b1c012899ffa1231a9566ffe1440c25ee@eduardoem.com.br\",\n        \"github:pusher\": \"Eduardo Moreira\",\n        \"github:pusher_id\": \"49662698\",\n        \"github:pusher_node_id\": \"MDQ6VXNlcjQ5NjYyNjk4\",\n        \"github:datestamp\": \"2021-10-02\",\n        \"github:timestamp\": \"00:00:01\",\n        \"github:timezone\": \"UTC\",\n        \"github:parent\": \"3304d136-ccef-4cee-9ec3-169022547eff\"\n    },\n    \"source\": \"gharchive:patch\",\n    \"source-uuid\": \"80172ead-7023-496c-a4be-6ee280d8fbcf\"\n}\n~~~\n\n\n## License\n\n\nThis software is licensed under [GNU Affero General Public License version 3](http://www.gnu.org/licenses/agpl-3.0.html)\n\nCopyright (C) 2021-2023 CIRCL - Computer Incident Response Center Luxembourg\n\nCopyright (C) 2021-2023 David Cruciani\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fail-project%2Fail-feeder-gharchive","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fail-project%2Fail-feeder-gharchive","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fail-project%2Fail-feeder-gharchive/lists"}