{"id":19832056,"url":"https://github.com/ail-project/ail-feeder-ct","last_synced_at":"2026-02-02T20:07:34.755Z","repository":{"id":114094123,"uuid":"462242310","full_name":"ail-project/ail-feeder-ct","owner":"ail-project","description":"AIL feeder for certificate transparency","archived":false,"fork":false,"pushed_at":"2023-08-16T08:31:27.000Z","size":48,"stargazers_count":2,"open_issues_count":1,"forks_count":1,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-01-22T12:25:17.877Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ail-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-22T10:16:48.000Z","updated_at":"2024-08-12T20:20:55.000Z","dependencies_parsed_at":"2024-11-12T11:46:38.465Z","dependency_job_id":null,"html_url":"https://github.com/ail-project/ail-feeder-ct","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2Fail-feeder-ct","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2Fail-feeder-ct/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2Fail-feeder-ct/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2Fail-feeder-ct/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ail-project","download_url":"https://codeload.github.com/ail-project/ail-feeder-ct/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243806031,"owners_count":20350773,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T11:36:13.531Z","updated_at":"2026-02-02T20:07:29.715Z","avatar_url":"https://github.com/ail-project.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AIL - feeder from certificate transparency \r\nThis AIL feeder is a generic software to extract informations from certificate transparency \r\n\r\n\r\n\r\nFor the generation of domain name variations: [ail-typo-squatting](https://github.com/ail-project/ail-typo-squatting)\r\n\r\n\r\n\r\n# Requirements\r\n\r\n- [redis](https://github.com/redis/redis-py)\r\n- [cerstream](https://github.com/CaliDog/certstream-python)\r\n- [dnspython](https://github.com/rthalley/dnspython)\r\n- [ail-typo-squatting](https://github.com/ail-project/ail-typo-squatting)\r\n- [pyail](https://github.com/ail-project/PyAIL)\r\n- [beautifulsoup4](https://pypi.org/project/beautifulsoup4/)\r\n- [M2Crypto](https://github.com/mcepl/M2Crypto)\r\n\r\n\r\n\r\nFor M2Crypto:\r\n\r\n~~~bash\r\nsudo apt-get install libssl-dev swig python3-dev gcc\r\n~~~\r\n\r\n\r\n\r\n\r\n\r\n# How to run\r\n\r\nThe program need to run two script to be functional, `bin/ct.py` and `bin/feeder_ct.py`. \r\n\r\nThe first one will publish ct informations on a redis db and the other one will subscribe to the channel and use any entry to compare with a list of variations of domain name. Redis pub/sub is used here. \r\n\r\nIf a variation match with an entry from redis db, then the variation is send to AIL to crawl the website.\r\n\r\n# Usage\r\n\r\n~~~~shell\r\ndacru@dacru:~/git/ail-feeder-ct/bin$ python3 feeder_ct.py --help  \r\nusage: feeder_ct.py [-h] [-dn DOMAINNAME [DOMAINNAME ...]] [-fdn FILEDOMAINNAME] [-a] [-ats] [-ms] [-vt] [-w] [-o OUTPUT] [-v]\r\n\r\noptions:\r\n  -h, --help            show this help message and exit\r\n  -dn DOMAINNAME [DOMAINNAME ...], --domainName DOMAINNAME [DOMAINNAME ...]\r\n                        list of domain name\r\n  -fdn FILEDOMAINNAME, --filedomainName FILEDOMAINNAME\r\n                        file containing list of domain name\r\n  -a, --ail             Send domain to AIL crawler\r\n  -ats, --ail_typo_squatting\r\n                        Generate Variations for list pass in entry\r\n  -ms, --matching_string\r\n                        Match domain name if variations are in the domain name in any position\r\n  -vt, --virustotal     Check domain on virus total\r\n  -w, --warning\t\t\tIf CNAME is not the same as the matching domain then send a warning to Ail or just display a message.\r\n  -o OUTPUT, --output OUTPUT\r\n                        path to ouput location, default: ../output\r\n  -v                    verbose, more display\r\n\r\n~~~~\r\n\r\n\r\n\r\n# Example of use\r\n\r\n\u003cb\u003eFirst of all\u003c/b\u003e `ct.py` need to be run.\r\n\r\n\r\n\r\nNeed to pass a text file, where each line is a variation of the original domain name. Variations can be generate at this repository: [ail-typo-squatting](https://github.com/ail-project/ail-typo-squatting)\r\n\r\n~~~~shell\r\ndacru@dacru:~/git/ail-feeder-ct/bin$ python3 feeder_ct.py -fd circl.lu.txt\r\n~~~~\r\n\r\n\r\n\r\nIt's possible to generate variations directly in the program\r\n\r\n~~~~shell\r\ndacru@dacru:~/git/ail-feeder-ct/bin$ python3 feeder_ct.py -dn circl.lu -ats\r\n~~~~\r\n\r\n\r\n\r\nIt's possible to search variations in domain name instead of searching only if it's equal\r\n\r\n~~~~\r\ndacru@dacru:~/git/ail-feeder-ct/bin$ python3 feeder_ct.py -dn circl.lu -ats -ms\r\n~~~~\r\n\r\n\r\n\r\n# JSON output format\r\n\r\nthe name of the JSON file will be the domains matching the variation.\r\n\r\nif the dns resolving give no result, then the key \"dns_resolve\" will not be present in the JSON file.\r\n\r\n~~~~json\r\n{\r\n\t\"certificat\": \"\", \r\n    \"domains\": [], \r\n    \"domain_matching\": \"\", \r\n    \"variation_matching\": \"\", \r\n    \"dns_resolve\": {\"A\": [], \"MX\": [], \"TXT\": [], ...},\r\n    \"website_info\": {\"headers\":{...}, \"redirect\": 1, \"website_title\": \"\"}\r\n}\r\n~~~~\r\n\r\n\r\n\r\n# List of Resource Record use\r\n\r\n```\r\nNONE\r\nA\r\nNS\r\nMD\r\nMF\r\nCNAME\r\nSOA\r\nMB\r\nMG\r\nMR\r\nNULL\r\nWKS\r\nPTR\r\nHINFO\r\nMINFO\r\nMX\r\nTXT\r\nRP\r\nAFSDB\r\nX25\r\nISDN\r\nRT\r\nNSAP\r\nNSAP_PTR\r\nSIG\r\nKEY\r\nPX\r\nGPOS\r\nAAAA\r\nLOC\r\nNXT\r\nSRV\r\nNAPTR\r\nKX\r\nCERT\r\nA6\r\nDNAME\r\nOPT\r\nAPL\r\nDS\r\nSSHFP\r\nIPSECKEY\r\nRRSIG\r\nNSEC\r\nDNSKEY\r\nDHCID\r\nNSEC3\r\nNSEC3PARAM\r\nTLSA\r\nHIP\r\nCDS\r\nCDNSKEY\r\nCSYNC\r\nSPF\r\nUNSPEC\r\nEUI48\r\nEUI64\r\nTKEY\r\nTSIG\r\nIXFR\r\nAXFR\r\nMAILB\r\nMAILA\r\nANY\r\nURI\r\nCAA\r\nTA\r\nDLV\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fail-project%2Fail-feeder-ct","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fail-project%2Fail-feeder-ct","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fail-project%2Fail-feeder-ct/lists"}