{"id":13815094,"url":"https://github.com/MarshalX/telegram-crawler","last_synced_at":"2025-05-15T07:31:53.603Z","repository":{"id":37432898,"uuid":"360868092","full_name":"MarshalX/telegram-crawler","owner":"MarshalX","description":"🕷 Automatically detect changes made to the official Telegram sites, clients and servers.","archived":false,"fork":false,"pushed_at":"2025-05-08T06:14:06.000Z","size":808126,"stargazers_count":303,"open_issues_count":0,"forks_count":34,"subscribers_count":17,"default_branch":"main","last_synced_at":"2025-05-08T07:24:19.235Z","etag":null,"topics":["crawler","crawling","crawling-python","parser","telegram","telegram-org","telegram-updates"],"latest_commit_sha":null,"homepage":"https://t.me/tgcrawl","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MarshalX.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"MarshalX"}},"created_at":"2021-04-23T11:56:16.000Z","updated_at":"2025-05-03T19:13:45.000Z","dependencies_parsed_at":"2024-04-15T15:33:14.951Z","dependency_job_id":"d9a2f14f-8ee9-4083-90a0-d94503884b95","html_url":"https://github.com/MarshalX/telegram-crawler","commit_stats":{"total_commits":1367,"total_committers":8,"mean_commits":170.875,"dds":0.1923920994879298,"last_synced_commit":"e09262fe44c41130edcb85f2681c89b2d010bb5e"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarshalX%2Ftelegram-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarshalX%2Ftelegram-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarshalX%2Ftelegram-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarshalX%2Ftelegram-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MarshalX","download_url":"https://codeload.github.com/MarshalX/telegram-crawler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254295941,"owners_count":22047173,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","crawling","crawling-python","parser","telegram","telegram-org","telegram-updates"],"created_at":"2024-08-04T04:02:57.364Z","updated_at":"2025-05-15T07:31:53.093Z","avatar_url":"https://github.com/MarshalX.png","language":"Python","funding_links":["https://github.com/sponsors/MarshalX"],"categories":["Python"],"sub_categories":[],"readme":"## 🕷 Telegram Crawler\n\nThis project is developed to automatically detect changes made \nto the official Telegram sites and beta clients. This is necessary for \nanticipating future updates and other things \n(new vacancies, API updates, etc).\n\n| Name                 | Commits  | Status                                                                                                                                                        |\n|----------------------| -------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| Data tracker         | [Commits](https://github.com/MarshalX/telegram-crawler/commits/data)  | ![Fetch new content of tracked links and files](https://github.com/MarshalX/telegram-crawler/actions/workflows/make_files_tree.yml/badge.svg?branch=main)     |\n| Site links collector | [Commits](https://github.com/MarshalX/telegram-crawler/commits/main/tracked_links.txt)  | ![Generate or update list of tracked links](https://github.com/MarshalX/telegram-crawler/actions/workflows/make_tracked_links_list.yml/badge.svg?branch=main) |\n\n* ✅ passing – new changes\n* ❌ failing – no changes\n\nYou should to subscribe to **[channel with alerts](https://t.me/tgcrawl)** to stay updated.\nCopy of Telegram websites and client`s resources stored **[here](https://github.com/MarshalX/telegram-crawler/tree/data/data)**.\n\n![GitHub pretty diff example](https://i.imgur.com/BK8UAju.png)\n\n### How it works\n\n1. [Link crawling](make_tracked_links_list.py) runs **as often as possible**. \n   Starts crawling from the home page of the site. \n   Detects relative and absolute sub links and recursively repeats the operation. \n   Writes a list of unique links for future content comparison. \n   Additionally, there is the ability to add links by hand to help the script \n   find more hidden (links to which no one refers) links. To manage exceptions,\n   there is a [system of rules](#example-of-link-crawler-rules-configuration)\n   for the link crawler.\n\n2. [Content crawling](make_files_tree.py) is launched **as often as \n   possible** and uses the existing list of links collected in step 1. \n   Going through the base it gets contains and builds a system of subfolders \n   and files. Removes all dynamic content from files. It downloads beta version \n   of Android Client, decompiles it and track resources also. Tracking of \n   resources of Telegram for macOS presented too.\n\n3. Using of [GitHub Actions](.github/workflows/). Works without own servers.\n   You can just fork this repository and own tracker system by yourself.\n   Workflows launch scripts and commit changes. All file changes are tracked \n   by GIT and beautifully displayed on GitHub. GitHub Actions should be built\n   correctly only if there are changes on the Telegram website. Otherwise, the \n   workflow should fail. If build was successful, we can send notifications to \n   Telegram channel and so on.\n\n### FAQ\n\n**Q:** How often is \"**as often as possible**\"?\n\n**A:** TLTR: content update action runs every ~10 minutes. More info:\n- [Scheduled actions cannot be run more than once every 5 minutes.](https://github.blog/changelog/2019-11-01-github-actions-scheduled-jobs-maximum-frequency-is-changing/)\n- [GitHub Actions workflow not triggering at scheduled time](https://upptime.js.org/blog/2021/01/22/github-actions-schedule-not-working/).\n\n**Q:** Why there is 2 separated crawl scripts instead of one?\n\n**A:** Because the previous idea was to update tracked links once at hour.\nIt was so comfortably to use separated scripts and workflows.\nAfter Telegram 7.7 update, I realised that find new blog posts so slowly is bad idea.\n\n**Q:** Why alert for sending alerts have while loop?\n\n**A:** Because GitHub API doesn't return information about commit immediately \nafter push to repository. Therefore, script are waiting for information to appear...\n\n**Q:** Why are you using GitHub Personal Access Token in action/checkout workflow`s step?\n\n**A:** To have ability to trigger other workflows by on push trigger. More info:\n- [Action does not trigger another on push tag action ](https://github.community/t/action-does-not-trigger-another-on-push-tag-action/17148)\n\n**Q:** Why are you using GitHub PAT in [make_and_send_alert.py](make_and_send_alert.py)?\n\n**A:** To increase limits of GitHub API.\n\n**Q:** Why are you decompiling .apk file each run?\n\n**A:** Because it doesn't require much time. I am decompiling only \nresources (-s flag of apktool to disable disassembly of dex files). \nWriting a check for the need for decompilation by the hash of the apk file \nwould take more time.\n\n### Example of link crawler rules configuration\n\n```python\nCRAWL_RULES = {\n    # every rule is regex\n    # empty string means match any url\n    # allow rules with higher priority than deny\n    'translations.telegram.org': {\n        'allow': {\n            r'^[^/]*$',  # root\n            r'org/[^/]*/$',  # 1 lvl sub\n            r'/en/[a-z_]+/$'  # 1 lvl after /en/\n        },\n        'deny': {\n            '',  # all\n        }\n    },\n    'bugs.telegram.org': {\n        'deny': {\n            '',    # deny all sub domain\n        },\n    },\n}\n```\n\n### Current hidden urls list\n\n```python\nHIDDEN_URLS = {\n    # 'corefork.telegram.org', # disabled\n\n    'telegram.org/privacy/gmailbot',\n    'telegram.org/tos',\n    'telegram.org/tour',\n    'telegram.org/evolution',\n\n    'desktop.telegram.org/changelog',\n}\n```\n\n### License\n\nLicensed under the [MIT License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMarshalX%2Ftelegram-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMarshalX%2Ftelegram-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMarshalX%2Ftelegram-crawler/lists"}