{"id":20628282,"url":"https://github.com/huakunshen/cron-crawler-template","last_synced_at":"2026-05-15T08:36:57.984Z","repository":{"id":110922960,"uuid":"467552713","full_name":"HuakunShen/cron-crawler-template","owner":"HuakunShen","description":"Web Crawler Cron Job Template running with GitHub Action. Capable of sending email notifications.","archived":false,"fork":false,"pushed_at":"2022-08-01T01:33:07.000Z","size":21,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-19T04:10:56.250Z","etag":null,"topics":["crawler","github-actions","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HuakunShen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-08T14:46:22.000Z","updated_at":"2022-03-09T19:21:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"c337a476-5330-4e6a-8173-cefd4eeafba9","html_url":"https://github.com/HuakunShen/cron-crawler-template","commit_stats":null,"previous_names":[],"tags_count":0,"template":true,"template_full_name":null,"purl":"pkg:github/HuakunShen/cron-crawler-template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HuakunShen%2Fcron-crawler-template","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HuakunShen%2Fcron-crawler-template/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HuakunShen%2Fcron-crawler-template/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HuakunShen%2Fcron-crawler-template/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HuakunShen","download_url":"https://codeload.github.com/HuakunShen/cron-crawler-template/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HuakunShen%2Fcron-crawler-template/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33059618,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-13T13:14:54.681Z","status":"online","status_checked_at":"2026-05-15T02:00:06.351Z","response_time":103,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","github-actions","python"],"created_at":"2024-11-16T13:19:43.214Z","updated_at":"2026-05-15T08:36:57.954Z","avatar_url":"https://github.com/HuakunShen.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cron-crawler-template\n\nThis is a template allowing you to run crawler cron jobs with GitHub Action and send email notification to yourself.\n\nCrawlers should be defined within [jobs](./jobs) directory and extend abstract class [AbstractJob](./src/job.py).\n\nThere are 2 job sample files:\n\n- [jobs/worldtime.py](./jobs/worldtime.py)\n- [jobs/daily/worldtime2.py](./jobs/daily/worldtime2.py)\n\njobs can be placed in nested folders under [jobs](./jobs) folder.\n\nThe class name for all jobs must be `Job`. \n\nNon `.py` files can also be placed under [jobs](./jobs) folder but will be ignored, you may add template files if you want.\n\nHelper functions can also be defined in separate files under [jobs](./jobs) folder. All files with a `Job` class will be loaded and the `Job` class will be run.\n\nNote that the initializer of a `Job` class doesn't take any arguments, so define all variables you need within the class.\n\nYou can have a subfolder storing daily jobs, another one storing weekly jobs, etc.\n\n# How the jobs are run?\n\n## Run all jobs\n\n[main.py](./main.py) will run all jobs under [jobs](./jobs) directory.\n\n```bash\npython main.py\n```\n\n## Run a single file or a folder of files\n\n```bash\npython run.py --modules ./jobs/daily            # run all jobs under ./jobs/daily folder\npython run.py --modules ./jobs/worldtime.py     # run a single job\n```\n\n# GitHub Action\n\n[all-jobs-daily.yml](./.github/workflows/all-jobs-daily.yml) is a sample github workflow for running [main.py](./main.py) (all jobs) daily.\n\n[report-time-daily.yml](./.github/workflows/report-time-daily.yml) is a sample github workflow for running [run.py](./run.py) on a folder daily.\n\n# Email Notification\n\nTo send email notifications, the code is in file [./src/notifier.email_notifier.py](./src/notifier/email_notifier.py).\n\nSample usage can be found in the sample job file [jobs/worldtime.py](https://github.com/HuakunShen/cron-crawler-template/blob/master/jobs/worldtime.py#L24).\n\nIn order for email notification to work, you must provide your email server address, email addresses and email password.\n\nYou can put all non-secret environment variables in [public.env](./public.env).\n\nMy `MAIL_SERVER` variable is defined in [public.env](./public.env).\n\nSecret environment variables are not recommended to be pushed to GitHub.\n\nWhen developing locally, you can create a `.env` file to store secret environment variables.\n\nThe following 3 variables should be defined in order for email notification to work\n\n- `EMAIL_PASSWORD`\n- `FROM_EMAIL_ADDRESS`\n- `TO_EMAIL_ADDRESS`\n\nWhen running cron job with github action, you may add the variables to GitHub Action Secrets, and don't for forget to declare these varaibles in the workflow yaml files. \n\nFor example,\n\n```yaml\n...\njobs:\n  crawl:\n    steps:\n      - name: Run the main crawler\n        env:\n          EMAIL_PASSWORD: ${{ secrets.EMAIL_PASSWORD }}\n          FROM_EMAIL_ADDRESS: ${{ secrets.FROM_EMAIL_ADDRESS }}\n          TO_EMAIL_ADDRESS: ${{ secrets.TO_EMAIL_ADDRESS }}\n        run: python main.py\n```\n\n# Add Your Custom Features\n\nYou may add any custom features to for your crawler. This is just a template.\n\nAll source code can be found in [./src](./src) folder. If you need to add an SMS notifier, modify the email notifier or add a email html template, just add/modify the files in `src/notifier`.\n\nNote that if you add any secret environment variables, use `.env` when developing locally and GitHub Action Secrets when running as a GitHub Workflow, and don't forget to declare the variables used in your GitHub Workflow yaml files.\n\nIf any other python packages were used, don't forget to add the dependencies to [requirements.txt](./requirements.txt).\n\nWhen code is pushed to `develop` branch, the GitHub Workflows (the samples I provided) will be run, otherwise they will follow the cron schedule defined.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuakunshen%2Fcron-crawler-template","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuakunshen%2Fcron-crawler-template","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuakunshen%2Fcron-crawler-template/lists"}