{"id":13930034,"url":"https://github.com/swyxio/gh-action-data-scraping","last_synced_at":"2025-10-14T17:34:17.196Z","repository":{"id":37294992,"uuid":"235434692","full_name":"swyxio/gh-action-data-scraping","owner":"swyxio","description":"this shows how to use github actions to do periodic data scraping","archived":false,"fork":false,"pushed_at":"2025-10-12T15:00:30.000Z","size":20947,"stargazers_count":235,"open_issues_count":3,"forks_count":37,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-10-12T15:03:57.290Z","etag":null,"topics":["cron","gh-actions","git-scraping"],"latest_commit_sha":null,"homepage":"https://www.swyx.io/github-scraping/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/swyxio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-01-21T20:25:25.000Z","updated_at":"2025-10-12T15:00:34.000Z","dependencies_parsed_at":"2023-12-27T23:20:15.838Z","dependency_job_id":"8a7f4628-ffe0-471b-b32d-d995591b7c77","html_url":"https://github.com/swyxio/gh-action-data-scraping","commit_stats":null,"previous_names":["sw-yx/gh-action-data-scraping"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/swyxio/gh-action-data-scraping","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swyxio%2Fgh-action-data-scraping","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swyxio%2Fgh-action-data-scraping/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swyxio%2Fgh-action-data-scraping/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swyxio%2Fgh-action-data-scraping/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/swyxio","download_url":"https://codeload.github.com/swyxio/gh-action-data-scraping/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/swyxio%2Fgh-action-data-scraping/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279014662,"owners_count":26085554,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cron","gh-actions","git-scraping"],"created_at":"2024-08-07T18:02:50.417Z","updated_at":"2025-10-14T17:34:17.176Z","avatar_url":"https://github.com/swyxio.png","language":"JavaScript","funding_links":[],"categories":["others"],"sub_categories":[],"readme":"# gh-action-data-scraping\n\nthis repo shows how to use github actions to do automated data scraping, with storage in git itself! **free git storage and scheduled updates!!!**\n\n## 2021 Update\n\nYou can read more in the [Blog Writeup](https://www.swyx.io/github-scraping/). \n\nAs of May 2021, [Flat Data scraping](https://octo.github.com/projects/flat-data) is officially supported by GitHub, check them out.\n\n## Basic Idea\n\n- You set a cron triggered github action ([cron examples](https://crontab.guru/examples.html) - max frequency every 5 mins)\n- it checks out your repo with https://github.com/actions/checkout\n- `npm install` and run your scrape script, write files to somewhere in your repo. This repo uses Node, but you can use whatever language you want\n- check it back in with https://github.com/mikeal/publish-to-github-action\n\nThe script looks like:\n\n```yaml\n# /.github/workflows/daily.yml\non:\n  schedule:\n    - cron:  '0 8 * * *' # every day at 8am\nname: Pull Data and Build\njobs:\n  build:\n    name: Build\n    runs-on: ubuntu-latest\n    steps:\n    - uses: actions/checkout@master\n    - name: Build\n      run: npm install\n    - name: Scrape\n      run: npm run action \n      # env:\n      #   WHATEVER_TOKEN: ${{ secrets.YOU_WANT }}\n    - uses: mikeal/publish-to-github-action@master\n      env:\n        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # GitHub sets this for you\n```\n## How it should look\n\nFor people new to GH actions, this is how my Actions tab of this very repo looks if you need a reference point:\n\n\n![image](https://user-images.githubusercontent.com/6764957/72847135-efc62c80-3c6f-11ea-88d8-2a2545a292e7.png)\n\n\n\n## Limits\n\nYou can do whatever you like with this, including taking screenshots of sites!\n\nThe limits I can think of are the limits of GitHub and GitHub Actions:\n\n- The max frequency of cronjobs on GitHub actions is every 5 minutes. For more frequent scraping, you will have to turn elsewhere.\n- GitHub has a [soft storage limit of 1GB](https://www.quora.com/What-is-the-max-storage-limit-per-repository-in-GitHub)\n  - You can [work around this with Git LFS](https://twitter.com/mikeal/status/1219739811159801856) if you have to!\n- Actions are free for public repos, but incur costs for private repos\n  - [You get 6 Concurrent jobs, 1000 API requests an hour, and each job can take up to 6(!) hours](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-github-actions#usage-limits)\n\nIn addition to these limits, GitHub Actions should not be used for:\n\n- Content or activity that is illegal or otherwise prohibited by their Terms of Service or Community Guidelines.\n- Cryptomining\n- Serverless computing\n- Activity that compromises GitHub users or GitHub services.\n- Any other activity unrelated to the production, testing, deployment, or publication of the software project associated with the repository where GitHub Actions are used. In other words, be cool, don’t use GitHub Actions in ways you know you shouldn’t. \n\nBe a good citizen, **don't abuse it and F this up for the rest of us**!\n\n\n## This is heavily based on\n\n- https://github.com/mikeal/daily/blob/master/.github/workflows/daily.yml\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fswyxio%2Fgh-action-data-scraping","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fswyxio%2Fgh-action-data-scraping","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fswyxio%2Fgh-action-data-scraping/lists"}