{"id":24167318,"url":"https://github.com/s-thom/create-robots-txt-action","last_synced_at":"2026-05-15T23:11:16.697Z","repository":{"id":268352625,"uuid":"904050793","full_name":"s-thom/create-robots-txt-action","owner":"s-thom","description":"An action to create a robots.txt file from different sources","archived":false,"fork":false,"pushed_at":"2026-03-27T02:20:31.000Z","size":4653,"stargazers_count":1,"open_issues_count":11,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-30T05:39:47.693Z","etag":null,"topics":["action","actions","gh-action","gh-actions","github-action","github-actions","robots-txt","robotstxt"],"latest_commit_sha":null,"homepage":"https://github.com/marketplace/actions/create-robots-txt","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/s-thom.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-12-16T06:49:27.000Z","updated_at":"2026-01-02T19:08:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"b05e7507-9c0e-4617-907c-8fdf4996e7d7","html_url":"https://github.com/s-thom/create-robots-txt-action","commit_stats":null,"previous_names":["s-thom/create-robots-txt-action"],"tags_count":4,"template":false,"template_full_name":"actions/typescript-action","purl":"pkg:github/s-thom/create-robots-txt-action","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s-thom%2Fcreate-robots-txt-action","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s-thom%2Fcreate-robots-txt-action/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s-thom%2Fcreate-robots-txt-action/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s-thom%2Fcreate-robots-txt-action/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/s-thom","download_url":"https://codeload.github.com/s-thom/create-robots-txt-action/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s-thom%2Fcreate-robots-txt-action/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33082942,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-15T20:25:35.270Z","status":"ssl_error","status_checked_at":"2026-05-15T20:25:34.732Z","response_time":103,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["action","actions","gh-action","gh-actions","github-action","github-actions","robots-txt","robotstxt"],"created_at":"2025-01-12T21:13:06.038Z","updated_at":"2026-05-15T23:11:16.691Z","avatar_url":"https://github.com/s-thom.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Create Robots.txt Action\n\nAn action to create a [robots.txt](https://en.wikipedia.org/wiki/Robots.txt) file from a variety of sources.\n\n| Input name               | Example           | Description                                                                                                    |\n| ------------------------ | ----------------- | -------------------------------------------------------------------------------------------------------------- |\n| output-file              | `robots.txt`      | Where to write the resulting robots.txt file                                                                   |\n| input-file               | `base-robots.txt` | An existing robots.txt. Will be added to the top of the output file. Must not be the same as the `output-file` |\n| append-allow-rule        | `true`            | Whether to add an allow for all unspecified user agents to the end of the file                                 |\n| allowed-bot-names        |                   | Multiline string. Names of bots that should not be included in the blocklist                                   |\n| blocked-bot-names        |                   | Multiline string. Names of bots that should be included in the blocklist                                       |\n| cloudflare-api-token     |                   | An API token for Cloudflare. Enables Cloudflare's bot categories as a source for bots                          |\n| cloudflare-categories    | `AI Crawler`      | Bot categories to add to the blocklist. Required if `cloudflare-api-token` is set                              |\n| dark-visitors-api-token  |                   | An API token for Dark Visitors. Enables Dark Visitors' user agent categories as a source for bots              |\n| dark-visitors-categories | `AI Data Scraper` | User agent categories to add to the blocklist. Required if `dark-visitors-api-token` is set                    |\n\n## Example workflow.yml\n\n\u003e [!NOTE]\n\u003e\n\u003e You will need to enable the \"Allow GitHub Actions to create and approve pull requests\" option in your repository's `Settings \u003e Actions \u003e General`\n\n```yml\nname: Update robots.txt\n\non:\n  workflow_dispatch:\n  schedule:\n    - cron: \"13 6 * * 1\"\n\njobs:\n  update-robots-txt:\n    name: Update robots.txt\n    runs-on: ubuntu-latest\n    permissions:\n      contents: write\n      pull-requests: write\n    steps:\n      - name: Checkout\n        uses: actions/checkout@v4\n\n      - name: Create robots.txt\n        uses: s-thom/create-robots-txt-action@v1\n        with:\n          output-file: public/robots.txt\n          append-allow-rule: true\n          allowed-bot-names: |\n            Chrome-Lighthouse\n          cloudflare-api-token: ${{ secrets.CLOUDFLARE_RADAR_API_TOKEN }}\n          cloudflare-categories: |\n            YOUR BOT CATEGORIES HERE\n          dark-visitors-api-token: ${{ secrets.DARK_VISITORS_API_TOKEN }}\n          dark-visitors-categories: |\n            YOUR BOT CATEGORIES HERE\n\n      - name: Create Pull Request\n        uses: peter-evans/create-pull-request@v7\n        with:\n          add-paths: |\n            public/robots.txt\n          commit-message: \"Update robots.txt\"\n          branch: robots-txt\n          delete-branch: true\n          author: github-actions[bot] \u003c41898282+github-actions[bot]@users.noreply.github.com\u003e\n          committer: github-actions[bot] \u003c41898282+github-actions[bot]@users.noreply.github.com\u003e\n          title: \"Update robots.txt\"\n          body: |\n            # Automated update of robots.txt\n\n            Generated by [s-thom/create-robots-txt-action](https://github.com/s-thom/create-robots-txt-action)\n```\n\n## Known bot categories for each provider\n\n### Cloudflare\n\nTo create an API token, go to [the API Tokens page](https://dash.cloudflare.com/profile/api-tokens) in your profile settings and create a new token. Use the \"Read Cloudflare Radar data\" template or create a custom one with the \"Account \u003e Radar \u003e Read\" permission. Create the token and add it as an actions secret in your GitHub repository.\n\nIt is worth creating a new token for this workflow even if you already have one set up in your repository. It is good practice to give the least amount of privilege to any tokens given to third-party code, such as this action.\n\n- Accessibility\n- Advertising \u0026 Marketing\n- Aggregator\n- AI Assistant\n- AI Crawler\n- AI Search\n- Archiver\n- Feed Fetcher\n- Monitoring \u0026 Analytics\n- Page Preview\n- Search Engine Crawler\n- Search Engine Optimization\n- Security\n- Social Media Marketing\n- Webhooks\n- Other\n\n### Dark Visitors\n\nTo find your API token, go to the Settings page for your project. The access token is visible on this page. Copy the token and add it as an actions secret in your GitHub repository.\n\n- AI Assistants​\n- AI Data Scrapers​\n- AI Search Crawlers​\n\n\u003e [!NOTE]\n\u003e\n\u003e While Dark Visitors also defines these other categories, they do not include bots from these categories in their API.\n\u003e\n\u003e - Archivers​\n\u003e - Developer Helpers​\n\u003e - Fetchers​\n\u003e - Headless Browsers​\n\u003e - Intelligence Gatherers​\n\u003e - Scrapers​\n\u003e - Search Engine Crawlers​\n\u003e - SEO Crawlers​\n\u003e - Uncategorized​\n\u003e - Undocumented AI Agents\n\n\u003c!-- markdownlint-disable MD033 --\u003e\n\u003cdetails\u003e\n\u003csummary\u003eDevelopment instructions\u003c/summary\u003e\n\u003c!-- markdownlint-enable MD033 --\u003e\n\n## Initial Setup\n\nAfter you've cloned the repository to your local machine or codespace, you'll\nneed to perform some initial setup steps before you can develop your action.\n\n\u003e [!NOTE]\n\u003e\n\u003e You'll need to have a reasonably modern version of\n\u003e [Node.js](https://nodejs.org) handy (20.x or later should work!). If you are\n\u003e using a version manager like [`nodenv`](https://github.com/nodenv/nodenv) or\n\u003e [`fnm`](https://github.com/Schniz/fnm), this template has a `.node-version`\n\u003e file at the root of the repository that can be used to automatically switch to\n\u003e the correct version when you `cd` into the repository. Additionally, this\n\u003e `.node-version` file is used by GitHub Actions in any `actions/setup-node`\n\u003e actions.\n\n1. :hammer_and_wrench: Install the dependencies\n\n   ```bash\n   npm install\n   ```\n\n1. :building_construction: Package the TypeScript for distribution\n\n   ```bash\n   npm run bundle\n   ```\n\n1. :white_check_mark: Run the tests\n\n   ```bash\n   $ npm test\n\n   PASS  ./index.test.js\n     ✓ throws invalid number (3ms)\n     ✓ wait 500 ms (504ms)\n     ✓ test runs (95ms)\n\n   ...\n   ```\n\n## Update the Action Metadata\n\nThe [`action.yml`](action.yml) file defines metadata about your action, such as\ninput(s) and output(s). For details about this file, see\n[Metadata syntax for GitHub Actions](https://docs.github.com/en/actions/creating-actions/metadata-syntax-for-github-actions).\n\nWhen you copy this repository, update `action.yml` with the name, description,\ninputs, and outputs for your action.\n\n## Update the Action Code\n\nThe [`src/`](./src/) directory is the heart of your action! This contains the\nsource code that will be run when your action is invoked. You can replace the\ncontents of this directory with your own code.\n\nThere are a few things to keep in mind when writing your action code:\n\n- Most GitHub Actions toolkit and CI/CD operations are processed asynchronously.\n  In `main.ts`, you will see that the action is run in an `async` function.\n\n  ```javascript\n  import * as core from \"@actions/core\";\n  //...\n\n  async function run() {\n    try {\n      //...\n    } catch (error) {\n      core.setFailed(error.message);\n    }\n  }\n  ```\n\n  For more information about the GitHub Actions toolkit, see the\n  [documentation](https://github.com/actions/toolkit/blob/master/README.md).\n\nSo, what are you waiting for? Go ahead and start customizing your action!\n\n1. Create a new branch\n\n   ```bash\n   git checkout -b releases/v1\n   ```\n\n1. Replace the contents of `src/` with your action code\n1. Add tests to `__tests__/` for your source code\n1. Format, test, and build the action\n\n   ```bash\n   npm run all\n   ```\n\n   \u003e This step is important! It will run [`ncc`](https://github.com/vercel/ncc)\n   \u003e to build the final JavaScript action code with all dependencies included.\n   \u003e If you do not run this step, your action will not work correctly when it is\n   \u003e used in a workflow. This step also includes the `--license` option for\n   \u003e `ncc`, which will create a license file for all of the production node\n   \u003e modules used in your project.\n\n1. (Optional) Test your action locally\n\n   The [`@github/local-action`](https://github.com/github/local-action) utility\n   can be used to test your action locally. It is a simple command-line tool\n   that \"stubs\" (or simulates) the GitHub Actions Toolkit. This way, you can run\n   your TypeScript action locally without having to commit and push your changes\n   to a repository.\n\n   The `local-action` utility can be run in the following ways:\n   - Visual Studio Code Debugger\n\n     Make sure to review and, if needed, update\n     [`.vscode/launch.json`](./.vscode/launch.json)\n\n   - Terminal/Command Prompt\n\n     ```bash\n     # npx local action \u003caction-yaml-path\u003e \u003centrypoint\u003e \u003cdotenv-file\u003e\n     npx local-action . src/main.ts .env\n     ```\n\n   You can provide a `.env` file to the `local-action` CLI to set environment\n   variables used by the GitHub Actions Toolkit. For example, setting inputs and\n   event payload data used by your action. For more information, see the example\n   file, [`.env.example`](./.env.example), and the\n   [GitHub Actions Documentation](https://docs.github.com/en/actions/learn-github-actions/variables#default-environment-variables).\n\n1. Commit your changes\n\n   ```bash\n   git add .\n   git commit -m \"My first action is ready!\"\n   ```\n\n1. Push them to your repository\n\n   ```bash\n   git push -u origin releases/v1\n   ```\n\n1. Create a pull request and get feedback on your action\n1. Merge the pull request into the `main` branch\n\nYour action is now published! :rocket:\n\nFor information about versioning your action, see\n[Versioning](https://github.com/actions/toolkit/blob/master/docs/action-versioning.md)\nin the GitHub Actions toolkit.\n\n## Validate the Action\n\nYou can now validate the action by referencing it in a workflow file. For\nexample, [`ci.yml`](./.github/workflows/ci.yml) demonstrates how to reference an\naction in the same repository.\n\n```yaml\nsteps:\n  - name: Checkout\n    id: checkout\n    uses: actions/checkout@v4\n\n  - name: Test Local Action\n    id: test-action\n    uses: ./\n    with:\n      milliseconds: 1000\n\n  - name: Print Output\n    id: output\n    run: echo \"${{ steps.test-action.outputs.time }}\"\n```\n\nFor example workflow runs, check out the\n[Actions tab](https://github.com/actions/typescript-action/actions)! :rocket:\n\n## Usage\n\nAfter testing, you can create version tag(s) that developers can use to\nreference different stable versions of your action. For more information, see\n[Versioning](https://github.com/actions/toolkit/blob/master/docs/action-versioning.md)\nin the GitHub Actions toolkit.\n\nTo include the action in a workflow in another repository, you can use the\n`uses` syntax with the `@` symbol to reference a specific branch, tag, or commit\nhash.\n\n```yaml\nsteps:\n  - name: Checkout\n    id: checkout\n    uses: actions/checkout@v4\n\n  - name: Test Local Action\n    id: test-action\n    uses: actions/typescript-action@v1 # Commit with the `v1` tag\n    with:\n      milliseconds: 1000\n\n  - name: Print Output\n    id: output\n    run: echo \"${{ steps.test-action.outputs.time }}\"\n```\n\n## Publishing a New Release\n\nThis project includes a helper script, [`script/release`](./script/release)\ndesigned to streamline the process of tagging and pushing new releases for\nGitHub Actions.\n\nGitHub Actions allows users to select a specific version of the action to use,\nbased on release tags. This script simplifies this process by performing the\nfollowing steps:\n\n1. **Retrieving the latest release tag:** The script starts by fetching the most\n   recent SemVer release tag of the current branch, by looking at the local data\n   available in your repository.\n1. **Prompting for a new release tag:** The user is then prompted to enter a new\n   release tag. To assist with this, the script displays the tag retrieved in\n   the previous step, and validates the format of the inputted tag (vX.X.X). The\n   user is also reminded to update the version field in package.json.\n1. **Tagging the new release:** The script then tags a new release and syncs the\n   separate major tag (e.g. v1, v2) with the new release tag (e.g. v1.0.0,\n   v2.1.2). When the user is creating a new major release, the script\n   auto-detects this and creates a `releases/v#` branch for the previous major\n   version.\n1. **Pushing changes to remote:** Finally, the script pushes the necessary\n   commits, tags and branches to the remote repository. From here, you will need\n   to create a new release in GitHub so users can easily reference the new tags\n   in their workflows.\n\n\u003c/details\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs-thom%2Fcreate-robots-txt-action","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fs-thom%2Fcreate-robots-txt-action","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs-thom%2Fcreate-robots-txt-action/lists"}