https://github.com/ACP-CODE/astro-robots
A reliable robots.txt generator for Astro projects, offering zero-config setup and Verified Bots support.
https://github.com/ACP-CODE/astro-robots
astro-integration robots-txt verified-bots
Last synced: 3 months ago
JSON representation
A reliable robots.txt generator for Astro projects, offering zero-config setup and Verified Bots support.
- Host: GitHub
- URL: https://github.com/ACP-CODE/astro-robots
- Owner: ACP-CODE
- Created: 2023-09-04T11:10:01.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-12-16T10:31:41.000Z (7 months ago)
- Last Synced: 2025-03-30T23:31:51.596Z (3 months ago)
- Topics: astro-integration, robots-txt, verified-bots
- Language: TypeScript
- Homepage:
- Size: 240 KB
- Stars: 13
- Watchers: 1
- Forks: 3
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
Awesome Lists containing this project
README
**Latest Updates! 🎉 See the [change log](./CHANGELOG.md) for details.**
# astro-robots
It simplifies SEO management with a reliable robots.txt generator for Astro projects, offering zero-config setup and [Verified Bots](https://radar.cloudflare.com/traffic/verified-bots) support.
[](https://github.com/ACP-CODE/astro-robots/actions/workflows/release.yaml)
## Installation
> The package compatible with Astro 4.0.0 and later.
Quickly install with the `astro add` command:
```sh
npx astro add astro-robots
```
If you run into issues, try with [Manual Setup](#setup) guide.
## Usage
Manual Setup
#### Setup
To install manually, run:
```sh
npm install astro-robots
```
Add the integration to your Astro config file (`astro.config.*`):
```ts
// @ts-check
import { defineConfig } from "astro/config";
import robots from "astro-robots"; // Add code manually
export default defineConfig({
site: "https://example.com"; // If you want to support `@astrojs/sitemap` please provide this value
integrations: [robots()], // Add code manually
});
```
After installing, run `npm run build` or `yarn build` in terminal:
```sh
npm run build
```
This will output `robots.txt` to the `dist` folder with default rules:
```yaml
User-agent: *
Allow: /
# crawling rule(s) for above bots
Sitemap: https://example.com/sitemap-index.xml
```
#### Live Access
Start the server with `npm run dev`, then access the virtual `robots.txt` at `http://localhost:4321/robots.txt`.
Getting Started with Reference
To configure the integration, pass an object to the `robots()` function in your `astro.config.*` file:
```ts
// @ts-check
import { defineConfig } from "astro/config";
import robots from "astro-robots";
export default defineConfig({
//...
integrations: [
robots({
host: "https://example.com";,
sitemap: [
"https://example.com/sitemap.xml",
"https://www.example.com/sitemap.xml",
],
policy: [
{
userAgent: [
"Applebot",
"Googlebot",
"bingbot",
"Yandex",
"Yeti",
"Baiduspider",
"360Spider",
"*",
],
allow: ["/"],
disallow: ["/admin", "/login"],
crawlDelay: 5,
cleanParam: ["sid /", "s /forum/showthread"],
},
{
userAgent: "BLEXBot",
disallow: ["/assets", "/uploades/1989-08-21/*jpg$"],
},
],
}),
],
});
```
With the above configuration, the generated `robots.txt` file will look like this:
```yaml
User-agent: Applebot
User-agent: Googlebot
User-agent: bingbot
User-agent: Yandex
User-agent: Yeti
User-agent: Baiduspider
User-agent: 360Spider
User-agent: *
Allow: /
Disallow: /admin
Disallow: /login
Crawl-delay: 5
Clean-param: sid /
Clean-param: s /forum/showthread
User-agent: BLEXBot
Disallow: /assets
Disallow: /uploades/1989-08-21/*jpg$
# crawling rule(s) for above bots
Sitemap: https://example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap.xml
Host: example.com
```
> **Note:** Some directives like `Host`, `Clean-param`, and `Crawl-delay` may not be supported by all crawlers. For example, Yandex has ignored `Crawl-delay` since February 2018. To control Yandex's crawl rate, use the [Site crawl rate setting](https://yandex.com/support/webmaster/service/crawl-rate.html#crawl-rate) in Yandex Webmaster.
## Contributing
Submit your issues or feedback on our [GitHub](https://github.com/ACP-CODE/astro-robots/issues) channel.
## License
MIT