https://github.com/Xumeiquer/nobots

Caddy Server plugin to protect your website against web crawlers and bots
https://github.com/Xumeiquer/nobots

Last synced: 6 days ago
JSON representation

Caddy Server plugin to protect your website against web crawlers and bots

• Host: GitHub
• URL: https://github.com/Xumeiquer/nobots
• Owner: Xumeiquer
• License: mit
• Created: 2017-07-07T22:13:30.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2020-09-08T08:12:53.000Z (about 4 years ago)
• Last Synced: 2024-08-01T13:31:20.804Z (3 months ago)
• Language: Go
• Homepage:
• Size: 60.5 KB
• Stars: 30
• Watchers: 2
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# NoBots

Caddy Server plugin to protect your website against web crawlers and bots.

## Usage

The directive for the Caddyfile is really simple. First, you have to place the bomb path next to the `nobots` keyword, for example `bomb.gz` in the example below.

Then you can specify user agent either as strings or regular expresions. When using regular expresions you must add the `regexp` keyword in front of the regex.

Caddyfile example:

```

nobots "bomb.gz" {

  "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

  "DuckDuckBot"

  regexp "^[Bb]ot"

  regexp "bingbot"

}

```

There is another keyword that is useful in case you want to allow crawlers and bots navigate through specific parts of your website. The keyword is `public` and its values are regular expresions, so you can use it as following:

```

nobots "bomb.gz" {

  "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

  public "^/public"

  public "^/[a-z]{,5}/public"

}

```

The above example will send the bot to all URIs except those that match with `/public` and `[a-z]{,5}/public`.

NOTE: By default all URIs.

## How to create a bomb

The bomb is not provided within the plugin so you have to create one. In Linux it is really easy, you can use the following commands.

```

dd if=/dev/zero bs=1M count=1024 | gzip > 1G.gzip

dd if=/dev/zero bs=1M count=10240 | gzip > 10G.gzip

dd if=/dev/zero bs=1M count=1048576 | gzip > 1T.gzip

```

To optimize the final bomb you may compress the parts several times:

```

cat 10G.gzip | gzip > 10G.gzipx2

cat 1T.gzip | gzip | gzip | gzip > 1T.gzipx4

 ```

*NOTE*: The extension `.gzipx2` or `.gzipx4` is only to highlight how many times the file was compressed.

[![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/M4M625UW0)