https://github.com/shufo/plug_robots
An Elixir plug serving robots.txt.
https://github.com/shufo/plug_robots
elixir
Last synced: 8 months ago
JSON representation
An Elixir plug serving robots.txt.
- Host: GitHub
- URL: https://github.com/shufo/plug_robots
- Owner: shufo
- Created: 2017-04-04T16:45:55.000Z (about 9 years ago)
- Default Branch: master
- Last Pushed: 2023-12-15T05:42:35.000Z (over 2 years ago)
- Last Synced: 2025-02-06T22:54:27.534Z (over 1 year ago)
- Topics: elixir
- Language: Elixir
- Homepage: https://hex.pm/packages/plug_robots
- Size: 5.86 KB
- Stars: 3
- Watchers: 3
- Forks: 1
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# PlugRobots
An Elixir plug serving robots.txt.
## Installation
Add plug_robots to your list of dependencies in mix.exs:
```elixir
def deps do
[{:plug_robots, "~> 0.1.0"}]
end
```
## Usage
- Add it to your endpoint before the router
`endpoint.ex`
```elixir
plug PlugRobots, rules: :allow_all
plug MyApp.Router
```
## Examples
- Deny all crawler
```elixir
plug PlugRobots, rules: :deny_all
plug MyApp.Router
```
- Returns different rules by environment
```elixir
case Mix.env do
:prod -> plug PlugRobots, rules: :allow_all
_ -> plug PlugRobots, rules: :deny_all
end
plug MyApp.Router
```
- Returns custom rules
```elixir
@googlebot_rule """
User-agent: GoogleBot
Disallow: /images/dogs.jpg
"""
plug PlugRobots, rules: @googlebot_rule
plug MyApp.Router
```
### Options
- `rules`
- `:allow_all`: Allow all crawlers to all path
- `:deny_all`: Disallow all crawlers to all path
- string: any other custom rules