https://github.com/agility/netcore-robotsmiddleware
Sample boilerplate code for blocking bots from crawling your site over a given domain name
https://github.com/agility/netcore-robotsmiddleware
Last synced: about 1 year ago
JSON representation
Sample boilerplate code for blocking bots from crawling your site over a given domain name
- Host: GitHub
- URL: https://github.com/agility/netcore-robotsmiddleware
- Owner: agility
- Created: 2019-10-24T16:14:06.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2019-10-24T16:20:10.000Z (over 6 years ago)
- Last Synced: 2025-02-14T04:51:13.254Z (over 1 year ago)
- Language: C#
- Size: 2.93 KB
- Stars: 0
- Watchers: 4
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# NetCore-RobotsMiddleware
Sample code for NetCore middleware for blocking bots from crawling your site over a given domain name (i.e. *.azurewebsites.net*). It works by overriding your **robots.txt** response when it is loaded over a given domain and it returns:
```
User-agent: *
Disallow: /
```
## Usage
1. Add the **RobotsMiddleware.cs** to your project.
2. Set the domain that you want to return `Disallow: /`
3. Register your middleware in your **Startup.cs** file.
```
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
...
//custom middleware defined in this website code
app.MapWhen(context => context.Request.Path.Value.EndsWith("robots.txt", true, null),
appBranch => { appBranch.UseRobotsHandler(); });
...
}
```
## Test
Run the site over the domain you want to `Disallow: /` by making a request to `https://{your-site}/robots.txt` and you should see this response:
```
User-agent: *
Disallow: /
```