https://github.com/tanepiper/robots-webpack-plugin

A webpack plugin to output a robots.txt file with options per environment
https://github.com/tanepiper/robots-webpack-plugin

Last synced: about 1 month ago
JSON representation

A webpack plugin to output a robots.txt file with options per environment

• Host: GitHub
• URL: https://github.com/tanepiper/robots-webpack-plugin
• Owner: tanepiper
• Created: 2016-02-23T15:11:21.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2016-10-11T09:15:10.000Z (about 8 years ago)
• Last Synced: 2024-09-29T18:37:41.325Z (about 1 month ago)
• Language: JavaScript
• Size: 11.7 KB
• Stars: 14
• Watchers: 2
• Forks: 7
• Open Issues: 2
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
## Webpack Robots.txt

A webpack plugin for generating robots.txt files.

### Why is this plugin useful

When you have multiple environments such as dev, qa and production to want to ensure your non-production environments

are not exposed to search engines such as Google.

This plugin allows you to include this in your environment settings to generate a file.

### Install the plugin

`npm install --save-dev @tanepiper/robots-webpack-plugin`

### How to I use this plugin?

In your webpack config:

```

const RobotsPlugin = require('@tanepiper/robots-webpack-plugin');

const webpackConfig = {

  entry: 'src/index.js',

  output: {

    path: '/public',

    filename: 'index.js'

  },

  plugins: [new RobotsPlugin()]

};

```

The default output is an `Allow` directive for all user agents.  To configure, pass in an array to userAgents option.

Each one is an object with a `name:string`, `disallow:array` and `allow:array` in any combination.

There is also the sitemap options which will output a Sitemap directive at the end. Just provide a full url to the sitemap file.

```

plugins: [new RobotsPlugin({

  sitemap: 'https://example.com/sitemap.xml',

  userAgents: [{

    name: '*',

    disallow: ['/', '/cgi-bin'],

    allow: ['/blog']

  }, {

   name: 'Googlebot',

   disallow: ['/cgi-bin', '/some-path'],

   allow: ['/']

 }]

})]

```