Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/kouisamine/robots.txt-generator
A robots.txt file is a text file on a website that instructs web crawlers and search engine robots about which pages or sections of the site should not be crawled or indexed.
https://github.com/kouisamine/robots.txt-generator
generator html js online php robot-txt source-code tools txt-files website
Last synced: 6 days ago
JSON representation
A robots.txt file is a text file on a website that instructs web crawlers and search engine robots about which pages or sections of the site should not be crawled or indexed.
- Host: GitHub
- URL: https://github.com/kouisamine/robots.txt-generator
- Owner: KOUISAmine
- Created: 2024-01-01T21:36:51.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2024-01-01T21:41:35.000Z (11 months ago)
- Last Synced: 2024-01-02T00:09:34.189Z (11 months ago)
- Topics: generator, html, js, online, php, robot-txt, source-code, tools, txt-files, website
- Language: HTML
- Homepage: https://tools.waytolearnx.com/robots-txt-generator
- Size: 3.91 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0