https://github.com/ptsochantaris/can-proceed

A small, tested, no-frills parser of robots.txt files in Swift.
https://github.com/ptsochantaris/can-proceed

robots-parser robots-txt server-side-swift swift web-clients

Last synced: 5 days ago
JSON representation

A small, tested, no-frills parser of robots.txt files in Swift.

• Host: GitHub
• URL: https://github.com/ptsochantaris/can-proceed
• Owner: ptsochantaris
• License: mit
• Created: 2024-02-24T12:00:23.000Z (9 months ago)
• Default Branch: main
• Last Pushed: 2024-08-18T11:22:40.000Z (3 months ago)
• Last Synced: 2024-09-19T09:31:40.920Z (about 2 months ago)
• Topics: robots-parser, robots-txt, server-side-swift, swift, web-clients
• Language: Swift
• Homepage:
• Size: 23.4 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        


# CanProceed

A small, tested, no-frills parser of `robots.txt` files in Swift.

Currently used in

- [Bloo](https://github.com/ptsochantaris/bloo)

Full docs [can be found here](https://swiftpackageindex.com/ptsochantaris/can-proceed/1.0.0/documentation/canproceed)

## Quick example

```

let exampleRobotsFileLocation = URL(string: "https://www.bbc.co.uk/robots.txt")!

let robotsData = try! Data(contentsOf: exampleRobotsFileLocation)

let robotsText = String(data: robotsData, encoding: .utf8)!

let check = CanProceed.parse(robotsText)

let shouldProceed = check.agent("ChatGPT-User", canProceedTo: "/news"))

guard shouldProceed else {

    complain()

    return

}

...

check.sitemaps   // contains any sitemaps

check.crawlDelay // provides any crawl delay setting

check.sitemps    // contains any XML sitemaps

```

## License

Copyright (c) 2024 Paul Tsochantaris. Licensed under the MIT License, see LICENSE for details.