https://github.com/buren/stupid_crawler

Stupid crawler that looks for URLs on a given site
https://github.com/buren/stupid_crawler

cli crawler ruby rubygem

Last synced: 25 days ago
JSON representation

Stupid crawler that looks for URLs on a given site

• Host: GitHub
• URL: https://github.com/buren/stupid_crawler
• Owner: buren
• License: mit
• Created: 2016-08-24T16:00:05.000Z (about 8 years ago)
• Default Branch: master
• Last Pushed: 2021-11-29T16:19:57.000Z (almost 3 years ago)
• Last Synced: 2024-09-21T21:51:39.854Z (about 2 months ago)
• Topics: cli, crawler, ruby, rubygem
• Language: Ruby
• Size: 12.7 KB
• Stars: 1
• Watchers: 3
• Forks: 1
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.txt

Awesome Lists containing this project

README

        
# StupidCrawler

Stupid crawler that looks for URLs on a given site. Result is saved as two CSV files one with found URLs and another with failed URLs.

## Installation

Add this line to your application's Gemfile:

```ruby

gem 'stupid_crawler'

```

And then execute:

    $ bundle

Or install it yourself as:

    $ gem install stupid_crawler

## Usage

```

stupid-crawler --help

```

## Development

After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).

## Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/buren/stupid_crawler.

## License

The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).