https://github.com/namusyaka/botch

A simple DSL for web crawler.
https://github.com/namusyaka/botch

Last synced: about 1 month ago
JSON representation

A simple DSL for web crawler.

• Host: GitHub
• URL: https://github.com/namusyaka/botch
• Owner: namusyaka
• Created: 2013-07-07T15:05:48.000Z (almost 12 years ago)
• Default Branch: master
• Last Pushed: 2013-08-08T23:54:42.000Z (almost 12 years ago)
• Last Synced: 2024-04-26T01:05:23.891Z (about 1 year ago)
• Language: Ruby
• Homepage:
• Size: 242 KB
• Stars: 4
• Watchers: 3
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Botch

[![Build Status](https://travis-ci.org/namusyaka/botch.png)](https://travis-ci.org/namusyaka/botch)

Botch is a simple DSL for quickly creating web crawlers.

Inspired by Sinatra.

[Japanese](https://gist.github.com/namusyaka/6001467)

## Installation

add this line to your Gemfile.

`gem 'botch'`

or

`$ gem install botch`

## Usage

```ruby

require 'lib/botch'

require 'kconv'

require 'rack'

class SampleBotch < Botch::Base

  set :user_agent, "SampleBotch"

  helpers do

    def h(str)

      Rack::Utils.escape_html(str)

    end

  end

  filter :example, :map => "example.com" do

    status == 200

  end

  rule :example, :map => /example\.com/ do

    h(body.toutf8)

  end

end

if $0 == __FILE__

  SampleBotch.run("http://example.com/") do |response|

    puts response

  end

end

```

## TODO

- RSpec

- Documentation

## Contributing to Botch

1. fork the project.

2. create your feature branch. (`git checkout -b my-feature`)

3. commit your changes. (`git commit -am 'commit message.'`)

4. push to the branch. (`git push origin my-feature`)

5. send pull request.

## License

MIT