Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/stattleship/batcher

Last synced: 3 months ago
JSON representation

Host: GitHub
URL: https://github.com/stattleship/batcher
Owner: stattleship
License: mit
Created: 2011-12-16T19:08:51.000Z (almost 13 years ago)
Default Branch: master
Last Pushed: 2011-12-18T02:30:48.000Z (almost 13 years ago)
Last Synced: 2024-03-26T05:41:54.019Z (8 months ago)
Language: Ruby
Homepage:
Size: 95.7 KB
Stars: 45
Watchers: 2
Forks: 1
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

Batcher
======

Batcher is a tiny gem that allows you to batch process data and slice it by whatever you require.

It was extracted from [Stattleship](http://stattleship.com).

Why
---

Processing ActiveRecord data is impossible with large amounts of data because the memory footprint of tons of ActiveRecord objects is too large and unmanagable. ActiveRecord models provide a `find_each` method for batch processing of data; unfortunately it only allows you to slice the data by primary key.

Example
-------

Batcher accepts an object that responds to `count`, `limit` and `offset` (a subset of the [Active Record Query Interface](http://guides.rubyonrails.org/active_record_querying.html)).

Typically, you pass it an instance of `ActiveRecord::Relation`.

```ruby
class User < ActiveRecord::Base
end

Batcher(User.order('influence desc')).each do |user|
user.do_something
end
```

The default batch size is 1000, but you can pass in a different batch size to tweak accordingly:

```ruby
Batcher(User.order('influence desc'), batch_size: 2000).each do |user|
user.do_something
end
```

Note that batcher uses [Enumerator](http://www.ruby-doc.org/core-1.9.2/Enumerator.html). Therefore it is only supported on Ruby 1.9+.

License
-------