https://github.com/allegro/ecto-cursor-based-stream
Elixir library that allows for cursor-based streaming of Ecto records, that does not require database transaction.
https://github.com/allegro/ecto-cursor-based-stream
ecto elixir elixir-lang hex
Last synced: 3 months ago
JSON representation
Elixir library that allows for cursor-based streaming of Ecto records, that does not require database transaction.
- Host: GitHub
- URL: https://github.com/allegro/ecto-cursor-based-stream
- Owner: allegro
- License: apache-2.0
- Created: 2022-11-14T18:34:46.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-08-14T08:26:02.000Z (over 1 year ago)
- Last Synced: 2025-02-07T02:11:12.761Z (12 months ago)
- Topics: ecto, elixir, elixir-lang, hex
- Language: Elixir
- Homepage: https://hex.pm/packages/ecto_cursor_based_stream
- Size: 45.9 KB
- Stars: 88
- Watchers: 5
- Forks: 4
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# EctoCursorBasedStream
 [](https://hex.pm/packages/ecto_cursor_based_stream) [](https://hexdocs.pm/ecto_cursor_based_stream/)
Cursor-based streaming of Ecto records, that does not require database transaction.
Gives you a [`cursor_based_stream/2`](https://hexdocs.pm/ecto_cursor_based_stream/EctoCursorBasedStream.html#c:cursor_based_stream/2) function that mimics [`Ecto.Repo.stream/2`](https://hexdocs.pm/ecto/Ecto.Repo.html#c:stream/2) interface.
Advantages in comparison to the standard `Ecto.Repo.stream/2`:
- streaming can be stopped and continued at any point (by passing option `after_cursor: ...`),
- works with tables that have milions of records.
Only limitation is that you have to supply a _cursor column or columns_ (by passing option `cursor_field: ...`, defaults to `:id`). Such a column(s):
- must have unique values,
- should have a database index. (So that sorting by it, and returning a number of rows larger than `x` is a performant operation.)
## Usage
1. Add `ecto_cursor_based_stream` to your list of dependencies in `mix.exs`:
```elixir
def deps do
[
{:ecto_cursor_based_stream, "~> 1.2.0"}
]
end
```
2. Add `use EctoCursorBasedStream` to the module that uses `Ecto.Repo`:
```elixir
defmodule MyRepo do
use Ecto.Repo
use EctoCursorBasedStream
end
```
3. Stream the rows using `cursor_based_stream/2`:
```elixir
Post
|> MyRepo.cursor_based_stream()
|> Stream.each(...)
|> Stream.run()
```
## Useful links
- [Documentation](https://hexdocs.pm/ecto_cursor_based_stream/EctoCursorBasedStream.html)
- [Examples](https://github.com/allegro/ecto-cursor-based-stream/blob/main/test/ecto_cursor_based_stream_test.exs)
## Contributing
### Running tests
Run the following after cloning the repo:
```sh
mix deps.get
docker-compose up -d
mix test
```