https://github.com/iboard/data_source

Elixir Datasource and DataStage
https://github.com/iboard/data_source

consumer datasource elixir elixir-lang hex

Last synced: 2 months ago
JSON representation

Elixir Datasource and DataStage

• Host: GitHub
• URL: https://github.com/iboard/data_source
• Owner: iboard
• License: mit
• Created: 2018-09-02T19:10:55.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2019-03-02T20:30:37.000Z (almost 6 years ago)
• Last Synced: 2024-04-25T21:05:52.807Z (8 months ago)
• Topics: consumer, datasource, elixir, elixir-lang, hex
• Language: Elixir
• Homepage: https://hex.pm/packages/data_source
• Size: 45.9 KB
• Stars: 2
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • License: LICENSE.md

Awesome Lists containing this project

README

        
# DataSource

[![Documentation](https://img.shields.io/badge/docs-hexpm-blue.svg)](http://hexdocs.pm/data_source/)

[![Documentation](https://travis-ci.com/iboard/data_source.svg?branch=master)](https://travis-ci.com/iboard/data_source)

Part of the `[PocketData][]` project. A DataSource is a "Producer" (of data)

for the system. It can be a simple counter, or a complex "collector", reading

sensors from embedded systems or collecting data from foreign (web-)services.

## Installation

The package is [available in Hex](https://hex.pm/packages/data_source), the package can be installed

by adding `data_source` to your list of dependencies in `mix.exs`:

      def deps do

        [

          {:data_source, "~> 0.1"}

        ]

      end

## Example

The best way to figure out how you can use this library is by having a look at 

this [Test suite](https://github.com/iboard/hexpack-examples/blob/master/test/hexpack_examples_test.exs).

      # Remember consumed events in state

      defmodule ConsumerSpy do

        use GenStage

        def start_link(), do: GenStage.start_link(ConsumerSpy, [])

        def init(state), do: {:consumer, state}

        def handle_events(events, _from, state) do

          # Simulate load

          Process.sleep(10)

          {:noreply, [], [events | state]}

        end

        def handle_call(:get, _from, state) do

          {:reply, Enum.flat_map(state, & &1), [], state}

        end

      end

      {:ok, datasource} = Datasource.start_link(0, fn state -> {state, state + 1} end)

      {:ok, producer} = Datasource.DataStage.start_link(datasource)

      {:ok, consumer} = ConsumerSpy.start_link()

      GenStage.sync_subscribe(consumer, to: producer, max_demand: 1)

      Process.sleep(100)

      ConsumerSpy.call(consumer, :get)

      # => [0,1,2,3,...10]

Because the consumer delays for 10ms and we have 1 consumer only,

in 100ms we can expect about 10 events. To process more than 10

events you can increase the number of consumers.

## Datasources

Some Datasources are defined in `lib/data_source`. Such as

`Datasource.Counter`, `Datasource.File`, and more.

[PocketData]: https://github.com/iboard/pocketdata