Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/waiting-for-dev/follow

Haskell library to follow content published on any subject.
https://github.com/waiting-for-dev/follow

Last synced: 27 days ago
JSON representation

Haskell library to follow content published on any subject.

Awesome Lists containing this project

README 

        
# Follow



`Follow` is a Haskell library to build recipes which allow you to

follow the content published about any subject you are interested.



Here bellow you have a quick tutorial you can follow.  Just run the

snippets of code in the repl.



```haskell

:set -XOverloadedStrings

import Follow

import Data.Time (LocalTime)

import qualified Data.Text.IO as T (writeFile)

import Data.Yaml (decodeFileThrough)

```



## Subject



A subject is just a bunch of information about what is being

followed. It consists of a title, a description and a list of tags,



```haskell

haskell =

  Subject

    { sTitle = "Haskell"

    , sDescription = "Some resources about Haskell"

    , sTags = ["haskell", "programming"]

    }

```



## Directory



A directory is just a subject and a list of entries.



An entry is an item meant to contain an URI with content relative to

the associated subject along with associated information.



```haskell

manualDirectory =

  Directory

    { dSubject = haskell

    , dEntries =

        [ Entry

            { eURI =

                Just "https://bartoszmilewski.com/2013/06/19/basics-of-haskell/"

            , eGUID = Just "basics-of-haskell"

            , eTitle = Just "Basics of Haskell"

            , eDescription = Just "Introductory material for Haskell"

            , eAuthor = Just "Bartosz Milewski"

            , ePublishDate = Just (read "2013-06-19 14:14:00" :: LocalTime)

            }

        ]

    }

```



## Fetchers



Of course, building list of entries by hand is not very

useful. Fetchers are functions which usually reach the outside world

to return a list of entries and which can throw an error.



Any fetcher can be used, but `Follow` tries to ship with common

ones. Right now there are two fetchers available:



- [Feed](src/Follow/Fetchers/Feed.hs): Take entries from a RSS or Atom feed.

- [Web Scraping](src/Follow/Fetchers/WebScraping.hs): Take entries

  scraping the HTML of a web page.



The function `directoryFromFetched` can be used to glue a subject with

some fetched content:



```haskell

import qualified Follow.Fetchers.Feed as Feed



directory =

  directoryFromFetched (Feed.fetch "https://bartoszmilewski.com/feed/") haskell

```



## Middlewares



Fetched content may need some further processing in order to fit what

is actually desired. A middleware is a function which transforms a

directory into another directory, allowing us to do any kind of

transformation.



The aim of `Follow` is to provide some common middlewares. For now,

there are these ones:



- [Filter](src/Follow/Middlewares/Filter.hs): Filter entries according

  some predicate.

- [Sort](src/Follow/Middlewares/Sort.hs): Sort entries.

- [Decode](src/Follow/Middlewares/Decode.hs): Decodes entries from

  UTF8 or other encodings.



```haskell

import qualified Follow.Middlewares.Sort as Sort



sortedDirectory =

  Sort.apply (Sort.byGetter eTitle) <$> directory

```



## Digesters



Once you have your distillate content, you need some way to consume

it. A `Digester` is a function which transforms a directory into

anything that can be consumed by an end user.



As before, `Follow` aims to provide useful ones out of the box. Right

now the following are available:



- [Simple Text](src/Follow/Digesters/SimpleText.hs): Simple textual

  representation of the directory.

- [Pocket](src/Follow/Digesters/Pocket.hs): Send fetched links to

  [Pocket](https://getpocket.com).



```haskell

import qualified Follow.Digesters.SimpleText as SimpleText



content = SimpleText.digest <$> sortedDirectory

```



Now, for example, you are ready to save the content to a file:



```haskell

content >>= T.writeFile "/your/path/haskell.txt"

```



## Recipes: Combining sources and middlewares



Content is not limited to be fetched from a single source. Instead, a

directory can be built merging the entries fetched from different

sources. Also, the stack of middlewares to be applied to each source can be

given in a single shot.



This whole process specification is called a `Recipe`, and it contains

all the information needed to follow a subject.



To build the recipe you need to provide three fields:



- The subject being followed.

- A list of two field tuples where:

  - First field is some fetched content.

  - Second field is a list of middlewares to apply to the fetched content in the first field.

- A list of middlewares to apply to the directory resulted after applying the list of fetched/middlewares.



```haskell

haskellRecipe =

  Recipe

    { rSubject = haskell

    , rSteps =

        [ ( Feed.fetch "https://bartoszmilewski.com/feed/"

          , [Sort.apply (Sort.byGetter eTitle)])

        , (Feed.fetch "https://planet.haskell.org/rss20.xml", [])

        ]

    , rMiddlewares = []

    }

```



You can combine the function `directoryFromRecipe` and some digester

to quickly consume a recipe:



```haskell

SimpleText.digest <$> directoryFromRecipe haskellRecipe

```



## Collecting recipes



One nice thing in Follow is that you don't need to create the recipes

programmatically each time you need them. Instead, you can store them

in a [YAML](https://en.wikipedia.org/wiki/YAML) file and just parse

them when you need.



For example, the previous recipe can be represented in a file

`recipe.yml` as the following:



```yaml

subject:

  title: Haskell

  description: Some resources about Haskell

  tags: [haskell, programming]

steps:

  -

    - type: feed

      options:

        url: "https://bartoszmilewski.com/feed/"

    -

      - type: sort

        options:

          function:

            type: by_field

            options:

              field: title

middlewares: []

```



You can use now decode functions in

[`Data.Yaml`](https://hackage.haskell.org/package/yaml) to get the

recipe back:



```haskell

recipe' <- decodeFileThrow "/your/path/recipe.yml" :: IO (Recipe IO)

directory' = directoryFromRecipe recipe'

```



Look at [`src/Follow/Parser.hs`](src/Follow/Parser.hs) for details about

encoding each kind of fetcher and middleware.



## Contributing



Bug reports and pull requests are welcome on GitHub at

https://github.com/waiting-for-dev/follow. This project is

intended to be a safe, welcoming space for collaboration, and

contributors are expected to adhere to the [Contributor

Covenant](http://contributor-covenant.org) code of conduct.



## License



The package is available as open source under the terms of the [GNU

LGPLv3 License](http://opensource.org/licenses/LGPL-3.0).