https://github.com/gusty/scrapem

A monadic web scraping library
https://github.com/gusty/scrapem

computation-expressions extract fsharp monad scraper scrapping

Last synced: 6 months ago
JSON representation

A monadic web scraping library

• Host: GitHub
• URL: https://github.com/gusty/scrapem
• Owner: gusty
• License: apache-2.0
• Created: 2016-12-07T21:06:06.000Z (almost 9 years ago)
• Default Branch: master
• Last Pushed: 2018-10-10T13:15:45.000Z (about 7 years ago)
• Last Synced: 2025-03-24T07:26:23.975Z (7 months ago)
• Topics: computation-expressions, extract, fsharp, monad, scraper, scrapping
• Language: F#
• Size: 88.9 KB
• Stars: 17
• Watchers: 4
• Forks: 2
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# ScrapeM

A monadic web scraping library

This library makes [web scraping](https://en.wikipedia.org/wiki/Web_scraping) easier by providing ways to automatically maintain state through different request, handling cookies, form submission and http headers.

*One function to scrap'em all*

This is essentially a single-function library which integrates many existing libraries and present several ways to approach web scraping by using different monads.

All other common functions used here come from different libraries like [FSharp.Data](http://fsharp.github.io/FSharp.Data/), [Http.fs](https://github.com/haf/Http.fs) and [F#+](https://github.com/gusty/FSharpPlus)

*Scrapes the web with category*

It's possible to create stateful linq-style queries which simulates basic user interaction with form submission by using different flavours of State monads. Also sequences expressions are available to integrate the data being extracted from multiple webpages in the same query.

## Getting started

Important: At the moment this library is in a 'Prototype' stage

Recommended: Visual Studio 2017 to avoid slow compile time of generic code

In order to try the examples run:

    > build.cmd // on windows    

    $ ./build.sh  // on unix

    

Now you can try the sample files:

* [Basic query with state handling](Sample-State-1.fsx) - Extracts a text from a website with login.

* [Basic query with multiple results](Sample-Seq-1.fsx) - Extracts many tables from a website, using a type provider.

* [Advanced query with state handling and multiple results](Sample-StateT-Seq-1.fsx) - Extracts many texts from a website by using different logins.