Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/WardCunningham/seran-scrape

Federation scraper coded as a denowiki meta-site with wiki plugin start/stop/step controls
https://github.com/WardCunningham/seran-scrape

Last synced: 4 days ago
JSON representation

Federation scraper coded as a denowiki meta-site with wiki plugin start/stop/step controls

Host: GitHub
URL: https://github.com/WardCunningham/seran-scrape
Owner: WardCunningham
Created: 2020-03-18T13:17:15.000Z (over 4 years ago)
Default Branch: master
Last Pushed: 2020-07-02T20:51:30.000Z (over 4 years ago)
Last Synced: 2024-08-02T12:51:55.202Z (3 months ago)
Language: TypeScript
Size: 50.8 KB
Stars: 4
Watchers: 4
Forks: 2
Open Issues: 1
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

awesome-starred - WardCunningham/seran-scrape - Federation scraper coded as a denowiki meta-site with wiki plugin start/stop/step controls (others)

README

# seran-scrape
Federation scraper coded as a seran-wiki meta-site with wiki plugin start/stop/step controls

![Scraper Lineup](https://user-images.githubusercontent.com/12127/77013634-aa4f7400-692d-11ea-90db-b384b1ab9ed7.png)

Let's explain that description word for word from left to right:
- The federation consists of wiki sites that have pages with some history on other sites.
- We read wiki sitemaps to discover new pages and then read them to discover new sites.
- We've coded the federation traversal algorithm in TypeScript using site and page loops separated by queues.
- We run our code in the deno runtime where we have new implementations of wiki client and server.
- Seran-wiki can load modules (meta-sites) that serve any number of wiki pages, often algorithmetically produced.
- This meta-site runs a scraper and offers pages for starting or stopping the site and page loops.
- The start/stop/step controls are provided by a plugin with configurable client and server parts.

Related documentation.
- [Stepping the Async Scrape](http://ward.asia.wiki.org/stepping-the-async-scrape.html)
- [Federation Friendly Outposts](http://ward.asia.wiki.org/federation-friendly-outposts.html)