https://github.com/simonsdave/cloudfeaster

Cloudfeaster Spider Development
https://github.com/simonsdave/cloudfeaster

docker python selenium-webdriver spider webcrawler

Last synced: 3 months ago
JSON representation

Cloudfeaster Spider Development

• Host: GitHub
• URL: https://github.com/simonsdave/cloudfeaster
• Owner: simonsdave
• License: mit
• Created: 2013-10-13T14:50:36.000Z (over 12 years ago)
• Default Branch: master
• Last Pushed: 2024-10-14T16:51:24.000Z (over 1 year ago)
• Last Synced: 2025-04-23T12:16:00.355Z (about 1 year ago)
• Topics: docker, python, selenium-webdriver, spider, webcrawler
• Language: Python
• Homepage: https://www.cloudfeaster.com/
• Size: 3.72 MB
• Stars: 3
• Watchers: 1
• Forks: 0
• Open Issues: 21
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • Contributing: docs/contributing.md
  • License: LICENSE.md

Awesome Lists containing this project

README

          
# cloudfeaster

![Maintained](https://img.shields.io/maintenance/yes/2022.svg?style=flat)

![license](https://img.shields.io/pypi/l/cloudfeaster.svg?style=flat)

![PythonVersions](https://img.shields.io/pypi/pyversions/cloudfeaster.svg?style=flat)

![status](https://img.shields.io/pypi/status/cloudfeaster.svg?style=flat)

[![PyPI](https://img.shields.io/pypi/v/cloudfeaster.svg?style=flat)](https://pypi.python.org/pypi/cloudfeaster)

[![CodeFactor](https://www.codefactor.io/repository/github/simonsdave/cloudfeaster/badge/master)](https://www.codefactor.io/repository/github/simonsdave/cloudfeaster/overview/master)

[![CircleCI](https://circleci.com/gh/simonsdave/cloudfeaster/tree/master.svg?style=shield)](https://circleci.com/gh/simonsdave/cloudfeaster/tree/master)

[![codecov](https://codecov.io/gh/simonsdave/cloudfeaster/branch/master/graph/badge.svg)](https://codecov.io/gh/simonsdave/cloudfeaster)

[![docker-simonsdave/cloudfeaster-dev-env](https://img.shields.io/badge/docker-simonsdave%2Fcloudfeaster--dev--env-blue.svg)](https://hub.docker.com/r/simonsdave/cloudfeaster-dev-env)

[![docker-simonsdave/cloudfeaster](https://img.shields.io/badge/docker-simonsdave%2Fcloudfeaster-blue.svg)](https://hub.docker.com/r/simonsdave/cloudfeaster)

[![docker-simonsdave/cloudfeaster-lite](https://img.shields.io/badge/docker-simonsdave%2Fcloudfeaster--lite-blue.svg)](https://hub.docker.com/r/simonsdave/cloudfeaster-lite)

Cloudfeaster is a web crawling and business process automation infrastructure that leverages

modern software engineering tools, services and trends to create:

* (this repo is) [a spider authoring and maintenance environment](https://github.com/simonsdave/cloudfeaster)

* a service infrastructure for discovering and running spiders via a RESTful API

[This](docs/story.md) is the story behind Cloudfeaster.

## What Next

* [this](docs/spider_authors.md) describes

  how to author spiders using Cloudfeaster

* see [these](docs/contributing.md) instructions

  describe how to setup your development environment and

  start contributing to Cloudfeaster

* take a look at [cloudfeaster/samples](cloudfeaster/samples/) for some

  spiders that illustrate the core concepts

* and [simonsdave/gaming-spiders](https://github.com/simonsdave/gaming-spiders) is an

  entire repo of spiders so you can get a sense of the end-to-end