https://github.com/ulixee/platform
Home of the Ulixee Open Data Platform
https://github.com/ulixee/platform
Last synced: about 1 year ago
JSON representation
Home of the Ulixee Open Data Platform
- Host: GitHub
- URL: https://github.com/ulixee/platform
- Owner: ulixee
- License: mit
- Created: 2021-07-15T22:12:19.000Z (almost 5 years ago)
- Default Branch: main
- Last Pushed: 2024-12-12T21:55:20.000Z (over 1 year ago)
- Last Synced: 2025-04-03T00:01:38.541Z (about 1 year ago)
- Language: TypeScript
- Homepage: https://ulixee.org/
- Size: 33.2 MB
- Stars: 50
- Watchers: 3
- Forks: 10
- Open Issues: 9
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Funding: .github/FUNDING.yml
- License: LICENSE.md
Awesome Lists containing this project
README
# Ulixee
Ulixee is a scraping [engine][hero] with a built-in deployment [unit][datastore] that enables out-of-the-box [querying][stream] across a horizontal [deployment][cloud].
This repository is the development home to several of the tools that make it easy to build and manage these scripts, including [Ulixee Desktop](desktop), [Cloud][cloud] and [Datastore][datastore].
## Projects
- Hero `/hero`. The Automated Browser Engine built for scraping. (repository home - https://github.com/ulixee/hero).
- Datastore `/datastore`. Packaged "database" containing API access to crawler functions and extractor functions.
- Cloud `/cloud`. Run Ulixee tooling on a remote machine.
- Stream `/stream`. Query, transform and compose Datastores running on any machine.
- Desktop `/desktop`. Supercharge scraper script development using a Hero Replay toolset, remote Datastore viewer and Error troubleshooter (repository home - https://github.com/ulixee/desktop).
## Tooling
Try out [Ulixee Desktop!](https://github.com/ulixee/desktop). It's a helpful tool for developing and managing your Ulixee scripts.
### Docker
We publish a Docker image of the latest Ulixee Cloud to:
- Github Container Registry: `docker pull ghcr.io/ulixee/ulixee-cloud && docker tag ghcr.io/ulixee/ulixee-cloud ulixee/ulixe-cloud`
- DockerHub: `docker pull ulixee/ulixee-cloud`
To use the image, we have a [run.sh](./cloud/tools/docker/run.sh) script that will run with a non-root user on your choice of port. All environmental configurations are listed [here](./cloud/main/.env.defaults).
## Developer Environment
This project serves as a Monorepo for developing the Ulixee Datastore and Cloud. If you are developing, you might wish to have [hero][hero] as a project adjacent to this one.
1Run `yarn build:all` from the this repository to build all the projects.
Learn more about Ulixee at [ulixee.org](https://ulixee.org).
## Contributing
See [How to Contribute](https://ulixee.org/how-to-contribute) for ways to get started.
This project has a [Code of Conduct](https://ulixee.org/code-of-conduct). By interacting with this repository, organization, or community you agree to abide by its terms.
We'd love your help in making Ulixee a better set of tools. Please don't hesitate to send a pull request.
## License
[MIT](LICENSE.md)
[hero]: https://github.com/ulixee/hero
[datastore]: datastore
[stream]: ./
[cloud]: cloud