https://github.com/khulnasoft/datalineup
https://github.com/khulnasoft/datalineup
Last synced: 7 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/khulnasoft/datalineup
- Owner: khulnasoft
- License: mit
- Created: 2024-10-26T23:41:00.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2024-10-27T01:17:40.000Z (12 months ago)
- Last Synced: 2025-02-01T02:16:15.730Z (8 months ago)
- Language: Python
- Size: 544 KB
- Stars: 0
- Watchers: 3
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# datalineup
Datalineup is a job scheduling and data processing system developed for web crawling needs at Khulnasoft Systems.
## Documentation
- [**Architecture Overview**](docs/architecture_overview.md)
## Installation
- [PyPI](https://pypi.org/project/datalineup-engine/): `datalineup-engine`.
- [Docker Hub](https://hub.docker.com/repository/docker/khulnasoft/datalineup): `khulnasoft/datalineup`.## Development
Install [nox](https://nox.thea.codes/en/stable/) and [poetry](https://python-poetry.org/docs/).
- To run all tests: `nox`
- To format code: `nox -rs format`You can also work from the shell with:
```console
$ # Install the project locally.
$ poetry install --all-extras
$ poetry shell
$ # Run the utilities.
$ py.test tests -xsvv
$ mypy src tests
```