https://github.com/oslokommune/okdata-pipeline
Collection of pipeline components for Origo Dataplattform
https://github.com/oslokommune/okdata-pipeline
dataplatform
Last synced: 28 days ago
JSON representation
Collection of pipeline components for Origo Dataplattform
- Host: GitHub
- URL: https://github.com/oslokommune/okdata-pipeline
- Owner: oslokommune
- Created: 2020-11-18T13:38:48.000Z (about 5 years ago)
- Default Branch: main
- Last Pushed: 2025-10-29T08:30:31.000Z (4 months ago)
- Last Synced: 2025-10-29T10:30:06.440Z (4 months ago)
- Topics: dataplatform
- Language: Python
- Homepage:
- Size: 795 KB
- Stars: 0
- Watchers: 5
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# okdata-pipeline
Collection of pipeline components for [Origo
Dataplattform](https://oslokommune.github.io/dataplattform/).
## Components
- converters
- [csv](doc/converters/csv.md)
- [json](doc/converters/json.md)
- [xls](doc/converters/xls.md)
- [lambda_invoker](doc/lambda_invoker.md)
- validators
- [csv](doc/validators/csv.md)
- [json](doc/validators/json.md)
- writers
- [s3](doc/writers/s3.md)
## Setup
```sh
make init
```
## Test
Tests are run using [tox](https://pypi.org/project/tox/):
```sh
make test
```
For tests and linting we use [pytest](https://pypi.org/project/pytest/),
[flake8](https://pypi.org/project/flake8/), and
[black](https://pypi.org/project/black/).
## Deploy
Example GitHub Actions for deploying to dev and prod on push to `main` is
included in `.github/workflows`.
You can also deploy from a local machine to dev with:
```sh
make deploy
```
Or to prod with:
```sh
make deploy-prod
```