https://github.com/cognitedata/python-extractor-utils
Framework for developing extractors in Python
https://github.com/cognitedata/python-extractor-utils
cognite-data-fusion cognite-extractor data-integration python
Last synced: 6 months ago
JSON representation
Framework for developing extractors in Python
- Host: GitHub
- URL: https://github.com/cognitedata/python-extractor-utils
- Owner: cognitedata
- License: apache-2.0
- Created: 2019-06-27T10:26:09.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2024-05-22T11:37:06.000Z (over 1 year ago)
- Last Synced: 2024-05-22T12:05:53.062Z (over 1 year ago)
- Topics: cognite-data-fusion, cognite-extractor, data-integration, python
- Language: Python
- Homepage: https://cognite-extractor-utils.readthedocs-hosted.com/en/latest/index.html
- Size: 939 KB
- Stars: 12
- Watchers: 65
- Forks: 5
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
Cognite Python `extractor-utils`
================================
[](https://github.com/cognitedata/python-extractor-utils/actions)
[](https://cognite-extractor-utils.readthedocs-hosted.com/en/latest/?badge=latest)
[](https://codecov.io/gh/cognitedata/python-extractor-utils)
[](https://pypi.org/project/cognite-extractor-utils)
[](LICENSE)
[](https://github.com/ambv/black)
The `extractor-utils` package is an extension of the Cognite Python SDK intended to simplify the development of data
extractors or other integrations for Cognite Data Fusion.
Documentation is hosted [here](https://cognite-extractor-utils.readthedocs-hosted.com/en/latest/), including a
[quickstart tutorial](https://cognite-extractor-utils.readthedocs-hosted.com/en/latest/quickstart.html).
The changelog is found [here](./CHANGELOG.md).
## Overview
The best way to start a new extractor project is to use the `cogex` CLI. You can install that from PyPI:
``` bash
pip install cognite-extractor-manager
```
To initialize a new extractor project, run
``` bash
cogex init
```
in the directory you want your extractor project in. The `cogex` CLI will prompt you for some information about your
project, and then set up a poetry environment, git repository, commit hooks with type and style checks and load a
template.
## Contributing
The package is open source under the [Apache 2.0 license](./LICENSE), and contribtuions are welcome.
This project adheres to the [Contributor Covenant v2.0](https://www.contributor-covenant.org/version/2/0/code_of_conduct/)
as a code of conduct.
### Development environment
We use [uv](https://docs.astral.sh/uv/) to manage dependencies and to administrate virtual environments. To develop
`extractor-utils`, follow the following steps to set up your local environment:
1. [Install uv](https://docs.astral.sh/uv/getting-started/installation/) if you haven't already.
2. Clone repository:
```
$ git clone git@github.com:cognitedata/python-extractor-utils.git
```
3. Move into the newly created local repository:
```
$ cd python-extractor-utils
```
4. Create virtual environment and install dependencies:
```
$ uv sync
```
### Code requirements
All code must pass [black](https://github.com/ambv/black) and [isort](https://github.com/timothycrosley/isort) style
checks to be merged. It is recommended to install pre-commit hooks to ensure this locally before commiting code:
```
$ poetry run pre-commit install
```
Each public method, class and module should have docstrings. Docstrings are written in the [Google
style](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings). Please include unit and/or
integration tests for submitted code, and remember to update the [changelog](./CHANGELOG.md).