https://github.com/raynardj/gallop
🐎 New level of python develop
https://github.com/raynardj/gallop
Last synced: 9 months ago
JSON representation
🐎 New level of python develop
- Host: GitHub
- URL: https://github.com/raynardj/gallop
- Owner: raynardj
- License: mit
- Created: 2022-08-30T07:20:15.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2022-09-09T13:34:50.000Z (almost 4 years ago)
- Last Synced: 2025-05-30T00:19:24.655Z (about 1 year ago)
- Language: Python
- Size: 36.1 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# gallop
🐎 New level of python develop. Source code from [github](https://github.com/raynardj/gallop)
> More dynamic in design, more configuration driven
[](https://pypi.org/project/gallop/)
[](https://github.com/raynardj/gallop/actions/workflows/test.yml)
[](https://github.com/raynardj/gallop/actions/workflows/publish.yml)
```
[-;-/=_
`-; \=_ ___ _________ __ __ ____ ____
) ,"-...--./===--__ / ____/ | / / / / / __ \/ __ \
__|/ / ]` / / __/ /| | / / / / / / / / /_/ /
/;--> >...-\ <\_ / /_/ / ___ |/ /___/ /___/ /_/ / ____/
`- <<, 7/-.\, \____/_/ |_/_____/_____/\____/_/
`- /( `-
```
## Install
```shell
pip install gallop
```
## Configuration Management
```python
from gallop.config import BaseConfig
config = BaseConfig(a=1)
config.a = 2
config.b = 3
config.to_json("some/path.json")
config.to_yaml("some/path.yaml")
```
## Run python task from config
You can turn any callable execution in to configuration, eg save the following
```yaml
func_name: use:logging.warning
args:
- "hello world"
```
to `sometask.yaml` and run `gallop sometask`
Is the same to run the python script
```python
import logging
logging.warning("hello world")
```
### `param:` configuration
We can change the value on the run, while pointing to the position in the config, using a chain of keys (keys to dict or list).
eg. to change the key `args`, of its 1st element, we can run
```shell
gallop sometask --param:args.0 changed_world
```
## Advanced usage
### Some simple grammar
* `checkin: somekey`, save the result to a centralized dictionary with key `somekey`
* `checkout: somekey`, use the result from the centralized dictionary with key `somekey`
* `checkout: env:DATA_HOME`, use the result from the environment variable `DATA_HOME`
* `use:some.module`, use or import the module `some.module` as the callable, eg
* `func_name: use:os.path.join`, use the function `os.path.join`
* `func_name: use:pandas.DataFrame`, use the class `pandas.DataFrame`
### Examples
#### Inference BERT model
> If you have `transformers` installed, you can run the following example directly to featurize a sentence
```yaml
pred_task:
- func_name: use:transformers.AutoModel.from_pretrained
args:
- bert-base-uncased
checkin: model
description: |
Load pretrained model
- func_name: use:transformers.AutoTokenizer.from_pretrained
args:
- bert-base-uncased
checkin: tokenizer
description: |
Load pretrained tokenizer
- func_name: tokenizer
args:
- - "Hello, the capital of [MASK] is Paris"
kwargs:
return_tensors: pt
max_length: 128
truncation: True
padding: True
checkin: inputs
- func_name: use:logging.warning
args:
- func_name: model
kwargs:
input_ids:
checkout: inputs.input_ids
attention_mask:
checkout: inputs.attention_mask
checkin: features
```
This equals to the following `python` script
```python
import logging
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("bert-base-uncased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
inputs = tokenizer(
"Hello, the capital of [MASK] is Paris",
return_tensors="pt",
max_length=128,
truncation=True,
padding=True,
)
logging.warning(model(
input_ids=inputs.input_ids,
attention_mask=inputs.attention_mask,
))
```
Save the yaml to `run_bert.yaml` and you can use `gallop run_bert --output features` to run the task.
Run in commandline with changed value, and printout one of the checkout value
```shell
gallop run_bert --loglevel debug --output features
```
And run it with `CHANGED VALUE`
```
gallop run_bert 、
--param:pred_task.2.args.0.0 "The [MASK] house is where the POTUS live and work"
```
## Related project
> From the same lead author
* Python [category](https://github.com/raynardj/category) management accelerated with Rust
* Data science basic toolset [forgebox](https://github.com/raynardj/forgebox)