https://github.com/a-poor/datatransform.jl
A package for defining (and performing) tabular-data transformations with JSON.
https://github.com/a-poor/datatransform.jl
data data-science data-transformation etl feature-engineering json julia julia-package tabular-data
Last synced: about 2 months ago
JSON representation
A package for defining (and performing) tabular-data transformations with JSON.
- Host: GitHub
- URL: https://github.com/a-poor/datatransform.jl
- Owner: a-poor
- License: mit
- Created: 2021-05-06T17:10:11.000Z (about 5 years ago)
- Default Branch: main
- Last Pushed: 2021-05-07T00:15:11.000Z (about 5 years ago)
- Last Synced: 2025-03-26T06:35:57.648Z (over 1 year ago)
- Topics: data, data-science, data-transformation, etl, feature-engineering, json, julia, julia-package, tabular-data
- Language: Julia
- Homepage:
- Size: 8.79 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# DataTransform.jl
_created by Austin Poor_
__Under Construction__
This library is intended to help develop a standardized data-transformation schema for tabular data.
Data transformation files are defined in JSON so they are human-readable, trackable in VC, and so they can be passed between languages. `DataTransform`'s goal is to promote reproducablity and clean transformations (rather than hack-y fixes, whenever possible).
You pass in a rectangular dataset and define a stack of transformations to apply -- `DataTransform.jl` does the rest!
## Quick Start
...
## Installation
...
## DataTransform Schema Reference
Data transform files are defined in JSON.
* Select Column
* by Name
* Exact Match
* Starts With
* Ends With
* Contains
* RegEx
* by Index
* by Data Type
* Filter Rows
* by Value
* Exact
* In List
* Range
* RegEx (for Text)
* Apply Function to Column
* Arange Columns
* Sort Names
* In Specified Orders
* Arange Rows
* Sort by Column (or Column Combinations)
* Apply Function by Column (or Row?)
* Mapping
* Convert Type
* Case Function (`if/else-if/else` or `case-when`)
* Predefined function from `FeatureEng`
* Duplicate Column
* (Using Column Select Operations)
* Duplicate Rows
* (Using Row Select Operations)
* Rename Column
* Mapping
* RegEx
* ~~Join with Other Dataset?~~
The basic schema structure is:
```json
{
"version": "0.1", // Schema Version Number for Compatability
"transformations": [...] // List of Transformation Objects
}
```
Deciding how to handle errors. Set error handling globally.
Example file: [sample-transform.json](./sample-transform.json).
## API Reference
...
## Contributing
Any contribution would be greatly appreciated! You can help by suggestint additions to the package or changes you think would help. In addition, please feel free to submit an issue or PR!
Thanks!
Austin