https://github.com/mgjules/harvit
Harvit harvests data from different sources (e.g websites, APIs), converts and transforms it.
https://github.com/mgjules/harvit
data-converter data-harvester data-transformer go golang
Last synced: 2 months ago
JSON representation
Harvit harvests data from different sources (e.g websites, APIs), converts and transforms it.
- Host: GitHub
- URL: https://github.com/mgjules/harvit
- Owner: mgjules
- License: apache-2.0
- Created: 2022-04-30T07:32:34.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-05-06T16:28:23.000Z (about 1 year ago)
- Last Synced: 2025-03-18T11:11:31.622Z (2 months ago)
- Topics: data-converter, data-harvester, data-transformer, go, golang
- Language: Go
- Homepage:
- Size: 183 KB
- Stars: 5
- Watchers: 1
- Forks: 0
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# harvit
[](https://godoc.org/github.com/mgjules/harvit)
[](https://conventionalcommits.org)
[](LICENSE)
Harvit harvests data from different sources (e.g websites, APIs), converts and transforms it.
## Contents
- [harvit](#harvit)
- [Contents](#contents)
- [Requirements](#requirements)
- [Usage](#usage)
- [Example](#example)
- [plan.yml](#planyml)
- [transformers/sample.js](#transformerssamplejs)
- [Result](#result)
- [License](#license)
- [Stability](#stability)
## Requirements
- [Go 1.18+](https://golang.org/doc/install)
- [Mage](https://github.com/magefile/mage) - replacement for Makefile in Go.
- [Golangci-lint](https://github.com/golangci/golangci-lint) - Fast Go linters runner.
- [Ginkgo](https://github.com/onsi/ginkgo) - Expressive testing framework.
- [Docker](https://www.docker.com) - Containerization.
## Usage
Harvit uses a `plan` in yaml format (see [example](#planyml)) to define the data source, fields and the transformer to be performed.
```shell
$ ./harvit harvest [command options] plan
```
```
NAME:
harvit harvest - Let's harvest some data!
USAGE:
harvit harvest [command options] plan
OPTIONS:
--debug whether running in PROD or DEBUG mode (default: false) [$HARVIT_DEBUG]
--help, -h show help (default: false)
```
## Example
```shell
$ ./harvit harvest | jq
```
### plan.yml
```yaml
source: https://mgjules.dev
type: website
fields:
- name: firstJobName
type: raw
selector: "#experience > div:nth-child(2) > ul > li:nth-child(1) > div.flex.flex-wrap.items-center.justify-between > h3"
- name: secondJobStartYear
type: datetime
selector: "#experience > div:nth-child(2) > ul > li:nth-child(2) > div.flex.flex-wrap.items-center.justify-between > span"
regex: \d{2}/(\d{4})\sā
format: Y
- name: secondJobEndDateTime
type: datetime
selector: "#experience > div:nth-child(2) > ul > li:nth-child(2) > div.flex.flex-wrap.items-center.justify-between > span"
regex: ā\s(?:[a-zA-Z]+|(\d{2}/\d{4}))
format: m/Y
timezone: Indian/Mauritius
- name: topLinks
type: text
selector: "body > div.relative.px-4.pt-4.sm\\:pt-16.print\\:pt-0.sm\\:px-6.lg\\:px-8 > div.max-w-4xl.mx-auto.text-lg > div:nth-child(2) > div.flex.flex-wrap.items-center.justify-center.gap-x-4.gap-y-2.print\\:hidden > a > div > span"
- name: experiencePlaces
type: text
selector: "#experience > div:nth-child(2) > ul > li > div.flex.flex-wrap.items-center.justify-between > h3"
- name: contributionsYears
type: datetime
selector: "#contributions > div:nth-child(2) > ul > li > div > span"
regex: (\d{4})
format: Y
- name: contributionsYearsNumbers
type: number
selector: "#contributions > div:nth-child(2) > ul > li > div > span"
regex: (\d{4})
- name: interestsTitle
type: text
selector: "#interests > div:nth-child(2) > ul > li > span"
transformer: transformers/sample.js
```
### transformers/sample.js
```js
data['interestsTitle'] = data['interestsTitle'].map(v => v === 'Space Exploration' ? 'SpaceX' : v);
```
### Result
```json
{
"contributionsYears": [
"2022-01-01T00:00:00+04:00",
"2021-01-01T00:00:00+04:00",
"2020-01-01T00:00:00+04:00",
"2020-01-01T00:00:00+04:00",
"2019-01-01T00:00:00+04:00",
"2019-01-01T00:00:00+04:00",
"2019-01-01T00:00:00+04:00",
"2019-01-01T00:00:00+04:00"
],
"contributionsYearsNumbers": [
2022,
2021,
2020,
2020,
2019,
2019,
2019,
2019
],
"experiencePlaces": [
"Ringier SA",
"Bocasay",
"La Sentinelle Digital Ltd",
"Expat-Blog Ltd",
"Noveo IT Ltd"
],
"firstJobName": "
Ringier SA
",
"interestsTitle": [
"SpaceX",
"Artificial Intelligence",
"Skateboarding",
"Anime",
"Gaming",
"Movie"
],
"secondJobEndDateTime": "2021-02-01T00:00:00+04:00",
"secondJobStartYear": "2020-01-01T00:00:00+04:00",
"topLinks": [
"Developer",
"Github",
"LinkedIn",
"Mail",
"Mauritius"
]
}
```
## License
Harvit is Apache 2.0 licensed.
## Stability
This project follows [SemVer](http://semver.org/) strictly and is not yet `v1`.
Breaking changes might be introduced until `v1` is released.
This project follows the [Go Release Policy](https://golang.org/doc/devel/release.html#policy). Each major version of Go is supported until there are two newer major releases.