https://github.com/peterk/remote-af
Quick analysis sketch of Swedish remote job ads using DuckDB, PyArrow and Parquet
https://github.com/peterk/remote-af
Last synced: 9 months ago
JSON representation
Quick analysis sketch of Swedish remote job ads using DuckDB, PyArrow and Parquet
- Host: GitHub
- URL: https://github.com/peterk/remote-af
- Owner: peterk
- License: mit
- Created: 2024-01-23T20:27:11.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-01-23T20:59:13.000Z (over 2 years ago)
- Last Synced: 2025-06-05T13:05:16.390Z (about 1 year ago)
- Language: Python
- Size: 5.86 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# remote-af
Quick analysis sketch of Swedish remote job ads using [DuckDB](https://duckdb.org/), [PyArrow](https://arrow.apache.org/docs/python/index.html) and Parquet.
Please note that each year will download approximately 5GB json data.
## 1. Install Python dependencies
```
pip install -r requirements.txt
```
## 2. fetch_job_ads.sh
JSON line oriented data files are fetched from the [Swedish Employment Agency's open data store](https://data.jobtechdev.se/annonser/historiska/index.html).
For each downloaded json dump file, the make_parquet_from_json.py script is called, creating one parquet file for each year.
## 3. make_graph.py
Use DuckDB to query the parquet files and plot the cart.