https://github.com/denisecase/nlp-04-api-text-data
https://github.com/denisecase/nlp-04-api-text-data
Last synced: 2 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/denisecase/nlp-04-api-text-data
- Owner: denisecase
- License: mit
- Created: 2026-03-09T04:54:56.000Z (5 months ago)
- Default Branch: main
- Last Pushed: 2026-04-01T13:28:48.000Z (4 months ago)
- Last Synced: 2026-04-01T15:26:50.898Z (4 months ago)
- Language: Python
- Size: 104 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Citation: CITATION.cff
- Agents: .github/AGENTS.md
Awesome Lists containing this project
README
# nlp-04-api-text-data
[](#)
[](./LICENSE)
> Structured EVTL pipeline for reliable ingestion and transformation of JSON data from web APIs.
Web Mining and Applied NLP require reliable acquisition and processing of structured and semi-structured text data.
This project implements a reproducible pipeline for working with JSON data from web APIs.
The pipeline follows an EVTL architecture:
- Extract data from an external API
- Validate structure and content before use
- Transform JSON into a structured representation
- Load results into a persistent, analyzable format
The emphasis is on correctness, inspectability, and repeatability:
every stage has explicit inputs, outputs, and logging,
and intermediate artifacts are preserved for verification.
## This Project
This project demonstrates how to work with JSON data retrieved from web APIs using a structured EVTL pipeline.
The workflow:
- Acquire JSON data from an external source
- Inspect and validate its structure
- Transform it into a tabular representation
- Persist results for downstream analysis
Each stage is implemented as a modular component with explicit inputs and outputs.
## Key Files
These files define the EVTL pipeline and the components you will update for your project.
- **src/nlp/pipeline_api_json.py** - Main pipeline orchestrator (no changes required)
- **src/nlp/config_case.py** - Configuration for API access and paths (**copy and edit** for your project)
- **src/nlp/stage01_extract.py** - Extract stage: retrieves data from the API (no changes required)
- **src/nlp/stage02_validate_case.py** - Validate stage: inspects and verifies JSON structure (**copy and edit**)
- **src/nlp/stage03_transform_case.py** - Transform stage: converts JSON into structured data (**copy and edit**)
- **src/nlp/stage04_load.py** - Load stage: writes output to persistent storage (no changes required)
- **pyproject.toml** - Project metadata and dependencies (**update** authorship, links, and dependencies)
- **zensical.toml** - Documentation configuration (**update** authorship and links)
## First: Follow These Instructions
Follow the [step-by-step workflow guide](https://denisecase.github.io/pro-analytics-02/workflow-b-apply-example-project/) to complete:
1. Phase 1. **Start & Run**
2. Phase 2. **Change Authorship**
3. Phase 3. **Read & Understand**
## Success
After running the script successfully, you will see:
```shell
========================
Pipeline executed successfully!
========================
```
The following artifacts will be created:
- project.log - confirming successful run
- data/raw/case_raw.json - dump of the fetched JSON
- data/processed/case_processed.csv - final loaded result
## Command Reference
The commands below are used in the workflow guide above.
They are provided here for convenience.
Follow the guide for the **full instructions**.
Show command reference
### In a machine terminal (open in your `Repos` folder)
After you get a copy of this repo in your own GitHub account,
open a machine terminal in your `Repos` folder:
```shell
# Replace username with YOUR GitHub username.
git clone https://github.com/username/nlp-04-api-text-data
cd nlp-04-api-text-data
code .
```
### In a VS Code terminal
```shell
uv self update
uv python pin 3.14
uv sync --extra dev --extra docs --upgrade
uvx pre-commit install
git add -A
uvx pre-commit run --all-files
# repeat if changes were made
git add -A
uvx pre-commit run --all-files
# Later, we install spacy data model and
# en_core_web_sm = english, core, web, small
# It's big: spacy+data ~200+ MB w/ model installed
# ~350–450 MB for .venv is normal for NLP
# uv run python -m spacy download en_core_web_sm
# First, run the module
# IMPORTANT: Close each figure after viewing so execution continues
uv run python -m nlp.pipeline_api_json
uv run ruff format .
uv run ruff check . --fix
uv run zensical build
git add -A
git commit -m "update"
git push -u origin main
```
## Notes
- Use the **UP ARROW** and **DOWN ARROW** in the terminal to scroll through past commands.
- Use `CTRL+f` to find (and replace) text within a file.
## Example Artifact (Output)
```text
START PIPELINE
ROOT_PATH = .
DATA_PATH = data
RAW_PATH = data\raw
PROCESSED_PATH = data\processed
========================
STAGE 01: EXTRACT starting...
========================
SOURCE PATH = https://jsonplaceholder.typicode.com/posts
SINK PATH = data\raw\case_raw.json
========================
STAGE 02: VALIDATE starting...
========================
JSON STRUCTURE INSPECTION:
Top-level type: list
Keys in first record: ['userId', 'id', 'title', 'body']
Field types:
userId: int
id: int
title: str
body: str
Validation passed.
Sink: validated JSON object
========================
STAGE 03: TRANSFORM starting...
========================
Transformation complete.
DataFrame preview:
shape: (5, 6)
...preview of dataframe...
Sink: Polars DataFrame created
========================
STAGE 04: LOAD starting...
========================
SINK PATH = data\processed\case_processed.csv
========================
Pipeline executed successfully!
========================
```
## Enhancements
In production systems, validation is often automated using tools
such as **Great Expectations** or **Soda**.
Within the EVTL architecture, **VALIDATE** is a key stage
with a clear source, process, and sink:
- **Source**: JSON data extracted from the API
- **Process**: checking structure, confirming assumptions, and identifying data quality issues
- **Sink**: validated JSON passed to the TRANSFORM stage
This stage ensures the data is in a **consistent and reliable form**
before transformation begins,
so later steps can run without errors or unexpected results.
In this project, validation is implemented directly,
so all checks are visible, repeatable, and easy to review as part
of the pipeline.