{"id":50508870,"url":"https://github.com/denisecase/nlp-04-api-text-data","last_synced_at":"2026-06-02T18:31:13.459Z","repository":{"id":348524166,"uuid":"1176508695","full_name":"denisecase/nlp-04-api-text-data","owner":"denisecase","description":null,"archived":false,"fork":false,"pushed_at":"2026-04-01T13:28:48.000Z","size":107,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-01T15:26:50.898Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/denisecase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":".github/AGENTS.md","dco":null,"cla":null}},"created_at":"2026-03-09T04:54:56.000Z","updated_at":"2026-04-01T13:28:52.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/denisecase/nlp-04-api-text-data","commit_stats":null,"previous_names":["denisecase/nlp-04-api-text-data"],"tags_count":null,"template":true,"template_full_name":null,"purl":"pkg:github/denisecase/nlp-04-api-text-data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fnlp-04-api-text-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fnlp-04-api-text-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fnlp-04-api-text-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fnlp-04-api-text-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/denisecase","download_url":"https://codeload.github.com/denisecase/nlp-04-api-text-data/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fnlp-04-api-text-data/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33833277,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-02T02:00:07.132Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-02T18:31:12.392Z","updated_at":"2026-06-02T18:31:13.453Z","avatar_url":"https://github.com/denisecase.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nlp-04-api-text-data\n\n[![Python 3.14+](https://img.shields.io/badge/python-3.14%2B-blue?logo=python)](#)\n[![MIT](https://img.shields.io/badge/license-see%20LICENSE-yellow.svg)](./LICENSE)\n\n\u003e Structured EVTL pipeline for reliable ingestion and transformation of JSON data from web APIs.\n\nWeb Mining and Applied NLP require reliable acquisition and processing of structured and semi-structured text data.\nThis project implements a reproducible pipeline for working with JSON data from web APIs.\n\nThe pipeline follows an EVTL architecture:\n\n- Extract data from an external API\n- Validate structure and content before use\n- Transform JSON into a structured representation\n- Load results into a persistent, analyzable format\n\nThe emphasis is on correctness, inspectability, and repeatability:\nevery stage has explicit inputs, outputs, and logging,\nand intermediate artifacts are preserved for verification.\n\n## This Project\n\nThis project demonstrates how to work with JSON data retrieved from web APIs using a structured EVTL pipeline.\n\nThe workflow:\n\n- Acquire JSON data from an external source\n- Inspect and validate its structure\n- Transform it into a tabular representation\n- Persist results for downstream analysis\n\nEach stage is implemented as a modular component with explicit inputs and outputs.\n\n## Key Files\n\nThese files define the EVTL pipeline and the components you will update for your project.\n\n- **src/nlp/pipeline_api_json.py** - Main pipeline orchestrator (no changes required)\n- **src/nlp/config_case.py** - Configuration for API access and paths (\u003cmark\u003e**copy and edit**\u003c/mark\u003e for your project)\n- **src/nlp/stage01_extract.py** - Extract stage: retrieves data from the API (no changes required)\n- **src/nlp/stage02_validate_case.py** - Validate stage: inspects and verifies JSON structure (\u003cmark\u003e**copy and edit**\u003c/mark\u003e)\n- **src/nlp/stage03_transform_case.py** - Transform stage: converts JSON into structured data (\u003cmark\u003e**copy and edit**\u003c/mark\u003e)\n- **src/nlp/stage04_load.py** - Load stage: writes output to persistent storage (no changes required)\n- **pyproject.toml** - Project metadata and dependencies (\u003cmark\u003e**update**\u003c/mark\u003e authorship, links, and dependencies)\n- **zensical.toml** - Documentation configuration (\u003cmark\u003e**update**\u003c/mark\u003e authorship and links)\n\n## First: Follow These Instructions\n\nFollow the [step-by-step workflow guide](https://denisecase.github.io/pro-analytics-02/workflow-b-apply-example-project/) to complete:\n\n1. Phase 1. **Start \u0026 Run**\n2. Phase 2. **Change Authorship**\n3. Phase 3. **Read \u0026 Understand**\n\n## Success\n\nAfter running the script successfully, you will see:\n\n```shell\n========================\nPipeline executed successfully!\n========================\n```\n\nThe following artifacts will be created:\n\n- project.log - confirming successful run\n- data/raw/case_raw.json - dump of the fetched JSON\n- data/processed/case_processed.csv - final loaded result\n\n## Command Reference\n\nThe commands below are used in the workflow guide above.\nThey are provided here for convenience.\n\nFollow the guide for the **full instructions**.\n\n\u003cdetails\u003e\n\u003csummary\u003eShow command reference\u003c/summary\u003e\n\n### In a machine terminal (open in your `Repos` folder)\n\nAfter you get a copy of this repo in your own GitHub account,\nopen a machine terminal in your `Repos` folder:\n\n```shell\n# Replace username with YOUR GitHub username.\ngit clone https://github.com/username/nlp-04-api-text-data\ncd nlp-04-api-text-data\ncode .\n```\n\n### In a VS Code terminal\n\n```shell\nuv self update\nuv python pin 3.14\nuv sync --extra dev --extra docs --upgrade\n\nuvx pre-commit install\ngit add -A\nuvx pre-commit run --all-files\n\n# repeat if changes were made\ngit add -A\nuvx pre-commit run --all-files\n\n# Later, we install spacy data model and\n# en_core_web_sm = english, core, web, small\n# It's big: spacy+data ~200+ MB w/ model installed\n#           ~350–450 MB for .venv is normal for NLP\n# uv run python -m spacy download en_core_web_sm\n\n# First, run the module\n# IMPORTANT: Close each figure after viewing so execution continues\nuv run python -m nlp.pipeline_api_json\n\nuv run ruff format .\nuv run ruff check . --fix\nuv run zensical build\n\ngit add -A\ngit commit -m \"update\"\ngit push -u origin main\n```\n\n\u003c/details\u003e\n\n## Notes\n\n- Use the **UP ARROW** and **DOWN ARROW** in the terminal to scroll through past commands.\n- Use `CTRL+f` to find (and replace) text within a file.\n\n## Example Artifact (Output)\n\n```text\nSTART PIPELINE\nROOT_PATH = .\nDATA_PATH = data\nRAW_PATH = data\\raw\nPROCESSED_PATH = data\\processed\n========================\nSTAGE 01: EXTRACT starting...\n========================\nSOURCE PATH = https://jsonplaceholder.typicode.com/posts\nSINK PATH = data\\raw\\case_raw.json\n========================\nSTAGE 02: VALIDATE starting...\n========================\nJSON STRUCTURE INSPECTION:\nTop-level type: list\nKeys in first record: ['userId', 'id', 'title', 'body']\nField types:\nuserId: int\nid: int\ntitle: str\nbody: str\nValidation passed.\nSink: validated JSON object\n========================\nSTAGE 03: TRANSFORM starting...\n========================\nTransformation complete.\nDataFrame preview:\nshape: (5, 6)\n...preview of dataframe...\nSink: Polars DataFrame created\n========================\nSTAGE 04: LOAD starting...\n========================\nSINK PATH = data\\processed\\case_processed.csv\n========================\nPipeline executed successfully!\n========================\n```\n\n## Enhancements\n\nIn production systems, validation is often automated using tools\nsuch as **Great Expectations** or **Soda**.\n\nWithin the EVTL architecture, **VALIDATE** is a key stage\nwith a clear source, process, and sink:\n\n- **Source**: JSON data extracted from the API\n- **Process**: checking structure, confirming assumptions, and identifying data quality issues\n- **Sink**: validated JSON passed to the TRANSFORM stage\n\nThis stage ensures the data is in a **consistent and reliable form**\nbefore transformation begins,\nso later steps can run without errors or unexpected results.\n\nIn this project, validation is implemented directly,\nso all checks are visible, repeatable, and easy to review as part\nof the pipeline.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenisecase%2Fnlp-04-api-text-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdenisecase%2Fnlp-04-api-text-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenisecase%2Fnlp-04-api-text-data/lists"}