{"id":27058713,"url":"https://github.com/ekoepplin/dlt-data-dumper","last_synced_at":"2026-04-10T20:42:01.421Z","repository":{"id":259010908,"uuid":"876103727","full_name":"ekoepplin/dlt-data-dumper","owner":"ekoepplin","description":"Illustrates how to use dlt (data load tool) in combination with newsapi (source), aws s3 (destination) and duckDB (local testing)","archived":false,"fork":false,"pushed_at":"2024-12-16T20:56:58.000Z","size":134,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-05T12:16:29.360Z","etag":null,"topics":["dlt","dlthub","duckdb","github-actions","newsapi-org"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ekoepplin.png","metadata":{"files":{"readme":"README.md","changelog":"newsapi_pipeline.py","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-21T12:07:01.000Z","updated_at":"2024-12-16T20:57:02.000Z","dependencies_parsed_at":"2024-11-28T14:28:27.442Z","dependency_job_id":"dee2c0e5-d7fc-48d0-ab5c-c5ad8f9f6995","html_url":"https://github.com/ekoepplin/dlt-data-dumper","commit_stats":null,"previous_names":["ekoepplin/dlt-newsapi-dumper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ekoepplin/dlt-data-dumper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekoepplin%2Fdlt-data-dumper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekoepplin%2Fdlt-data-dumper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekoepplin%2Fdlt-data-dumper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekoepplin%2Fdlt-data-dumper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ekoepplin","download_url":"https://codeload.github.com/ekoepplin/dlt-data-dumper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekoepplin%2Fdlt-data-dumper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269809476,"owners_count":24478545,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-10T02:00:08.965Z","response_time":71,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dlt","dlthub","duckdb","github-actions","newsapi-org"],"created_at":"2025-04-05T12:16:31.984Z","updated_at":"2026-04-10T20:42:01.372Z","avatar_url":"https://github.com/ekoepplin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NewsAPI Data Ingestion with DLT\n\nThis project demonstrates how to ingest data from the NewsAPI using the Data Load Tool (DLT) library. It fetches top headlines, article searches, and news sources, then loads them into a DuckDB database.\n\n## Features\n\n- Fetch top headlines from NewsAPI\n- Search for articles on specific topics\n- Retrieve news sources information\n- Load data into DuckDB using DLT\n- Jupyter notebook for interactive data exploration\n\n## Prerequisites\n\n- Python 3.11.8\n- Pipenv for dependency management\n\n## Installation\n\n1. Clone this repository:\n   ```\n   git clone \u003crepository-url\u003e\n   cd dlt-data-dumper\n   ```\n\n2. Install dependencies using Pipenv:\n   ```\n   pipenv install\n   ```\n\n3. Activate the virtual environment:\n   ```\n   pipenv shell\n   ```\n\n4. Set up your NewsAPI key:\n   - Sign up for a free API key at [https://newsapi.org/](https://newsapi.org/)\n   - Set the API key as an environment variable:\n     ```\n     export NEWS_API_KEY=your_api_key_here\n     ```\n\n## Usage\n\n### Running the Pipeline\n\nTo run the data ingestion pipeline, use the `newsapi_pipeline.py` script. This script supports various command-line options for different execution modes:\n\n1. Normal mode:\n   ```\n   python newsapi_pipeline.py\n   ```\n\n2. Test mode (uses DuckDB instead of filesystem):\n   ```\n   python newsapi_pipeline.py --test\n   ```\n\n3. Full refresh (replaces existing data instead of appending):\n   ```\n   python newsapi_pipeline.py --full-refresh\n   ```\n\n4. Custom log level:\n   ```\n   python newsapi_pipeline.py --log-level DEBUG\n   ```\n\nYou can also combine these options as needed. For example:\n```\npython newsapi_pipeline.py --test --full-refresh --log-level DEBUG\n```\n\nThis script will fetch data from NewsAPI and load it into either a filesystem-based storage (default) or a DuckDB database (in test mode).\n\n### Exploring Data with Jupyter Notebook\n\n1. Start Jupyter Notebook:\n   ```\n   jupyter notebook\n   ```\n\n2. Open the `eda-newsapi.ipynb` notebook in your browser.\n\n3. Run the cells to fetch data and perform exploratory data analysis.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\nThis project is open-source and available under the [MIT License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fekoepplin%2Fdlt-data-dumper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fekoepplin%2Fdlt-data-dumper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fekoepplin%2Fdlt-data-dumper/lists"}