{"id":19044249,"url":"https://github.com/arturogonzalezm/world_bank_data","last_synced_at":"2026-07-02T23:04:32.194Z","repository":{"id":249221185,"uuid":"830809496","full_name":"arturogonzalezm/world_bank_data","owner":"arturogonzalezm","description":null,"archived":false,"fork":false,"pushed_at":"2024-07-21T03:59:36.000Z","size":49,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-27T20:41:53.483Z","etag":null,"topics":["concurrent-futures","python3","rest-api","tenacity"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arturogonzalezm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-07-19T03:43:13.000Z","updated_at":"2024-07-21T03:59:38.000Z","dependencies_parsed_at":"2024-07-19T12:15:36.842Z","dependency_job_id":"335f9fed-3e8b-462e-851d-3b05cc9a4c35","html_url":"https://github.com/arturogonzalezm/world_bank_data","commit_stats":null,"previous_names":["arturogonzalezm/world_bank_data"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/arturogonzalezm/world_bank_data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arturogonzalezm%2Fworld_bank_data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arturogonzalezm%2Fworld_bank_data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arturogonzalezm%2Fworld_bank_data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arturogonzalezm%2Fworld_bank_data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arturogonzalezm","download_url":"https://codeload.github.com/arturogonzalezm/world_bank_data/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arturogonzalezm%2Fworld_bank_data/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35065706,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-02T02:00:06.368Z","response_time":173,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrent-futures","python3","rest-api","tenacity"],"created_at":"2024-11-08T22:45:18.248Z","updated_at":"2026-07-02T23:04:32.171Z","avatar_url":"https://github.com/arturogonzalezm.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![codecov](https://codecov.io/gh/arturogonzalezm/world_bank_data/graph/badge.svg?token=cW9uPbPSUD)](https://codecov.io/gh/arturogonzalezm/world_bank_data)\n[![PyLint](https://github.com/arturogonzalezm/world_bank_data/actions/workflows/workflow.yml/badge.svg)](https://github.com/arturogonzalezm/world_bank_data/actions/workflows/workflow.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-purple.svg)](https://opensource.org/licenses/MIT)\n\n# World Bank Data Downloader\n\nThis project provides a Python 3.12 class `WorldBankDataDownloader` for downloading data from the World Bank API. It includes functionality to fetch country codes, indicator codes, and data for specific country-indicator pairs.\n\n## Setup\n\nFollow these steps to set up your development environment:\n\n1. **Clone the repository**\n   ```\n   git clone https://github.com/arturogonzalezm/world_bank_data.git\n   cd world-bank-data-downloader\n   ```\n\n2. **Create a virtual environment**\n   - For Unix or MacOS:\n     ```\n     python3 -m venv .venv\n     source .venv/bin/activate\n     ```\n   - For Windows:\n     ```\n     python -m venv .venv\n     .venv\\Scripts\\activate\n     ```\n\n3. **Upgrade pip**\n   ```\n   pip install --upgrade pip\n   ```\n\n4. **Install dependencies**\n   ```\n   pip install -r requirements.txt\n   ```\n\n   If you don't have a `requirements.txt` file, you can create one with the following content:\n   ```\n    requests==2.32.3\n    pytest==8.2.2\n    coverage==7.6.0\n    pytest-cov==5.0.0\n    tenacity==8.5.0\n   ```\n   Then run the install command above.\n\n## Class Structure\n\nThe `WorldBankDataDownloader` class is structured as follows:\n\n```mermaid\nclassDiagram\n    class WorldBankDataDownloader {\n        +base_url : str\n        +country_codes : list\n        +indicator_codes : list\n        +__init__()\n        +get_country_codes() : list\n        +get_indicators() : list\n        +fetch_data(country_code: str, indicator_code: str) : list\n        +download_all_data() : dict\n        +save_data_to_file(data: dict, filename: str)\n        +load_data_from_file(filename: str) : dict\n    }\n```\n\n## Key Features\n\n1. **Retry Mechanism**: Uses the `tenacity` library to implement retry logic for API requests.\n2. **Pagination Handling**: Manages paginated responses from the World Bank API.\n3. **Rate Limiting**: Implements delays between requests to avoid overwhelming the API.\n4. **Error Handling**: Robust error handling for API requests and data processing.\n5. **Data Persistence**: Methods to save and load data to/from JSON files.\n\n## Sequence Diagram\n\nThe following sequence diagram illustrates the main interactions of the `WorldBankDataDownloader` class:\n\n```mermaid\nsequenceDiagram\n    participant Client\n    participant WBD as WorldBankDataDownloader\n    participant API as World Bank API\n    participant FileSystem\n\n    Client-\u003e\u003eWBD: create()\n    WBD-\u003e\u003eAPI: Fetch country codes\n    WBD-\u003e\u003eAPI: Fetch indicator codes\n    Client-\u003e\u003eWBD: download_all_data()\n    loop for each country and indicator\n        WBD-\u003e\u003eAPI: Fetch data\n    end\n    Client-\u003e\u003eWBD: save_data_to_file(data)\n    WBD-\u003e\u003eFileSystem: Write JSON\n    Client-\u003e\u003eWBD: load_data_from_file()\n    FileSystem--\u003e\u003eWBD: Read JSON\n    WBD--\u003e\u003eClient: Return data\n```\n\n## Flowchart\n\nThis flowchart outlines the main process of the `WorldBankDataDownloader`:\n\n```mermaid\ngraph TD\n    A[Start] --\u003e B[Initialize WorldBankDataDownloader]\n    B --\u003e C[Fetch country codes]\n    C --\u003e D[Fetch indicator codes]\n    D --\u003e E[Download all data]\n    E --\u003e F{For each country and indicator}\n    F --\u003e G[Fetch data]\n    G --\u003e H{More pairs?}\n    H --\u003e|Yes| F\n    H --\u003e|No| I[Save data to file]\n    I --\u003e J[End]\n\n    K[Load data from file] --\u003e L[Read and deserialize JSON]\n    L --\u003e M[Return data]\n```\n\n## Usage\n\nHere's a basic example of how to use the `WorldBankDataDownloader`:\n\n```python\ndownloader = WorldBankDataDownloader()\nall_data = downloader.download_all_data()\ndownloader.save_data_to_file(all_data, 'data/world_bank_data_optimised.json')\n```\n\n## Unit Tests\n\nThe project includes a comprehensive suite of unit tests using pytest. The test structure is as follows:\n\n```mermaid\ngraph TD\n    A[Setup Fixtures] --\u003e B[Test Initialization]\n    A --\u003e C[Test API Methods]\n    A --\u003e D[Test Data Processing]\n    A --\u003e E[Test File I/O]\n    A --\u003e F[Test Error Handling]\n    \n    C --\u003e C1[Test get_country_codes]\n    C --\u003e C2[Test get_indicators]\n    C --\u003e C3[Test fetch_data]\n    \n    D --\u003e D1[Test download_all_data]\n    \n    E --\u003e E1[Test save_data_to_file]\n    E --\u003e E2[Test load_data_from_file]\n    \n    F --\u003e F1[Test API Error Scenarios]\n```\n\n### Test Coverage\n\nThe unit tests cover:\n\n1. Class initialization\n2. API interaction methods (with mocked responses)\n3. Data processing logic\n4. File I/O operations\n5. Error handling scenarios\n\n### Running the Tests\n\nTo run the tests:\n\n1. Ensure you're in your virtual environment\n2. Run the command: `pytest test_data_downloader.py`\n\n## Dependencies\n\n- requests (for making HTTP requests)\n- tenacity (for retry logic)\n- pytest (for running tests)\n- pytest-cov (for test coverage)\n- coverage (for test coverage)\n\n## Notes\n\n- The World Bank API has rate limits. The class implements a basic delay between requests, but for large-scale data fetching, you may need to implement more sophisticated rate limiting.\n- Always check the World Bank API documentation for the most up-to-date information on endpoints and usage guidelines.\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farturogonzalezm%2Fworld_bank_data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farturogonzalezm%2Fworld_bank_data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farturogonzalezm%2Fworld_bank_data/lists"}