{"id":15050530,"url":"https://github.com/ashakoen/bls-data-extract","last_synced_at":"2026-04-01T17:17:32.426Z","repository":{"id":257257162,"uuid":"857741771","full_name":"ashakoen/bls-data-extract","owner":"ashakoen","description":"This repository contains scripts and a database schema to set up and manage a local SQLite database for storing and querying the Average Price data from the U.S. Bureau of Labor Statistics. It includes tools for downloading the latest data from the BLS website and fetching Consumer Price Index (CPI) data via the BLS API.","archived":false,"fork":false,"pushed_at":"2024-09-15T14:10:14.000Z","size":9,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-28T00:37:12.544Z","etag":null,"topics":["data","government","sqlite","us"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ashakoen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-15T13:47:20.000Z","updated_at":"2024-09-15T14:10:17.000Z","dependencies_parsed_at":"2024-09-15T15:40:05.073Z","dependency_job_id":null,"html_url":"https://github.com/ashakoen/bls-data-extract","commit_stats":null,"previous_names":["ashakoen/bls-data-extract"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ashakoen/bls-data-extract","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashakoen%2Fbls-data-extract","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashakoen%2Fbls-data-extract/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashakoen%2Fbls-data-extract/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashakoen%2Fbls-data-extract/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ashakoen","download_url":"https://codeload.github.com/ashakoen/bls-data-extract/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashakoen%2Fbls-data-extract/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31290537,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T13:12:26.723Z","status":"ssl_error","status_checked_at":"2026-04-01T13:12:25.102Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","government","sqlite","us"],"created_at":"2024-09-24T21:27:12.166Z","updated_at":"2026-04-01T17:17:32.401Z","avatar_url":"https://github.com/ashakoen.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Average Price Data (AP) Database\n\n## Table of Contents\n\n- [Introduction](#introduction)\n- [Database Structure](#database-structure)\n  - [Tables and Their Relationships](#tables-and-their-relationships)\n  - [Schema Definition](#schema-definition)\n- [Data Flow for Query Construction](#data-flow-for-query-construction)\n- [Setup Instructions](#setup-instructions)\n  - [Prerequisites](#prerequisites)\n  - [Installing Dependencies](#installing-dependencies)\n  - [Setting Up the Database](#setting-up-the-database)\n  - [Downloading Data](#downloading-data)\n  - [Fetching CPI Data via API](#fetching-cpi-data-via-api)\n- [Usage Examples](#usage-examples)\n  - [Sample Query Structure](#sample-query-structure)\n- [Important Notes](#important-notes)\n- [Contributing](#contributing)\n- [License](#license)\n\n---\n\n## Introduction\n\nThe **Average Price Data (AP)** from the **Bureau of Labor Statistics (BLS)** provides detailed information on average consumer prices for household fuels, motor fuels, and food items. Collected monthly across various urban areas in the United States, this data is crucial for measuring the price levels of specific items over time and across different regions.\n\nThis repository contains scripts and a database schema to set up and manage a local SQLite database for storing and querying the AP data. It includes tools for downloading the latest data from the BLS website and fetching Consumer Price Index (CPI) data via the BLS API.\n\n---\n\n## Database Structure\n\nThe database comprises several tables that store data about items, areas, periods, series, and the actual price observations. Understanding the schema and relationships between these tables is crucial for constructing accurate SQL queries and extracting meaningful insights.\n\n### Tables and Their Relationships\n\n1. **```ap_item```**\n\n   - **Purpose**: Stores information about the items for which average prices are recorded.\n   - **Fields**:\n     - `item_code` (TEXT, PRIMARY KEY): Unique identifier for each item.\n     - `item_name` (TEXT): Descriptive name of the item.\n   - **Example Entries**:\n     - `701111`: Flour, white, all purpose, per lb. (453.6 gm)\n     - `702111`: Sugar, white, all sizes, per lb. (453.6 gm)\n\n2. **```ap_area```**\n\n   - **Purpose**: Contains information about the geographic areas covered in the survey.\n   - **Fields**:\n     - `area_code` (TEXT, PRIMARY KEY): Unique identifier for each area.\n     - `area_name` (TEXT): Descriptive name of the area.\n   - **Example Entries**:\n     - `0000`: U.S. city average\n     - `A100`: Northeast Urban\n     - `S200`: South Urban\n\n3. **```ap_period```**\n\n   - **Purpose**: Defines the periods (months) for which data is collected.\n   - **Fields**:\n     - `period` (TEXT, PRIMARY KEY): Code representing the period (e.g., M01 for January).\n     - `period_abbr` (TEXT): Abbreviation of the period name (e.g., JAN).\n     - `period_name` (TEXT): Full name of the period (e.g., January).\n   - **Example Entries**:\n     - `M01`: JAN, January\n     - `M02`: FEB, February\n\n4. **```ap_series```**\n\n   - **Purpose**: Provides metadata about each time series, linking items and areas.\n   - **Fields**:\n     - `series_id` (TEXT, PRIMARY KEY): Unique identifier for each time series.\n     - `area_code` (TEXT): References `ap_area.area_code`.\n     - `item_code` (TEXT): References `ap_item.item_code`.\n     - `series_title` (TEXT): Title describing the series.\n     - `footnote_codes` (TEXT): Any associated footnotes.\n     - `begin_year` (INTEGER): First year of data availability.\n     - `begin_period` (TEXT): First period of data availability.\n     - `end_year` (INTEGER): Last year of data availability.\n     - `end_period` (TEXT): Last period of data availability.\n   - **Relationships**:\n     - `ap_series.area_code` → `ap_area.area_code`\n     - `ap_series.item_code` → `ap_item.item_code`\n\n5. **```ap_data_current```**\n\n   - **Purpose**: Holds current year-to-date average price data.\n   - **Fields**:\n     - `series_id` (TEXT): References `ap_series.series_id`.\n     - `year` (INTEGER): Year of the observation.\n     - `period` (TEXT): References `ap_period.period`.\n     - `value` (REAL): Observed average price.\n     - `footnote_codes` (TEXT): Any associated footnotes.\n   - **Primary Key**: `(series_id, year, period)`\n   - **Relationships**:\n     - `ap_data_current.series_id` → `ap_series.series_id`\n     - `ap_data_current.period` → `ap_period.period`\n\n6. **```ap_data_food```**\n\n   - **Purpose**: Contains average price data for food items.\n   - **Fields and Relationships**: Same as `ap_data_current`.\n\n7. **```ap_data_gasoline```**\n\n   - **Purpose**: Contains average price data for gasoline.\n   - **Fields and Relationships**: Same as `ap_data_current`.\n\n8. **```ap_data_householdfuels```**\n\n   - **Purpose**: Contains average price data for household fuels.\n   - **Fields and Relationships**: Same as `ap_data_current`.\n\n9. **```ap_seasonal```**\n\n   - **Purpose**: Stores information about seasonal adjustment codes.\n   - **Fields**:\n     - `seasonal_code` (TEXT, PRIMARY KEY): Code indicating seasonal adjustment.\n     - `seasonal_text` (TEXT): Description of the seasonal code.\n\n### Schema Definition\n\nBelow is the SQL schema used to create the tables:\n\n```sql\nCREATE TABLE ap_item (\n    item_code TEXT PRIMARY KEY,\n    item_name TEXT\n);\n\nCREATE TABLE ap_area (\n    area_code TEXT PRIMARY KEY,\n    area_name TEXT\n);\n\nCREATE TABLE ap_period (\n    period TEXT PRIMARY KEY,\n    period_abbr TEXT,\n    period_name TEXT\n);\n\nCREATE TABLE ap_seasonal (\n    seasonal_code TEXT PRIMARY KEY,\n    seasonal_text TEXT\n);\n\nCREATE TABLE ap_series (\n    series_id TEXT PRIMARY KEY,\n    area_code TEXT,\n    item_code TEXT,\n    series_title TEXT,\n    footnote_codes TEXT,\n    begin_year INTEGER,\n    begin_period TEXT,\n    end_year INTEGER,\n    end_period TEXT\n);\n\nCREATE TABLE ap_data_current (\n    series_id TEXT,\n    year INTEGER,\n    period TEXT,\n    value REAL,\n    footnote_codes TEXT,\n    PRIMARY KEY(series_id, year, period)\n);\n\nCREATE TABLE ap_data_food (\n    series_id TEXT,\n    year INTEGER,\n    period TEXT,\n    value REAL,\n    footnote_codes TEXT,\n    PRIMARY KEY(series_id, year, period)\n);\n\nCREATE TABLE ap_data_gasoline (\n    series_id TEXT,\n    year INTEGER,\n    period TEXT,\n    value REAL,\n    footnote_codes TEXT,\n    PRIMARY KEY(series_id, year, period)\n);\n\nCREATE TABLE ap_data_householdfuels (\n    series_id TEXT,\n    year INTEGER,\n    period TEXT,\n    value REAL,\n    footnote_codes TEXT,\n    PRIMARY KEY(series_id, year, period)\n);\n\nCREATE TABLE cpi_info (\n    series_id TEXT,\n    year INTEGER,\n    period TEXT,\n    value REAL,\n    footnote_codes TEXT,\n    PRIMARY KEY(series_id, year, period)\n);\n```\n\n---\n\n## Data Flow for Query Construction\n\nTo construct a query that retrieves specific average price data, follow these steps:\n\n1. **Identify the Item**:\n   - Use `ap_item` to find the `item_code` corresponding to the desired `item_name`.\n\n2. **Identify the Area**:\n   - Use `ap_area` to find the `area_code` corresponding to the desired `area_name`.\n\n3. **Find the Series ID**:\n   - Use `ap_series` to find the `series_id` matching both the `item_code` and `area_code`.\n\n4. **Retrieve Data Observations**:\n   - Use the `series_id` to query the appropriate `ap_data_*` table (`ap_data_food`, `ap_data_gasoline`, etc.) for the desired `year` and `period`.\n\n5. **Join Period Information**:\n   - Use `ap_period` to translate `period` codes into readable `period_name` values.\n\n---\n\n## Setup Instructions\n\n### Prerequisites\n\n- **Python 3.6+**\n- **SQLite3**\n- **pip** (Python package installer)\n- **Virtual Environment** (recommended)\n\n### Installing Dependencies\n\n```bash\n# Clone the repository\ngit clone https://github.com/yourusername/ap-database.git\ncd ap-database\n\n# Create a virtual environment (optional but recommended)\npython -m venv venv\nsource venv/bin/activate  # On Windows, use venv\\Scripts\\activate\n\n# Install required Python packages\npip install -r requirements.txt\n```\n\n### Setting Up the Database\n\nRun the `seed_data.py` script to initialize the database:\n\n```bash\npython seed_data.py\n```\n\nThis script will:\n\n- Create the SQLite database named `average_price_data.db`.\n- Create all the tables as per the schema.\n- Load data from local CSV files into the database.\n\n### Downloading Data\n\nUse the `get_http.py` script to download the necessary data files from the BLS website:\n\n```bash\npython get_http.py\n```\n\nThis script will:\n\n- Download specified files from the BLS FTP site.\n- Save them in the `downloads` directory.\n\n**Note**: Ensure that the `downloads` directory exists or will be created by the script.\n\n### Fetching CPI Data via API\n\nUse the `get_api.py` script to fetch Consumer Price Index (CPI) data via the BLS API:\n\n1. **Obtain a BLS API Key**:\n\n   - Register at the [BLS website](https://data.bls.gov/registrationEngine/) to obtain an API key.\n   - Store the API key in a `.env` file in the project root:\n\n     ```\n     BLS_API_KEY=your_api_key_here\n     ```\n\n2. **Run the Script**:\n\n   ```bash\n   python get_api.py\n   ```\n\n   This script will:\n\n   - Fetch CPI data for specified `series_id`, `start_year`, and `end_year`.\n   - Save the data into text files and insert it into the `cpi_info` table in the database.\n\n---\n\n## Usage Examples\n\n### Sample Query Structure\n\nTo retrieve specific average price data, you can use the following SQL query structure:\n\n```sql\nSELECT\n  d.year,\n  p.period_name,\n  i.item_name,\n  a.area_name,\n  d.value\nFROM\n  ap_data_food AS d\nJOIN\n  ap_series AS s ON d.series_id = s.series_id\nJOIN\n  ap_item AS i ON s.item_code = i.item_code\nJOIN\n  ap_area AS a ON s.area_code = a.area_code\nJOIN\n  ap_period AS p ON d.period = p.period\nWHERE\n  i.item_name = 'Sugar, white, all sizes, per lb. (453.6 gm)'\n  AND a.area_name = 'U.S. city average'\nORDER BY\n  d.year, p.period_name;\n```\n\nThis query will:\n\n- Retrieve the average price of sugar per pound in U.S. city averages.\n- Display the data ordered by year and month.\n\n---\n\n## Important Notes\n\n- **Primary Keys**:\n  - Ensure uniqueness and efficient data retrieval.\n\n- **Foreign Keys**:\n  - Maintain referential integrity between tables.\n\n- **Data Partitioning**:\n  - Data is divided into specific tables based on item categories for optimized access.\n\n- **Understanding Period Codes**:\n  - Monthly Periods:\n    - `M01` to `M12` represent January to December.\n  - Annual Averages:\n    - `M13` may be used to represent annual average data.\n\n---\n\n## Contributing\n\nContributions are welcome! Please open an issue or submit a pull request for any improvements or bug fixes.\n\n---\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE).\n\n---","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashakoen%2Fbls-data-extract","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fashakoen%2Fbls-data-extract","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashakoen%2Fbls-data-extract/lists"}