https://github.com/alimghmi/ice-client
Web scraper for ICE (Intercontinental Exchange) Markit Settlement Prices. Transforms and inserts data into an MSSQL database.
https://github.com/alimghmi/ice-client
automated-data-fetch data-transformation database-integration financial-data ice markit-settlement-prices mssql python web-scraping
Last synced: 8 months ago
JSON representation
Web scraper for ICE (Intercontinental Exchange) Markit Settlement Prices. Transforms and inserts data into an MSSQL database.
- Host: GitHub
- URL: https://github.com/alimghmi/ice-client
- Owner: alimghmi
- Created: 2023-10-21T08:08:38.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2023-11-07T14:58:09.000Z (almost 2 years ago)
- Last Synced: 2025-01-11T16:50:35.030Z (9 months ago)
- Topics: automated-data-fetch, data-transformation, database-integration, financial-data, ice, markit-settlement-prices, mssql, python, web-scraping
- Language: Python
- Homepage:
- Size: 24.4 KB
- Stars: 3
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# ICE Data Scraper with Docker
Scrape, transform, and insert ICE (Intercontinental Exchange) settlement prices into an MSSQL database, with Docker support for easy deployment and scaling.
## 📌 Features:
- **Efficient Scraping:** Targets the Markit ICE Settlement Prices from ICE's official page.
- **Data Transformation:** Tailored data transformation for easy database insertion.
- **MSSQL Support:** Built-in support to insert data into a Microsoft SQL Server database.
- **Dockerized:** Simplified deployment and setup using Docker.
- **Robust Error Handling:** Multi-retry mechanisms and comprehensive logging.
## Getting Started:
### Environment Setup
1. Clone the repository:
``` bash
git clone git@github.com:alimghmi/ice-client.git
cd ice-client
```
2. Create an `.env` file in the project root and configure the following:
```
URL="https://www.ice.com/public-web/cds-settlement-prices/icc/single-names"
LOG_LEVEL="INFO"
OUTPUT_TABLE=
INSERTER_MAX_RETRIES=2
REQUEST_MAX_RETRIES=3
REQUEST_BACKOFF_FACTOR=2
MSSQL_SERVER=
MSSQL_DATABASE=
MSSQL_USERNAME=
MSSQL_PASSWORD=
```
Replace the placeholders (`<...>`) with the appropriate values.
### Running with Python
1. Install the required packages:
```bash
pip install -r requirements.txt
```
2. Run the `main.py` script:
```bash
python main.py
```
### Running with Docker
1. Build the Docker image:
```bash
docker build -t ice-data-scraper .
```
2. Run the Docker container:
```bash
docker run --env-file .env ice-data-scraper
```
## Contribution
Feel free to fork the repository, make changes, and open a pull request. All contributions are welcomed!