{"id":19874268,"url":"https://github.com/alimghmi/ice-client","last_synced_at":"2026-04-10T23:07:08.677Z","repository":{"id":203358072,"uuid":"708001352","full_name":"alimghmi/ice-client","owner":"alimghmi","description":"Web scraper for ICE (Intercontinental Exchange) Markit Settlement Prices. Transforms and inserts data into an MSSQL database.","archived":false,"fork":false,"pushed_at":"2023-11-07T14:58:09.000Z","size":25,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-11T16:50:35.030Z","etag":null,"topics":["automated-data-fetch","data-transformation","database-integration","financial-data","ice","markit-settlement-prices","mssql","python","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alimghmi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-21T08:08:38.000Z","updated_at":"2024-11-19T01:49:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"cea8021b-3467-422d-8a70-e08938c0babb","html_url":"https://github.com/alimghmi/ice-client","commit_stats":null,"previous_names":["alimghmi/ice-client"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alimghmi%2Fice-client","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alimghmi%2Fice-client/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alimghmi%2Fice-client/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alimghmi%2Fice-client/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alimghmi","download_url":"https://codeload.github.com/alimghmi/ice-client/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241301993,"owners_count":19940747,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automated-data-fetch","data-transformation","database-integration","financial-data","ice","markit-settlement-prices","mssql","python","web-scraping"],"created_at":"2024-11-12T16:22:34.702Z","updated_at":"2026-04-10T23:07:08.617Z","avatar_url":"https://github.com/alimghmi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ICE Data Scraper with Docker\n\nScrape, transform, and insert ICE (Intercontinental Exchange) settlement prices into an MSSQL database, with Docker support for easy deployment and scaling.\n\n\n## 📌 Features:\n\n- **Efficient Scraping:** Targets the Markit ICE Settlement Prices from ICE's official page.\n- **Data Transformation:** Tailored data transformation for easy database insertion.\n- **MSSQL Support:** Built-in support to insert data into a Microsoft SQL Server database.\n- **Dockerized:** Simplified deployment and setup using Docker.\n- **Robust Error Handling:** Multi-retry mechanisms and comprehensive logging.\n\n\n## Getting Started:\n\n### Environment Setup\n\n1. Clone the repository:\n   ``` bash \n    git clone git@github.com:alimghmi/ice-client.git\n    cd ice-client\n   ```\n2. Create an `.env` file in the project root and configure the following:\n   ```\n    URL=\"https://www.ice.com/public-web/cds-settlement-prices/icc/single-names\"\n    LOG_LEVEL=\"INFO\"\n    OUTPUT_TABLE=\u003cname_of_the_output_table\u003e\n    INSERTER_MAX_RETRIES=2\n    REQUEST_MAX_RETRIES=3\n    REQUEST_BACKOFF_FACTOR=2\n    MSSQL_SERVER=\u003cmssql_server\u003e\n    MSSQL_DATABASE=\u003cmssql_database\u003e\n    MSSQL_USERNAME=\u003cmssql_username\u003e\n    MSSQL_PASSWORD=\u003cmssql_password\u003e\n   ```\n    Replace the placeholders (`\u003c...\u003e`) with the appropriate values.\n\n### Running with Python\n\n1. Install the required packages:\n   ```bash\n   pip install -r requirements.txt\n   ```\n2. Run the `main.py` script:\n   ```bash\n   python main.py\n   ```\n\n### Running with Docker\n\n1. Build the Docker image:\n   ```bash\n   docker build -t ice-data-scraper .\n   ```\n2. Run the Docker container:\n   ```bash\n   docker run --env-file .env ice-data-scraper\n   ```\n\n## Contribution\n\nFeel free to fork the repository, make changes, and open a pull request. All contributions are welcomed!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falimghmi%2Fice-client","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falimghmi%2Fice-client","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falimghmi%2Fice-client/lists"}