{"id":30193498,"url":"https://github.com/johnbdfilio/bitget-data-collector-10ma","last_synced_at":"2025-08-13T01:59:14.210Z","repository":{"id":303445155,"uuid":"968267632","full_name":"johnbdfilio/Bitget-Data-Collector-10MA","owner":"johnbdfilio","description":"A Python script that collects and stores cryptocurrency price data from the Bitget API into MongoDB time-series collections. This tool is designed to maintain a rolling window of the latest UNLIMITED records for specified trading pairs.","archived":false,"fork":false,"pushed_at":"2025-07-13T06:37:51.000Z","size":13,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-06T23:41:06.176Z","etag":null,"topics":["aitrading","bitget","bitget-api","candles","crypto","cryptocurrency","datacollection","datacollector","timeseries","timeseries-forecasting","trading-algorithms","tradingbot","unlimited"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/johnbdfilio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-17T19:35:12.000Z","updated_at":"2025-07-13T06:37:54.000Z","dependencies_parsed_at":"2025-07-07T17:47:53.157Z","dependency_job_id":"ec313f60-78f2-4a6d-8f49-cfe3c186da98","html_url":"https://github.com/johnbdfilio/Bitget-Data-Collector-10MA","commit_stats":null,"previous_names":["benrey916/bitget-data-collector-10ma","johnbdfilio/bitget-data-collector-10ma"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/johnbdfilio/Bitget-Data-Collector-10MA","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnbdfilio%2FBitget-Data-Collector-10MA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnbdfilio%2FBitget-Data-Collector-10MA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnbdfilio%2FBitget-Data-Collector-10MA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnbdfilio%2FBitget-Data-Collector-10MA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/johnbdfilio","download_url":"https://codeload.github.com/johnbdfilio/Bitget-Data-Collector-10MA/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnbdfilio%2FBitget-Data-Collector-10MA/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270166107,"owners_count":24538445,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-12T02:00:09.011Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aitrading","bitget","bitget-api","candles","crypto","cryptocurrency","datacollection","datacollector","timeseries","timeseries-forecasting","trading-algorithms","tradingbot","unlimited"],"created_at":"2025-08-13T01:58:27.104Z","updated_at":"2025-08-13T01:59:14.193Z","avatar_url":"https://github.com/johnbdfilio.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Crypto Price Data Collector || Bitget API\n\nA Python script that collects and stores cryptocurrency price data from the Bitget API into MongoDB time-series collections. This tool is designed to maintain a rolling window of the latest 3,000 or UNLIMITED records for specified trading pairs.\n\n## Features\n\n- **Historical Data Fetching**: Initial bulk fetch of up to 3,000 historical candlestick records per token.\n- **Time-Series Collections**: Utilizes MongoDB time-series collections for efficient storage and querying.\n- **Gap Filling**: Detects and fills missing data between the last recorded entry and the current time.\n- **Live Updates**: Continuously fetches new data at the start of each minute.\n- **Duplicate Prevention**: Checks for existing timestamps to avoid redundant data.\n- **Automatic Cleanup**: Ensures only the latest 3,000 records are retained per token.\n\n## Supported Tokens \u0026 Intervals\n- **Tokens**: `ETHUSDT`, `ADAUSDT` (configurable).\n- **Interval**: 1-minute candlesticks.\n\n## Prerequisites\n\n- Python 3.8+\n- MongoDB Atlas cluster (or local MongoDB instance with time-series support).\n- Python Libraries: \n  ```bash\n  pip install requests pymongo python-dotenv\n  ```\n\n## Configuration\n\n1. **MongoDB Connection**:\n   - Replace the `MONGODB_URI` in the script with your MongoDB connection string.\n   - Format: `mongodb+srv://\u003cusername\u003e:\u003cpassword\u003e@cluster0.iittg.mongodb.net/alert3`.\n\n2. **Script Parameters** (Modify in Code):\n   ```python\n   TOKENS = [\"ETHUSDT\", \"ADAUSDT\"]  # Add/remove trading pairs\n   INTERVAL = \"1min\"                 # Supported: 1min, 5min, 15min, etc. (check Bitget API)\n   LIMIT = 200                       # Max records per API request (do not exceed API limits)\n   TOTAL_RECORDS = 3000              # Records to retain per token\n   SLEEP_TIME = 2                    # Seconds between API requests to avoid rate limits\n   ```\n\n## Installation\n\n1. **Clone the Repository**:\n   ```bash\n   git clone https://github.com/yourusername/crypto-data-collector.git\n   cd crypto-data-collector\n   ```\n\n2. **Install Dependencies**:\n   ```bash\n   pip install -r requirements.txt  # Create a requirements.txt with the libraries\n   ```\n\n## Usage\n\n### Step 1: Run the Script\n```bash\npython data_collector.py\n```\n\n### Workflow\n1. **MongoDB Connection Check**: Verifies connectivity to your database.\n2. **Create Time-Series Collections**: If they don't exist.\n3. **Fetch Initial Data**: Populates up to 3,000 historical records per token.\n4. **Fill Gaps**: Ensures no missing data between the oldest record and the current time.\n5. **Live Updates**: Runs indefinitely, fetching new data every minute.\n\n## Database Structure\nEach token has a dedicated time-series collection named `\u003cTOKEN\u003e_timeseries` (e.g., `ETHUSDT_timeseries`). Documents include:\n```javascript\n{\n  \"token\": \"ETHUSDT\",           // Trading pair\n  \"timestamp\": ISODate(\"2023-10-01T00:00:00Z\"),  // UTC time\n  \"open\": 1700.5,               // Opening price\n  \"high\": 1712.3,               // Highest price during the interval\n  \"low\": 1698.2,                // Lowest price\n  \"close\": 1705.7,              // Closing price\n  \"base_volume\": 450.2,         // Volume in the base currency (e.g., ETH)\n  \"quote_volume\": 765432.1      // Volume in the quote currency (e.g., USDT)\n}\n```\n\n## Troubleshooting\n\n- **Connection Errors**:\n  - Ensure the MongoDB URI is correct and whitelisted in Atlas.\n  - Check network connectivity.\n\n- **No Data Fetched**:\n  - Verify the token symbols match Bitget's supported pairs.\n  - Check Bitget API status for outages.\n\n- **Gap Filling Issues**:\n  - The loop condition in `fill_gaps()` may contain a bug. Replace `\u0026` with `and`:\n    ```python\n    while fetched \u003c total_minutes and total_minutes \u003e 0:\n    ```\n\n- **Rate Limits**:\n  - Increase `SLEEP_TIME` if encountering HTTP 429 errors.\n\n## License\nMIT License. Replace with your preferred license.\n\n---\n\n**Note**: Avoid committing sensitive data (e.g., MongoDB credentials). Use environment variables in production.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnbdfilio%2Fbitget-data-collector-10ma","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjohnbdfilio%2Fbitget-data-collector-10ma","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnbdfilio%2Fbitget-data-collector-10ma/lists"}