{"id":23136506,"url":"https://github.com/sdq/kaggle-crawler","last_synced_at":"2026-05-15T20:01:58.699Z","repository":{"id":81318891,"uuid":"103374113","full_name":"sdq/kaggle-crawler","owner":"sdq","description":"simple scrapy project for kaggle.com","archived":false,"fork":false,"pushed_at":"2017-09-13T08:43:21.000Z","size":25,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-21T11:52:32.920Z","etag":null,"topics":["crawler","kaggle"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sdq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-09-13T08:27:54.000Z","updated_at":"2025-04-26T12:57:33.000Z","dependencies_parsed_at":"2023-09-01T10:48:58.577Z","dependency_job_id":null,"html_url":"https://github.com/sdq/kaggle-crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sdq/kaggle-crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sdq%2Fkaggle-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sdq%2Fkaggle-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sdq%2Fkaggle-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sdq%2Fkaggle-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sdq","download_url":"https://codeload.github.com/sdq/kaggle-crawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sdq%2Fkaggle-crawler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33077924,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-15T11:35:32.926Z","status":"ssl_error","status_checked_at":"2026-05-15T11:35:31.362Z","response_time":103,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","kaggle"],"created_at":"2024-12-17T12:21:08.761Z","updated_at":"2026-05-15T20:01:58.680Z","avatar_url":"https://github.com/sdq.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kaggle-crawler\n\n```python\nscrapy crawl competitions\nscrapy crawl datasets\nscrapy crawl kernels\nscrapy crawl discussion\n```\n\ncompetitions result demo\n```json\n{\n  \"sortByOptions\": [\n    {\n      \"value\": \"prize\",\n      \"label\": \"Prize\"\n    },\n    {\n      \"value\": \"deadline\",\n      \"label\": \"Deadline\"\n    },\n    {\n      \"value\": \"numberOfTeams\",\n      \"label\": \"Number of teams\"\n    },\n    {\n      \"value\": \"relevance\",\n      \"label\": \"Relevance\"\n    },\n    {\n      \"value\": \"recentlyCreated\",\n      \"label\": \"Recently created\"\n    }\n  ],\n  \"selectionGroupOptions\": [\n    {\n      \"value\": \"active\",\n      \"label\": \"Active\"\n    },\n    {\n      \"value\": \"all\",\n      \"label\": \"All\"\n    },\n    {\n      \"value\": \"entered\",\n      \"label\": \"Entered\"\n    }\n  ],\n  \"hostSegmentOptions\": [\n    {\n      \"value\": \"\",\n      \"label\": \"All Categories\"\n    },\n    {\n      \"value\": \"featured\",\n      \"label\": \"Featured\"\n    },\n    {\n      \"value\": \"research\",\n      \"label\": \"Research\"\n    },\n    {\n      \"value\": \"recruitment\",\n      \"label\": \"Recruitment\"\n    },\n    {\n      \"value\": \"gettingStarted\",\n      \"label\": \"Getting Started\"\n    },\n    {\n      \"value\": \"playground\",\n      \"label\": \"Playground\"\n    },\n    {\n      \"value\": \"inClass\",\n      \"label\": \"InClass\"\n    }\n  ],\n  \"totalCompetitions\": 17,\n  \"competitions\": [\n    {\n      \"competitionId\": 6775,\n      \"competitionTitle\": \"Passenger Screening Algorithm Challenge\",\n      \"competitionDescription\": \"Improve the accuracy of the Department of Homeland Security\\\\u0027s threat recognition algorithms\",\n      \"competitionUrl\": \"/c/passenger-screening-algorithm-challenge\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/6775/logos/thumb76_76.png\",\n      \"deadline\": \"2017-12-16T07:59:00Z\",\n      \"totalTeams\": 164,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": 1500000,\n      \"rewardTypeName\": \"USD\",\n      \"organizationName\": \"Department of Homeland Security\",\n      \"organizationUrl\": \"dhs\",\n      \"hostSegment\": \"featured\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"$1,500,000\"\n    },\n    {\n      \"competitionId\": 6649,\n      \"competitionTitle\": \"Zillow Prize: Zillow’s Home Value Prediction (Zestimate)\",\n      \"competitionDescription\": \"Can you improve the algorithm that changed the world of real estate?\",\n      \"competitionUrl\": \"/c/zillow-prize-1\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/6649/logos/thumb76_76.png\",\n      \"deadline\": \"2018-01-17T15:59:00Z\",\n      \"totalTeams\": 2022,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": 1200000,\n      \"rewardTypeName\": \"USD\",\n      \"organizationName\": \"Zillow\",\n      \"organizationUrl\": \"zillow\",\n      \"hostSegment\": \"featured\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"$1,200,000\"\n    },\n    {\n      \"competitionId\": 6927,\n      \"competitionTitle\": \"Carvana Image Masking Challenge\",\n      \"competitionDescription\": \"Automatically identify the boundaries of the car in an image\",\n      \"competitionUrl\": \"/c/carvana-image-masking-challenge\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/6927/logos/thumb76_76.png\",\n      \"deadline\": \"2017-09-27T23:59:00Z\",\n      \"totalTeams\": 249,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": 25000,\n      \"rewardTypeName\": \"USD\",\n      \"organizationName\": \"Carvana\",\n      \"organizationUrl\": \"carvana\",\n      \"hostSegment\": \"featured\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"$25,000\"\n    },\n    {\n      \"competitionId\": 6644,\n      \"competitionTitle\": \"Instacart Market Basket Analysis\",\n      \"competitionDescription\": \"Which products will an Instacart consumer purchase again?\",\n      \"competitionUrl\": \"/c/instacart-market-basket-analysis\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/6644/logos/thumb76_76.png\",\n      \"deadline\": \"2017-08-14T23:59:00Z\",\n      \"totalTeams\": 2547,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": 25000,\n      \"rewardTypeName\": \"USD\",\n      \"organizationName\": \"Instacart\",\n      \"organizationUrl\": \"instacart\",\n      \"hostSegment\": \"featured\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"$25,000\"\n    },\n    {\n      \"competitionId\": 6768,\n      \"competitionTitle\": \"Web Traffic Time Series Forecasting\",\n      \"competitionDescription\": \"Forecast future traffic to Wikipedia pages\",\n      \"competitionUrl\": \"/c/web-traffic-time-series-forecasting\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/6768/logos/thumb76_76.png\",\n      \"deadline\": \"2017-09-10T23:59:00Z\",\n      \"totalTeams\": 530,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": 25000,\n      \"rewardTypeName\": \"USD\",\n      \"organizationName\": \"Google\",\n      \"organizationUrl\": \"google\",\n      \"hostSegment\": \"research\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"$25,000\"\n    },\n    {\n      \"competitionId\": 6841,\n      \"competitionTitle\": \"Personalized Medicine: Redefining Cancer Treatment\",\n      \"competitionDescription\": \"Predict the effect of Genetic Variants to enable Personalized Medicine\",\n      \"competitionUrl\": \"/c/msk-redefining-cancer-treatment\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/6841/logos/thumb76_76.png\",\n      \"deadline\": \"2017-10-03T06:59:00Z\",\n      \"totalTeams\": 705,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": 15000,\n      \"rewardTypeName\": \"USD\",\n      \"organizationName\": null,\n      \"organizationUrl\": null,\n      \"hostSegment\": \"research\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"$15,000\"\n    },\n    {\n      \"competitionId\": 6864,\n      \"competitionTitle\": \"NIPS 2017: Non-targeted Adversarial Attack\",\n      \"competitionDescription\": \"Imperceptibly transform images in ways that fool classification models\",\n      \"competitionUrl\": \"/c/nips-2017-non-targeted-adversarial-attack\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/6864/logos/thumb76_76.png\",\n      \"deadline\": \"2017-10-01T23:59:00Z\",\n      \"totalTeams\": 0,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": 0,\n      \"rewardTypeName\": \"Swag\",\n      \"organizationName\": \"Google Brain\",\n      \"organizationUrl\": \"google-brain\",\n      \"hostSegment\": \"research\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"Swag\"\n    },\n    {\n      \"competitionId\": 6866,\n      \"competitionTitle\": \"NIPS 2017: Targeted Adversarial Attack\",\n      \"competitionDescription\": \"Develop an adversarial attack that causes image classifiers to predict a specific target class\",\n      \"competitionUrl\": \"/c/nips-2017-targeted-adversarial-attack\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/6866/logos/thumb76_76.png\",\n      \"deadline\": \"2017-10-01T23:59:00Z\",\n      \"totalTeams\": 0,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": 0,\n      \"rewardTypeName\": \"Swag\",\n      \"organizationName\": \"Google Brain\",\n      \"organizationUrl\": \"google-brain\",\n      \"hostSegment\": \"research\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"Swag\"\n    },\n    {\n      \"competitionId\": 6867,\n      \"competitionTitle\": \"NIPS 2017: Defense Against Adversarial Attack\",\n      \"competitionDescription\": \"Create an image classifier that is robust to adversarial attacks\",\n      \"competitionUrl\": \"/c/nips-2017-defense-against-adversarial-attack\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/6867/logos/thumb76_76.png\",\n      \"deadline\": \"2017-10-01T23:59:00Z\",\n      \"totalTeams\": 0,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": 0,\n      \"rewardTypeName\": \"Swag\",\n      \"organizationName\": \"Google Brain\",\n      \"organizationUrl\": \"google-brain\",\n      \"hostSegment\": \"research\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"Swag\"\n    },\n    {\n      \"competitionId\": 6799,\n      \"competitionTitle\": \"ImageNet Object Localization Challenge\",\n      \"competitionDescription\": \"Identify the objects in images\",\n      \"competitionUrl\": \"/c/imagenet-object-localization-challenge\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/6796/logos/thumb76_76.png\",\n      \"deadline\": \"2029-12-31T07:00:00Z\",\n      \"totalTeams\": 0,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": null,\n      \"rewardTypeName\": \"Knowledge\",\n      \"organizationName\": \"ImageNet\",\n      \"organizationUrl\": \"image-net\",\n      \"hostSegment\": \"research\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"Knowledge\"\n    },\n    {\n      \"competitionId\": 6796,\n      \"competitionTitle\": \"ImageNet Object Detection Challenge\",\n      \"competitionDescription\": \"Identify and label everyday objects in images\",\n      \"competitionUrl\": \"/c/imagenet-object-detection-challenge\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/6796/logos/thumb76_76.png\",\n      \"deadline\": \"2029-12-31T07:00:00Z\",\n      \"totalTeams\": 0,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": null,\n      \"rewardTypeName\": \"Knowledge\",\n      \"organizationName\": \"ImageNet\",\n      \"organizationUrl\": \"image-net\",\n      \"hostSegment\": \"research\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"Knowledge\"\n    },\n    {\n      \"competitionId\": 6800,\n      \"competitionTitle\": \"ImageNet Object Detection from Video Challenge\",\n      \"competitionDescription\": \"Identify and label ordinary objects in videos\",\n      \"competitionUrl\": \"/c/imagenet-object-detection-from-video-challenge\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/6800/logos/thumb76_76.png\",\n      \"deadline\": \"2029-12-31T07:00:00Z\",\n      \"totalTeams\": 0,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": null,\n      \"rewardTypeName\": \"Knowledge\",\n      \"organizationName\": \"ImageNet\",\n      \"organizationUrl\": \"image-net\",\n      \"hostSegment\": \"research\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"Knowledge\"\n    },\n    {\n      \"competitionId\": 6960,\n      \"competitionTitle\": \"New York City Taxi Trip Duration\",\n      \"competitionDescription\": \"Share code and data to improve ride time predictions\",\n      \"competitionUrl\": \"/c/nyc-taxi-trip-duration\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/6960/logos/thumb76_76.png\",\n      \"deadline\": \"2017-09-15T23:59:00Z\",\n      \"totalTeams\": 546,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": 30000,\n      \"rewardTypeName\": \"USD\",\n      \"organizationName\": \"Kaggle\",\n      \"organizationUrl\": \"kaggle\",\n      \"hostSegment\": \"playground\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"$30,000\"\n    },\n    {\n      \"competitionId\": 6469,\n      \"competitionTitle\": \"Invasive Species Monitoring\",\n      \"competitionDescription\": \"Identify images of invasive hydrangea\",\n      \"competitionUrl\": \"/c/invasive-species-monitoring\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/6469/logos/thumb76_76.png\",\n      \"deadline\": \"2017-08-15T23:59:00Z\",\n      \"totalTeams\": 465,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": null,\n      \"rewardTypeName\": \"Knowledge\",\n      \"organizationName\": null,\n      \"organizationUrl\": null,\n      \"hostSegment\": \"playground\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"Knowledge\"\n    },\n    {\n      \"competitionId\": 3136,\n      \"competitionTitle\": \"Titanic: Machine Learning from Disaster\",\n      \"competitionDescription\": \"Start here! Predict survival on the Titanic and get familiar with ML basics\",\n      \"competitionUrl\": \"/c/titanic\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/3136/logos/thumb76_76.png\",\n      \"deadline\": \"2020-04-07T00:00:00Z\",\n      \"totalTeams\": 7367,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": 0,\n      \"rewardTypeName\": \"Knowledge\",\n      \"organizationName\": \"Kaggle\",\n      \"organizationUrl\": \"kaggle\",\n      \"hostSegment\": \"getting started\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"Knowledge\"\n    },\n    {\n      \"competitionId\": 5407,\n      \"competitionTitle\": \"House Prices: Advanced Regression Techniques\",\n      \"competitionDescription\": \"Predict sales prices and practice feature engineering, RFs, and gradient boosting\",\n      \"competitionUrl\": \"/c/house-prices-advanced-regression-techniques\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/5407/logos/thumb76_76.png\",\n      \"deadline\": \"2020-03-01T23:59:00Z\",\n      \"totalTeams\": 1613,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": 0,\n      \"rewardTypeName\": \"Knowledge\",\n      \"organizationName\": \"Kaggle\",\n      \"organizationUrl\": \"kaggle\",\n      \"hostSegment\": \"getting started\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"Knowledge\"\n    },\n    {\n      \"competitionId\": 3004,\n      \"competitionTitle\": \"Digit Recognizer\",\n      \"competitionDescription\": \"Learn computer vision fundamentals with the famous MNIST data\",\n      \"competitionUrl\": \"/c/digit-recognizer\",\n      \"thumbnailImageUrl\": \"https://kaggle2.blob.core.windows.net/competitions/kaggle/3004/logos/thumb76_76.png\",\n      \"deadline\": \"2020-01-07T00:00:00Z\",\n      \"totalTeams\": 1731,\n      \"totalKernels\": 0,\n      \"rewardQuantity\": 0,\n      \"rewardTypeName\": \"Knowledge\",\n      \"organizationName\": \"Kaggle\",\n      \"organizationUrl\": \"kaggle\",\n      \"hostSegment\": \"getting started\",\n      \"isLimited\": false,\n      \"isPrivate\": false,\n      \"isInClass\": false,\n      \"userHasEntered\": false,\n      \"rewardDisplay\": \"Knowledge\"\n    }\n  ],\n  \"values\": {\n    \"page\": 1,\n    \"group\": \"active\",\n    \"sortBy\": \"prize\",\n    \"segment\": null,\n    \"search\": \"\"\n  },\n  \"banner\": {\n    \"bannerName\": \"competitions\",\n    \"isVisible\": true,\n    \"dismissBannerUrl\": \"/account/dismiss-banner?bannerName=competitions\"\n  },\n  \"canSeeNewWorldInClassCreation\": false\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsdq%2Fkaggle-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsdq%2Fkaggle-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsdq%2Fkaggle-crawler/lists"}