{"id":20065681,"url":"https://github.com/collaborative-ai/coldata","last_synced_at":"2025-03-11T05:47:04.363Z","repository":{"id":270819575,"uuid":"642649399","full_name":"Collaborative-AI/coldata","owner":"Collaborative-AI","description":"Dataset Search Engine with Vector Database","archived":false,"fork":false,"pushed_at":"2025-02-28T11:59:19.000Z","size":28349,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-28T18:18:46.814Z","etag":null,"topics":["beautifulsoup","dataset","milvus","mongodb","search-engine","vector-database"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Collaborative-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-19T03:32:19.000Z","updated_at":"2025-02-28T11:59:22.000Z","dependencies_parsed_at":"2025-02-06T06:26:50.913Z","dependency_job_id":"ad0a4d39-93c6-43a1-b5d9-30e73818062a","html_url":"https://github.com/Collaborative-AI/coldata","commit_stats":null,"previous_names":["collaborative-ai/coldata"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Collaborative-AI%2Fcoldata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Collaborative-AI%2Fcoldata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Collaborative-AI%2Fcoldata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Collaborative-AI%2Fcoldata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Collaborative-AI","download_url":"https://codeload.github.com/Collaborative-AI/coldata/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242980786,"owners_count":20216285,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beautifulsoup","dataset","milvus","mongodb","search-engine","vector-database"],"created_at":"2024-11-13T13:52:02.361Z","updated_at":"2025-03-11T05:47:04.355Z","avatar_url":"https://github.com/Collaborative-AI.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ColData\n_**Col**laborative **Data**set Search Engine with Vector Database_\n\n**coldata** is an open-source dataset search engine designed to help researchers, data scientists, developers, and the broader community **collaboratively** discover, share, and access relevant datasets across a variety of sources.  \nThe engine crawls metadata from popular dataset hosting platforms with **BeautifulSoup**, stores it in **MongoDB**, and transforms it into a vector-based database using **MilvuDB** for enhanced search and retrieval.\n\n\n## Features\n\n- **Multi-source Crawling**: We gather metadata from major dataset repositories.\n  \n- **Vector-based Search**: The metadata is converted into vector embeddings using the language model, enabling powerful semantic search capabilities.\n\n- **Interface**: With the help of **Gradio**, we offer a simple demo to interact with the engine and quickly locate datasets.\n\n- **Scalable**: The underlying database, **MongoDB**, and vector engine, **MilvuDB**, ensure that the system scales as the number of crawled datasets grows.\n\n## Datasets\n\nWe currently support crawling and indexing datasets from the following sources:\n\n| Dataset Name                          | Number of Datasets    | Completed  |\n|---------------------------------------|-----------------------|------------|\n| **UCI**                               | 675                   | ✅         |\n| **Kaggle**                            | 40,000+               | ✅         |\n| **Registry of Open Data on AWS**      | 496                   | ✅         |\n| **Papers With Code**                 | 8,966                 | ✅         |\n| **Figshare**                          | 1,856,206             |          |\n| **Mendeley Data**                     | 1,307,514             |          |\n| **Hugging Face Datasets**             | 88,179                |         |\n| **Zenodo**                            | 234,972               |          |\n| **IEEE Dataport**                     | 1,170                 |         |\n| **Open Data Lab**                     | 6,432                 |          |\n| **Roboflow Universe**                 | 200,000+              |          |\n\n## Installation\n\nClone the repository:\n\n```bash\ngit clone https://github.com/yourusername/coldata.git\ncd coldata\n```\n\nInstall dependencies:\n\n```bash\npip install -r requirements.txt\n```\n\n\n## Configuration\n\nThe system can be customized via the `config.yml` file, where you can configure hyperparameters for both MongoDB and MilvuDB.\n\n\n## Quick Start\n\n1. **Start MongoDB**: \n\n   Run `start_mongo.sh` to start a local MongoDB instance.\n\n   ```bash\n   ./start_mongo.sh\n   ```\n\n2. **Start Milvus DB**:\n\n   Run `manage_milvus.sh` to start the Milvus vector database.\n\n   ```bash\n   ./manage_milvus.sh\n   ```\n\n3. **Run Scheduler**: \n\n   Set up the scheduler to crawl datasets at a specified interval by running:\n\n   ```bash\n   python scheduler.py\n   ```\n\n4. **Demo Interface**: \n\n   To quickly test the dataset search, you can use the Gradio-based demo:\n\n   ```bash\n   python demo.py\n   ```\n   \n## How to Contribute\n\nWe welcome contributions to improve **coldata**. If you have ideas for new features, bug fixes, or dataset sources to include, please feel free to open an issue or submit a pull request.\n\n\n## License\n\nThis project is licensed under the MIT License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcollaborative-ai%2Fcoldata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcollaborative-ai%2Fcoldata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcollaborative-ai%2Fcoldata/lists"}