{"id":20478647,"url":"https://github.com/do-me/copernicus-services-semantic-search","last_synced_at":"2025-06-14T23:33:28.459Z","repository":{"id":202804328,"uuid":"708167675","full_name":"do-me/copernicus-services-semantic-search","owner":"do-me","description":"A basic semantic search app based on 834 entries from Copernicus Services Catalogue","archived":false,"fork":false,"pushed_at":"2024-10-17T11:35:54.000Z","size":9090,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-13T13:15:36.724Z","etag":null,"topics":["copernicus","semantic-search","semanticsearch","transformers"],"latest_commit_sha":null,"homepage":"https://do-me.github.io/copernicus-services-semantic-search/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/do-me.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-21T18:14:37.000Z","updated_at":"2025-01-29T10:42:11.000Z","dependencies_parsed_at":"2024-10-20T13:03:41.684Z","dependency_job_id":null,"html_url":"https://github.com/do-me/copernicus-services-semantic-search","commit_stats":null,"previous_names":["do-me/copernicus-services-semantic-search"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/do-me%2Fcopernicus-services-semantic-search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/do-me%2Fcopernicus-services-semantic-search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/do-me%2Fcopernicus-services-semantic-search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/do-me%2Fcopernicus-services-semantic-search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/do-me","download_url":"https://codeload.github.com/do-me/copernicus-services-semantic-search/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248717238,"owners_count":21150389,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["copernicus","semantic-search","semanticsearch","transformers"],"created_at":"2024-11-15T15:38:34.796Z","updated_at":"2025-04-13T13:15:42.626Z","avatar_url":"https://github.com/do-me.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Copernicus Services Semantic Search\n\n## [App here](https://do-me.github.io/copernicus-services-semantic-search/)\n\n## [Tutorial here](https://geo.rocks/post/semantic-search-tutorial/) with lots of tips\n\nA basic semantic search app based on 834 entries from [Copernicus Services Catalogue](https://www.copernicus.eu/en/accessing-data-where-and-how/copernicus-services-catalogue) chunked and indexed (mean embedding of all content chunks) in a ~2.4MB gzipped json with [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2). Enter any query and hit submit or enter. App loads ~27Mb of resources of data and scripts. The ML model runs entirely in the browser thanks to [transformers.js](https://github.com/xenova/transformers.js).\n\n![](copernicus-services-semantic-search-interface-dark.png)\n\n## Advanced search \n\nIf you'd like to search within the result's content, consider installing the Chrome extension of [SemanticFinder](https://chrome.google.com/webstore/detail/semanticfinder/ddmgffoffelnhnonpoiblaoboaeieejl), [GitHub repo](https://github.com/do-me/SemanticFinder).\n\n![](semantic-finder-results.png)\n\nIt finds the most relevant sections to your query in the actual content of the results by performing semantic search on the fly.\n\n## Data mining tutorial \n\nThe process of creating the data dump includingcan be repeated with the included [Jupyter Notebook](copernicus_services_miner.ipynb). It includes the whole processing pipeline:\n- data mining with requests and beautifulsoup\n- preprocessing in pandas\n- chunking the document text in smaller paragraphs of the right size for the ML model\n- creating embeddings for each chunk\n- calculating the mean embedding for each document\n- saving as gzipped json (small file size \u0026 easy and fast to read in js with pako.js)\n\nYou can re-run the process for updates (if you do so, please open a pull request for this repo or write so I can keep the data dump updated) or use other indexing models like the current [MTEB](https://huggingface.co/spaces/mteb/leaderboard) leaders of the bge or gte family. You could also use a multilingual model to perform search queries in other languages than English. The current dump holds 834 entries from 21 October 2023. \n\n![](copernicus-services-df.png)\n\n## Export all 834 entries for large LLM context \n- Just use this plain text file for copy \u0026 paste: https://raw.githubusercontent.com/do-me/copernicus-services-semantic-search/main/copernicus-services.txt.\n- In Gemini (https://aistudio.google.com), this text counts roughly 1.5 Mio tokens, so you can still add large prompts within the 2 Mio context window.\n\n\u003cdetails\u003e\n  \u003csummary\u003eHow to create this text file with JS\u003c/summary\u003e\n- Run a search and display all results (enter 1000 as limit). The results are ordered by similarity.\n- Open the browser console with F12\n- Use this JS and execute it: \n\n```javascript\n document.querySelectorAll('.position-relative').forEach(function(element) {\n    // Remove each element from the DOM\n    element.remove();\n  });\n\nfunction tableToText() {\n            // Select the table\n            const table = document.getElementById('results-table');\n            let resultText = '';\n\n            // Loop through each row\n            for (let row of table.rows) {\n                // Loop through each cell in the row, excluding the \"Similarity\" column (index 2)\n                for (let i = 0; i \u003c row.cells.length; i++) {\n                    if (i === 5) continue; // Skip Similarity\n\n                    const cell = row.cells[i];\n\n                    // For the first two columns, check if there are anchor tags\n                    if ((i === 1) | (i===2)) {\n                        const link = cell.querySelector('a');\n                        if (link) {\n                            // Use the href attribute of the anchor tag\n                            resultText += link.href + '\\n';\n                        } else {\n                            resultText += cell.innerText + '\\n';  // Fallback to normal text\n                        }\n                    } else {\n                        resultText += cell.innerText + '\\n';  // For other columns, use innerText\n                    }\n                }\n                resultText += '\\n\\n';  // Add two line breaks between rows\n            }\n\n            console.log(resultText);  // Log the result to the console\n        }\n\n// Call the function to convert table to text and log it\ntableToText();\n```\n- Copy the output with the copy button (e.g. in Chrome or select the whole text)\n![image](https://github.com/user-attachments/assets/c970ae68-5bca-46fd-b2ee-c228a77ee881)\n\u003c/details\u003e\n\n## Qdrant Instance \nI provide a public Qdrant instance over Qdrant Cloud that you can access to create nice plots for the collection via dimensionality reduction or graph-based links.\nAccess the collection with the API key `A-KWBxWl_8G3cnXv3MlpCThEDTdS6FYnTzn-h9k9TE95f5cvMUAGbQ` under: \n\nhttps://8f35f088-fc2e-426e-92a3-4f4f26f64812.europe-west3-0.gcp.cloud.qdrant.io:6333/dashboard#/collections/Copernicus_Services/\n\n### Scatterplot\nClick on `visualize` or access [this link](https://8f35f088-fc2e-426e-92a3-4f4f26f64812.europe-west3-0.gcp.cloud.qdrant.io:6333/dashboard#/collections/Copernicus_Services/visualize). Then enter this code an hit `RUN`:\n```\n{\n  \"limit\": 5000,\n  \"color_by\": \"Copernicus_Service\"\n}\n```\n![image](https://github.com/user-attachments/assets/b7b8b118-5bb9-43df-b6d2-1c9d5ef98549)\n\n### Graph\nClick on `graph` or access [this link](https://8f35f088-fc2e-426e-92a3-4f4f26f64812.europe-west3-0.gcp.cloud.qdrant.io:6333/dashboard#/collections/Copernicus_Services/graph). Then hit `RUN`.\n\n![image](https://github.com/user-attachments/assets/108a8511-e081-47fe-943f-f56635dd99d2)\n\n### Download Qdrant Snapshot\nYou can download the snapshot [here](https://8f35f088-fc2e-426e-92a3-4f4f26f64812.europe-west3-0.gcp.cloud.qdrant.io:6333/dashboard#/collections/Copernicus_Services#snapshots) and run it locally too.\n\nIf you like this project, ⭐ the repo or give a shoutout on social media. Let me know if you build something cool with it!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdo-me%2Fcopernicus-services-semantic-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdo-me%2Fcopernicus-services-semantic-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdo-me%2Fcopernicus-services-semantic-search/lists"}