{"id":13706358,"url":"https://github.com/bytefish/ElasticsearchCodeSearch","last_synced_at":"2025-05-05T20:30:50.695Z","repository":{"id":176903768,"uuid":"657475057","full_name":"bytefish/ElasticsearchCodeSearch","owner":"bytefish","description":"Index and Search Git Repositories using Elasticsearch","archived":false,"fork":false,"pushed_at":"2024-07-17T09:32:17.000Z","size":2396,"stargazers_count":18,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-11-13T14:41:38.146Z","etag":null,"topics":["aspnetcore","blazor","dotnet","elasticsearch","powershell"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bytefish.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-23T06:28:13.000Z","updated_at":"2024-10-10T16:04:01.000Z","dependencies_parsed_at":"2024-01-10T09:28:31.381Z","dependency_job_id":"75e8f4fb-9529-40e6-a599-a3580a634b87","html_url":"https://github.com/bytefish/ElasticsearchCodeSearch","commit_stats":null,"previous_names":["bytefish/elasticsearchcodesearch"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytefish%2FElasticsearchCodeSearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytefish%2FElasticsearchCodeSearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytefish%2FElasticsearchCodeSearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytefish%2FElasticsearchCodeSearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bytefish","download_url":"https://codeload.github.com/bytefish/ElasticsearchCodeSearch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252570893,"owners_count":21769743,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aspnetcore","blazor","dotnet","elasticsearch","powershell"],"created_at":"2024-08-02T22:00:54.512Z","updated_at":"2025-05-05T20:30:50.118Z","avatar_url":"https://github.com/bytefish.png","language":"C#","funding_links":[],"categories":["C# #"],"sub_categories":[],"readme":"# Elasticsearch Code Search #\n\nThis repository implements a Code Search Engine using ASP.NET Core and Elasticsearch. You can use it \nto index repositories from GitHub, Codeberg, GitLab and other sources. It comes with a Blazor \nFrontend built upon FluentUI Components.\n\nThere is a Page to get an overview for your Elasticsearch index:\n\n\u003ca href=\"https://raw.githubusercontent.com/bytefish/ElasticsearchCodeSearch/main/doc/img/ElasticsearchCodeSearch_SearchClusterOverview.jpg\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/bytefish/ElasticsearchCodeSearch/main/doc/img/ElasticsearchCodeSearch_SearchClusterOverview.jpg\" alt=\"The final Code Search with the Blazor Frontend\" width=\"100%\" /\u003e\n\u003c/a\u003e\n\nAnd there's a page to search for code using the Elasticsearch index:\n\n\u003ca href=\"https://raw.githubusercontent.com/bytefish/ElasticsearchCodeSearch/main/doc/img/ElasticsearchCodeSearch_SearchCode.jpg\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/bytefish/ElasticsearchCodeSearch/main/doc/img/ElasticsearchCodeSearch_SearchCode.jpg\" alt=\"The final Code Search with the Blazor Frontend\" width=\"100%\" /\u003e\n\u003c/a\u003e\n\nThere are more pages, like:\n\n* \u003ca href=\"https://raw.githubusercontent.com/bytefish/ElasticsearchCodeSearch/main/doc/img/ElasticsearchCodeSearch_IndexGitRepository.jpg\"\u003ePage for indexing a Git Repository\u003c/a\u003e\n* \u003ca href=\"https://raw.githubusercontent.com/bytefish/ElasticsearchCodeSearch/main/doc/img/ElasticsearchCodeSearch_IndexGitHubRepository.jpg\"\u003ePage for indexing a GitHub Repository\u003c/a\u003e\n* \u003ca href=\"https://raw.githubusercontent.com/bytefish/ElasticsearchCodeSearch/main/doc/img/ElasticsearchCodeSearch_DebuggingManageIndex.jpg\"\u003ePage for Managing the Elasticsearch Index\u003c/a\u003e\n\n## Getting Started ##\n\nGetting started is as simple as cloning this repository and running the following command:\n\n```\ndocker compose --profile dev up\n```\n\nYou can then navigate to `https://localhost:5001` and start searching and indexing Git Repositories.\n\n### Docker Profiles ###\n\nThere are 4 Docker Profiles:\n\n* `dev`\n    * Starts all 3 services.\n* `elastic`\n    * Starts the Elasticsearch Server.\n* `api`\n    * Starts the ElasticsearchCodeSearch API\n* `web`\n    * Starts the ElasticsearchCodeSearch Blazor App\n\nTo only run the Elasticsearch Server you would pass the `elastic` profile:\n\n```\ndocker compose --profile elastic up\n```\n\nTo run the Elasticsearch Server and the Code Search API you would pass the `elastic` and `api` profiles:\n\n```\ndocker compose --profile elastic --profile api up\n```\n\nThis is super useful, if you want to locally debug the ASP.NET Core services.\n\n## How it was done ##\n\n### Configuring HTTPS for Elasticsearch ###\n\nTo secure the HTTPS communication with Elasticsearch we need to generate a certificate \nfirst. The easiest way to do this is to use the `elasticsearch-certutil` command line \ntool.\n\nWe start by creating a Certificate Authority (CA):\n\n```powershell\nelasticsearch-certutil ca --silent --pem -out ./elastic-cert/ca.zip\n```\n\nIn the `docker/elasticsearch/elastic-cert` we will now find a `ca.zip` and unzip it.\n\nNext we create a `instances.yml`file for the local Elasticsearch instance `es01`:\n\n```yaml\ninstances:\n  - name: es01\n    dns:\n      - es01\n      - localhost\n    ip:\n        - 127.0.0.1\n```\n\nWe can then pass the `instances.yml` to the `elasticsearch-certutil` command line tool and \ncreate a certificate using our previously generated CA certificate:\n\n```powershell\nelasticsearch-certutil cert --silent --pem -out ./certs.zip --in ./instances.yml --ca-cert ./elastic-cert/ca/ca.crt --ca-key ./elastic-cert/ca/ca.key\n```\n\nIn an `.env` file we are defining the Environment variables as:\n\n```ini\nELASTIC_HOSTNAME=es01\nELASTIC_USERNAME=elastic\nELASTIC_PASSWORD=secret\nELASTIC_PORT=9200\nELASTIC_SECURITY=true\nELASTIC_SCHEME=https\nELASTIC_VERSION=8.14.1\n```\n\nIn the `docker-compose.yml` we then configure `elasticsearch` with our generated certificates as:\n\n```yaml\nservices:\n  elasticsearch:\n    image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION:-8.14.1}\n    container_name: ${ELASTIC_HOSTNAME:-es01}\n    hostname: ${ELASTIC_HOSTNAME:-es01}\n    restart: ${RESTART_MODE:-unless-stopped}\n    healthcheck:\n      test: [\"CMD-SHELL\", \"curl --user ${ELASTIC_USER:-elastic}:${ELASTIC_PASSWORD:-secret} --silent --fail https://localhost:9200/_cluster/health -k || exit 1\" ]\n      interval: 10s\n      timeout: 5s\n      retries: 3\n      start_period: 30s\n    env_file:\n      - ./.env\n    environment:\n      - node.name=es01\n      - discovery.type=single-node\n      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD:-secret}\n      - xpack.security.enabled=${ELASTIC_SECURITY:-true}\n      - xpack.security.http.ssl.enabled=true\n      - xpack.security.http.ssl.verification_mode=none\n      - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/cert/es01.key\n      - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/cert/es01.crt\n      - xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/cert/ca/ca.crt\n      - xpack.security.transport.ssl.enabled=${ELASTIC_SECURITY:-true}\n      - xpack.security.transport.ssl.verification_mode=none\n      - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/cert/ca/ca.crt\n      - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/cert/es01.crt\n      - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/cert/es01.key\n    ulimits:\n      memlock:\n        soft: -1\n        hard: -1\n    volumes:\n      - ./elasticsearch/elastic-data:/usr/share/elasticsearch/data\n      - ./elasticsearch/elastic-cert:/usr/share/elasticsearch/config/cert\n    ports:\n      - \"9200:9200\"\n      - \"9300:9300\"\n```\n\nWe can now verify, if Elasticsearch starts correctly by using `curl`:\n\n```Powershell\nC:\\Users\\philipp\u003ecurl -k https://127.0.0.1:9200  -u elastic:secret\n{\n  \"name\" : \"es01\",\n  \"cluster_name\" : \"docker-cluster\",\n  \"cluster_uuid\" : \"2pvSQcC-Tnu-cbQuc1AONw\",\n  \"version\" : {\n    \"number\" : \"8.14.1\",\n    \"build_flavor\" : \"default\",\n    \"build_type\" : \"docker\",\n    \"build_hash\" : \"93a57a1a76f556d8aee6a90d1a95b06187501310\",\n    \"build_date\" : \"2024-06-10T23:35:17.114581191Z\",\n    \"build_snapshot\" : false,\n    \"lucene_version\" : \"9.10.0\",\n    \"minimum_wire_compatibility_version\" : \"7.17.0\",\n    \"minimum_index_compatibility_version\" : \"7.0.0\"\n  },\n  \"tagline\" : \"You Know, for Search\"\n}\n```\n\nAnd finally we calculate the Certificate Fingerprint using:\n\n```powershell\nopenssl.exe x509 -fingerprint -sha256 -in .\\es01.crt\n```\n\nAnd get our Certificate Fingerprint as:\n\n```\n31a63ffca5275df7ea7d6fc7e92b42cfa774a0feed7d7fa8488c5e46ea9ade3f\n```\n\nAnd that's it.\n\nIn the .NET application I can then create the `ElasticsearchClient` like this:\n\n```csharp\n// Licensed under the MIT license. See LICENSE file in the project root for full license information.\n\n// ...\n\nnamespace ElasticsearchCodeSearch.Shared.Elasticsearch\n{\n    public class ElasticCodeSearchClient\n    {\n    \n        // ...\n    \n        public virtual ElasticsearchClient CreateClient(ElasticCodeSearchOptions options)\n        {\n            var settings = new ElasticsearchClientSettings(new Uri(options.Uri))\n                .CertificateFingerprint(options.CertificateFingerprint)\n                .Authentication(new BasicAuthentication(options.Username, options.Password));\n\n            return new ElasticsearchClient(settings);\n        }\n        \n        // ...\n    }\n}\n```\n\n### Setting up HTTPS Communication in ASP.NET Core with Docker ###\n\nWe will create and trust the self-signed certificates with the following command:\n\n```powershell\ndotnet dev-certs https -ep $env:USERPROFILE\\.aspnet\\https\\aspnetapp.pfx -p SuperStrongPassword\ndotnet dev-certs https --trust\n```\n\nNext is setting the GH Token for the Client:\n\n```powershell\ndotnet user-secrets set \"GitHubClient:AccessToken\" \"\u003cYour GH Token Here\u003e\"\n```\n\nIf you need to manage secrets in Production, you should consider using Docker Secrets, Azure KeyVault, Consul, ... or any other safe secret store.\n\n## Contributions ##\n\nThis is an Open Source project, feel free to contribute!\n\n## Articles ##\n\nThe Search Engine has been introduced in three articles so far: \n\n* [Implementing a Code Search: Elasticsearch and ASP.NET Core Backend (Part 1)](https://www.bytefish.de/blog/elasticsearch_code_search_part1_backend_elasticsearch.html)\n* [Implementing a Code Search: Indexing Git Repositories using PowerShell (Part 2)](https://www.bytefish.de/blog/elasticsearch_code_search_part2_indexer_powershell.html)\n* [Implementing a Code Search: A Frontend with ASP.NET Core Blazor (Part 3)](https://www.bytefish.de/blog/elasticsearch_code_search_part3_frontend_blazor.html)\n\n## License ##\n\nThe code in this repository is licensed under MIT License.\n\nThe Pagination component and project structure have been taken from the Fluent UI project.\n\n```\nMIT License\n\nCopyright (c) Microsoft Corporation. All rights reserved.\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbytefish%2FElasticsearchCodeSearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbytefish%2FElasticsearchCodeSearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbytefish%2FElasticsearchCodeSearch/lists"}