{"id":21420875,"url":"https://github.com/lord-turmoil/beyondsearchengine","last_synced_at":"2026-05-10T00:43:13.530Z","repository":{"id":214194789,"uuid":"735930196","full_name":"Lord-Turmoil/BeyondSearchEngine","owner":"Lord-Turmoil","description":"Search service for Beyond Scholar of BUAA 2023 System Analysis and Design course","archived":false,"fork":false,"pushed_at":"2024-01-02T13:22:53.000Z","size":271,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-23T06:32:12.158Z","etag":null,"topics":["asp-net-core","csharp","elasticsearch","openalex","redis"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Lord-Turmoil.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-26T13:44:11.000Z","updated_at":"2024-04-13T14:58:34.000Z","dependencies_parsed_at":"2024-01-02T14:41:16.943Z","dependency_job_id":null,"html_url":"https://github.com/Lord-Turmoil/BeyondSearchEngine","commit_stats":null,"previous_names":["lord-turmoil/beyondsearchengine"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lord-Turmoil%2FBeyondSearchEngine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lord-Turmoil%2FBeyondSearchEngine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lord-Turmoil%2FBeyondSearchEngine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lord-Turmoil%2FBeyondSearchEngine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Lord-Turmoil","download_url":"https://codeload.github.com/Lord-Turmoil/BeyondSearchEngine/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243926072,"owners_count":20369910,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asp-net-core","csharp","elasticsearch","openalex","redis"],"created_at":"2024-11-22T20:23:01.806Z","updated_at":"2026-05-10T00:43:13.470Z","avatar_url":"https://github.com/Lord-Turmoil.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Beyond Search Engine\n\n\u003e Copyright (C) Beyond 2023, Tony's Studio 2023\n\u003e\n\u003e Relics of BUAA 2023 Autumn *System Analysis and Design* course.\n\n---\n\n## Overview\n\nIt is the search engine API service for Beyond Scholar.\n\nPowered by [ASP.NET Core](https://dotnet.microsoft.com/en-us/), Beyond Search Engine is a cross-platform, high-performance, cloud-ready Web API. With MVC pattern, everything became so clear and organized, not to mention C# itself is an opiniated language.\n\nWe get all data from [OpenAlex](https://openalex.org/), and indexed them to our database - [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html). For our server, we have one 8 core 16 GB master server and two 2 core 4 GB slave servers, with ASP.NET Core project deployed only on master.\n\n---\n\n## Features\n\n- Fast response with Elasticsearch integration.\n- Hot update on hundred millions of data.\n- Cache support with Redis.\n\n---\n\n## Pitfalls\n\n- Failed to index some of OpenAlex's data, as some fields are `null` or missing.\n- Huge amount of data took too much time to index before optimizations applied.\n- The response is not consistent. Some DTO are crafted ugly and messy.\n\n---\n\n## Summary\n\nIt was an amazing experience working with ASP.NET Core and Elasticsearch. Here are some trails left behind.\n\nAt the very beginning, you need to [Setup Elasticsearch With Kibana](https://www.tonys-studio.top/posts/Setup-Elasticsearch-with-Kibana/). Then, if you need to move your data from a relational database, see how to [Ingest Data From MySQL to Elasticsearch](https://www.tonys-studio.top/posts/Ingest-Data-from-MySQL-to-Elasticsearch/). Next, as your first step towards cross-platform development with .NET, you can skim over [Cross-Platform Development With .NET Core](https://www.tonys-studio.top/posts/Cross-platform-Development-with-NET-Core/). At last, [Empower ASP.NET Core With Elasticsearch](https://www.tonys-studio.top/posts/Empower-ASP-NET-Core-with-Elasticsearch/), you can.\n\n\u003e If you encountered some performance issue when indexing large amount of data, you can reach out to [Bulk Task Optimization in C#](https://www.tonys-studio.top/posts/Bulk-Task-Optimization-in-C/).\n\n---\n\n## Epilogue\n\nThere's another backend project written in Python with Django. I have to say, \"ASP.NET Core superior, Django inferior.\" ASP.NET Core gains land slide advantages against Django, whether in performance, deploy complexity, or productivity.\n\nASP.NET Core is cloud-ready, yet Django is not. Thus all problems of the backend deployment failure comes from Django project, and Django is much harder to configure than the other.\n\nAs for speed, Python takes no chance. Even with muti-process of uWSGI and clusters, Django is easily overwhelmed by 1000 concurrent requests. While ASP.NET Core hosted by Kestrel can take more than 10000 requests with no failure.\n\n\u003cp style=\"text-align:center\"\u003e\u003ci\u003eLife is short, I use C#.\u003c/i\u003e\u003c/p\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flord-turmoil%2Fbeyondsearchengine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flord-turmoil%2Fbeyondsearchengine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flord-turmoil%2Fbeyondsearchengine/lists"}