{"id":22820037,"url":"https://github.com/agora-lab-ai/hpsa","last_synced_at":"2025-04-23T00:23:36.382Z","repository":{"id":266803194,"uuid":"899403880","full_name":"Agora-Lab-AI/HPSA","owner":"Agora-Lab-AI","description":"This paper introduces a theoretical framework for a revolutionary search algorithm that achieves unprecedented speed through a combination of probabilistic data structures, intelligent data partitioning, and a novel hierarchical index structure.","archived":false,"fork":false,"pushed_at":"2025-03-03T14:50:20.000Z","size":2465,"stargazers_count":5,"open_issues_count":4,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-29T20:23:28.726Z","etag":null,"topics":["bfs","dfs","search","search-algorithm","search-engine-optimization","tree-search"],"latest_commit_sha":null,"homepage":"https://agoralab.xyz","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Agora-Lab-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["kyegomez"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2024-12-06T07:39:16.000Z","updated_at":"2025-02-01T19:03:20.000Z","dependencies_parsed_at":"2024-12-06T08:26:51.902Z","dependency_job_id":"a86b58ad-cc8e-430e-b3f4-2be5d266faff","html_url":"https://github.com/Agora-Lab-AI/HPSA","commit_stats":null,"previous_names":["agora-lab-ai/hpsa"],"tags_count":0,"template":false,"template_full_name":"kyegomez/Python-Package-Template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Agora-Lab-AI%2FHPSA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Agora-Lab-AI%2FHPSA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Agora-Lab-AI%2FHPSA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Agora-Lab-AI%2FHPSA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Agora-Lab-AI","download_url":"https://codeload.github.com/Agora-Lab-AI/HPSA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250344963,"owners_count":21415228,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bfs","dfs","search","search-algorithm","search-engine-optimization","tree-search"],"created_at":"2024-12-12T15:16:03.564Z","updated_at":"2025-04-23T00:23:36.354Z","avatar_url":"https://github.com/Agora-Lab-AI.png","language":"Python","funding_links":["https://github.com/sponsors/kyegomez"],"categories":[],"sub_categories":[],"readme":"# Hierarchical Probabilistic Search Architecture: A Novel Framework for Sub-Linear Time Information Retrieval\n\n\n[![Join our Discord](https://img.shields.io/badge/Discord-Join%20our%20server-5865F2?style=for-the-badge\u0026logo=discord\u0026logoColor=white)](https://discord.gg/agora-999382051935506503) [![Subscribe on YouTube](https://img.shields.io/badge/YouTube-Subscribe-red?style=for-the-badge\u0026logo=youtube\u0026logoColor=white)](https://www.youtube.com/@kyegomez3242) [![Connect on LinkedIn](https://img.shields.io/badge/LinkedIn-Connect-blue?style=for-the-badge\u0026logo=linkedin\u0026logoColor=white)](https://www.linkedin.com/in/kye-g-38759a207/) [![Follow on X.com](https://img.shields.io/badge/X.com-Follow-1DA1F2?style=for-the-badge\u0026logo=x\u0026logoColor=white)](https://x.com/kyegomezb)\n\n\n## Abstract\n\nThis paper introduces a theoretical framework for a revolutionary search algorithm that achieves unprecedented speed through a combination of probabilistic data structures, intelligent data partitioning, and a novel hierarchical index structure. Our proposed architecture, HPSA (Hierarchical Probabilistic Search Architecture), achieves theoretical search times approaching O(log log n) in optimal conditions by leveraging advanced probabilistic filters and a new approach to data organization. We present both the theoretical foundation and a detailed implementation framework for this new approach to information retrieval.\n\nTraditional search engines operate with time complexities ranging from O(log n) to O(n), depending on the implementation and index structure. While considerable improvements have been made through various optimization techniques, fundamental mathematical limits have constrained theoretical advances in search speed. This paper presents a novel approach that challenges these traditional boundaries through a combination of probabilistic data structures and a new hierarchical architecture.\n\nCurrent search algorithms primarily rely on inverted indices, which maintain a mapping between terms and their locations in the document corpus. While efficient, these structures still require significant time for query processing, especially when handling complex boolean queries or performing relevance ranking. The theoretical lower bound for traditional search approaches has been well-established at Ω(log n) for most practical implementations.\n\nThe exponential growth of digital information demands new approaches to information retrieval that can scale beyond traditional limitations. Our work is motivated by the observation that document collections often exhibit natural clustering and hierarchical relationships that can be exploited for faster search operations.\n\n\n[PAPER LINK](hpsa.pdf)\n\n\n## Usage\n\n```python\n\n\ndef main():\n    \"\"\"Main benchmark execution\"\"\"\n    \n    # Test parameters\n    NUM_DOCUMENTS = 50000\n    VOCAB_SIZE = 100_000\n    DOC_LENGTH = 100\n    NUM_QUERIES = 10_000\n    TERMS_PER_QUERY = 3\n    \n    print(\"Generating test dataset...\")\n    documents = Benchmark.generate_dataset(NUM_DOCUMENTS, VOCAB_SIZE, DOC_LENGTH)\n    \n    print(\"Initializing HPSA...\")\n    hpsa = HPSA(NUM_DOCUMENTS)\n    \n    print(\"Indexing documents...\")\n    start_time = time.time()\n    for doc in documents:\n        hpsa.index_document(doc)\n    index_time = time.time() - start_time\n    print(f\"Indexing completed in {index_time:.2f} seconds\")\n    \n    print(\"Generating test queries...\")\n    queries = Benchmark.generate_queries(NUM_QUERIES, VOCAB_SIZE, TERMS_PER_QUERY)\n    \n    print(\"Running benchmark...\")\n    results = Benchmark.run_benchmark(hpsa, queries)\n    \n    print(\"\\nBenchmark Results:\")\n    print(f\"Queries per second: {results['queries_per_second']:.2f}\")\n    print(f\"Mean response time: {results['mean_response_time']*1000:.2f}ms\")\n    print(f\"95th percentile response time: {results['p95_response_time']*1000:.2f}ms\")\n    print(f\"99th percentile response time: {results['p99_response_time']*1000:.2f}ms\")\n\nif __name__ == \"__main__\":\n    main()\n\n\n```\n\n\n## Output\n\n```txt\nGenerating test dataset...\nInitializing HPSA...\nIndexing documents...\nIndexing completed in 183.26 seconds\nGenerating test queries...\nRunning benchmark...\n\nBenchmark Results:\nQueries per second: 3973.11\nMean response time: 0.25ms\n95th percentile response time: 0.47ms\n99th percentile response time: 5.94ms\n\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagora-lab-ai%2Fhpsa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fagora-lab-ai%2Fhpsa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagora-lab-ai%2Fhpsa/lists"}