{"id":24118940,"url":"https://github.com/cubesugar/minisearchengine","last_synced_at":"2026-07-03T20:32:23.680Z","repository":{"id":9771646,"uuid":"11742217","full_name":"CubeSugar/MiniSearchEngine","owner":"CubeSugar","description":"mini search engine","archived":false,"fork":false,"pushed_at":"2013-08-15T15:49:00.000Z","size":660,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-26T20:48:59.482Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CubeSugar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-07-29T14:58:33.000Z","updated_at":"2013-08-15T15:49:01.000Z","dependencies_parsed_at":"2022-07-12T15:03:45.824Z","dependency_job_id":null,"html_url":"https://github.com/CubeSugar/MiniSearchEngine","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/CubeSugar/MiniSearchEngine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CubeSugar%2FMiniSearchEngine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CubeSugar%2FMiniSearchEngine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CubeSugar%2FMiniSearchEngine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CubeSugar%2FMiniSearchEngine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CubeSugar","download_url":"https://codeload.github.com/CubeSugar/MiniSearchEngine/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CubeSugar%2FMiniSearchEngine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35101089,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-03T02:00:05.635Z","response_time":110,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-11T09:37:51.509Z","updated_at":"2026-07-03T20:32:23.661Z","avatar_url":"https://github.com/CubeSugar.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"MiniSearchEngine\n================\n\n__文档版权：__ 自由转载－非商用－非衍生－保持署名 / Creative Commons Attribution-NonCommercial-NoDerivs 3.0\n\n__代码版权：__ GPLv3\n\n## Brief Introduction ##\n---\n### 1. Description ###\n\n搜索引擎迷你项目，将网页/文本数据建库，去重，计算相似度生成检索资源；编写Linux守护进程以支持前台页面的查询请求。\n\n项目关键词：**分词**、**余弦相似度**、**VSM**、**倒排索引**、**守护进程**、**网络编程**等\n\n### 2. Develop Environment ###\n\n+ Ubuntu 12.04\n\n+ g++ 4.6.3\n\n+ python 3.3.2\n\n### 3. Directory Arch ###\n\n\u003cp\u003e\nMiniSearchEngine\u003cbr\u003e\n┣ bin\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┣ Configuration.cpp\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┣ dem.cppo\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┣ Document.cpp\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┣ DocumentKits.cpp\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┣ SearchEngineServer.cpp\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┣ TcpSocket.cpp\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┗ QueryClientDemo\u003cbr\u003e\n┣ data\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┣ Raw_Data\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┣ ripepage.lib\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┣ query.lib\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┗ index.lib\u003cbr\u003e\n┣ include\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┣ Configuration.h\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┣ Document.h\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┣ DocumentKits.h\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┣ MiniSearchEngineStd.h\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┗ TcpSocket.h\u003cbr\u003e\n┣ lib\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┣ libcode.a\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┗ libelus.a\u003cbr\u003e\n┣ log\u003cbr\u003e\n┣ src\u003cbr\u003e\n┃\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;┗ stopwordlist.txt\u003cbr\u003e\n┣ search_engine.conf\u003cbr\u003e\n┗ Makefile\u003cbr\u003e\n\u003c/p\u003e\n\n\n### 4. Module\u0007 ###\n\n#### index.lib ####\n从ripepage.lib中读取query word的偏移位置，存储在index.lib文件中，格式如下：\n\t\n\tqueryword1 \\t offset1\n\tqueryword2 \\t offset2\n\t…\n\t\n\n#### WordSegement ####\n\n调用分词库libcode.a libelus.a进行分词。\n\n分词原理的学习附在Reference部分。\n\n#### Delete Duplicate ####\n\n1. 指纹法去重\n\n\t特征码：该网页/文本中词频最高的10个词；\n\t判重算法：对于每两篇，比较topword10的交集，如果交集大于6个，则认为是互为重复的网页。\n\n2. 特征码去重\n\n\t特征码：提取该网页/文本中每个逗号前后的5个字节，将这些字符串顺序连接成一个特征字符串S。\n\t判重算法：对于两篇网页/文本，计算S1和S2的LCS，如果len(LCS)/min(len(S1), len(S2)) \u003e 0.6则判定为重复。\n\n去重后生成一个全词集合文本，格式如下：\n\n\tword1 \\t docid_i \\t offset_i \\t docid_k \\t offset_k \\t …\n\tword2 \\t docid_j \\t offset_j \\t …\n\t…\n\t\n\n\n#### Calculate Similarity ####\n\n网页/文本相似度计算基于VSM与余弦相似度，本项目中采用简化模型“tf-idf”。\n\nVSM、余弦相似度的相关资料附在Reference部分。\n\n#### SearchEngineServer Daemon Process ####\nLinux 守护进程，实现cache\n\n#### Front End ####\n使用PHP编写的前端网页\n\n\n##Project Time Schedule ##\n---\nLUD = last update date\n\n  N   |    LUD   |   Project Phase     |              Description\n:-----|:---------|:--------------------|:-------------------------------------------\n 01   | 20130809 | PreProcess          |\n 02   | 20130809 | DelDuplicate        |\n 03   | 20130812 | CalcSimilarity      |\n 04   | 20130815 | SearchEngineServer  | DaemonProcess\n 05   | -        | QueryFrontEnd       | PHP WebPage\n  \n\n### 1. PreProcess / Build Data Library File \u0026 Index ###\n\n\n  N   |    LUD   |    V   |             code file           |          Description       \n:----:|:---------|:-------|:--------------------------------|:-----------------------------\n 001  | 20130722 | 0.1    | getfileinfo.py                  | add:读取文件、解析内容方法\n 002  | 20130722 | 0.1    | ripepage.py                     | add:生成抓取结果库文件\n 003  | 20130730 | 0.1    | queryindex.py                   | add:生成关键词索引文件\n 004  | 20130809 | 0.2    | getfileinfo.py                  | fix:解析方式，以空格替换任意空白符\n 005  | 20130809 | 0.2    | ripepage.py                     | fix:不输出读取失败的文件、模拟重复\n\n\n\n### 2. DelDuplicate \u0026 Build Query Keywords File ###\n\n\n  N   |    LUD   |    V   |             code file           |          Description       \n:----:|:---------|:-------|:--------------------------------|:-----------------------------\n 001  | 20130805 | 0.1    | Document.h                      | add:定义Document类\n 002  | 20130805 | 0.1    | Document.cpp                    |\n 003  | 20130805 | 0.1    | DocumentKits.h                  | add:读取正文、载入文件、转换编码函数\n 004  | 20130805 | 0.1    | DocumentKits.cpp                |\n 005  | 20130805 | 0.1    | QueryDemo.cpp                   | add:查询测试模块\n 005  | 20130806 | 0.2    | Document.h                      | fix:以重载关系操作符代替比较函数\n 006  | 20130806 | 0.2    | Document.cpp                    |\n 007  | 20130806 | 0.2    | DocumentKits.h                  | add:分词、去重函数\n 008  | 20130806 | 0.2    | DocumentKits.cpp                |\n 009  | 20130808 | 0.3    | DocumentKits.h                  | update:封装为类\n 010  | 20130808 | 0.3    | DocumentKits.cpp                |\n 001  | 20130808 | 0.2    | QueryDemo.cpp                   | update:\n 011  | 20130809 | 0.4    | DocumentKits.h                  | add:生成查询词库方法\n 012  | 20130809 | 0.4    | DocumentKits.cpp                |\n 013  | 20130809 | 0.3    | QueryDemo.cpp                   |\n 014  | 20130815 | 0.1    | Configuration.h                 | add:配置类\n 015  | 20130815 | 0.1    | Configuration.cpp               |\n\n### 3. CalcSimilarity ###\n\n\n  N   |    LUD   |    V   |             code file           |          Description       \n:----:|:---------|:-------|:--------------------------------|:-----------------------------\n 001  | 20130812 | 0.5    | DocumentKits.h                  | add:初始化文档向量、计算相似度方法\n 002  | 20130812 | 0.5    | DocumentKits.cpp                |\n 003  | 20130812 | 0.4    | QueryDemo.cpp                   | update:测试计算相似度\n\n\n### 4. SearchEngineServer ###\n\n  N   |    LUD   |    V   |             code file           |          Description       \n:----:|:---------|:-------|:--------------------------------|:-----------------------------\n 001  | 20130815 | 0.1    | TcpSocket.h                     | add: 封装TCP 函数\n 002  | 20130815 | 0.1    | TcpSocket.cpp                   |\n 003  | 20130815 | 0.1    | SearchEngineServer.cpp          | add: 搜索引擎后台守护进程\n\n\n\n\n### 5. QueryFrontEnd ###\n\n  N   |    LUD   |    V   |             code file           |          Description       \n:----:|:---------|:-------|:--------------------------------|:-----------------------------\n- |\n\n\n\n## Issue ##\n---\n1. 读取数据文件编码问题\n\n\n2. 读取文件标签匹配问题\n\n\n3. 标题多于一行\n\n\n4. 文章空白符、换行符处理方式\n\n\n5. 生成文件索引的字节位置在query标签前\n\n\n6. 取top10 受排序影响，同词频的词导致不同的top10，并影响结果\n\n\n7. c++容器变量空间，堆／栈\n\n\n8. c++ STL map\u003ckey, value\u003e按value排序\n\n\n9. 内存优化\n\n\n10. 多线程优化\n\n\n## Q \u0026 A ##\n---\n\n\n\n\n\n## Some Comment ##\n---\n1. 设计模式 单例模式etc\n\n2. rebuild 重构概念\n\n3. 缓存cache\n\n---\n## Reference ##\n\n1. 分词原理\n2. VSM\n3. 余弦相似度\n4. 文本聚类\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcubesugar%2Fminisearchengine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcubesugar%2Fminisearchengine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcubesugar%2Fminisearchengine/lists"}