{"id":20475357,"url":"https://github.com/ashwinpn/wikisea","last_synced_at":"2025-03-05T14:26:57.284Z","repository":{"id":201600124,"uuid":"323153403","full_name":"ashwinpn/WikiSea","owner":"ashwinpn","description":"Search Engine for Wikipedia.","archived":false,"fork":false,"pushed_at":"2021-01-13T16:43:14.000Z","size":99,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-16T03:15:29.758Z","etag":null,"topics":["mapreduce-python","search-engine","wikipedia"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ashwinpn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-12-20T19:55:20.000Z","updated_at":"2021-01-13T16:48:44.000Z","dependencies_parsed_at":"2024-03-12T00:35:05.192Z","dependency_job_id":null,"html_url":"https://github.com/ashwinpn/WikiSea","commit_stats":null,"previous_names":["ashwinpn/wikisea"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwinpn%2FWikiSea","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwinpn%2FWikiSea/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwinpn%2FWikiSea/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwinpn%2FWikiSea/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ashwinpn","download_url":"https://codeload.github.com/ashwinpn/WikiSea/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242042135,"owners_count":20062355,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mapreduce-python","search-engine","wikipedia"],"created_at":"2024-11-15T15:15:48.080Z","updated_at":"2025-03-05T14:26:57.264Z","avatar_url":"https://github.com/ashwinpn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# WikiSea\nWikipedia search engine.\n\n![pic](https://github.com/ashwinpn/WikiSea/blob/main/resources/wsea.png)\n\n## Deploying on Courant Servers\n\nDeploy the search engine (indexer with job management system and queryer) on CIMS's server. In this case, we use 1% Wikipedia data and produce 2 data groups each has 2 shards.\n\n### Requirements\n\n- 2+ server (1 master, 1+ workers)\n- Python 3.6+\n- cross-server available data folder (CIMS `/data` directory or GCS with `gcs-fuse`)\n\n### Note\n\n**Different CIMS Servers may have a different python version. One may need to configure different virtualenv on different servers.**\n\n\n### Dependencies\n    \n    \n    pip install -r requirements.txt\n\n### Indexer\n\n#### Job Tracker\n```bash\n    export JOB_TRACKER_PORT=23233\n    cd /path/to/sea\n    export PYTHONPATH=$(pwd)\n    python -m master.job_tracker\n ```\n#### Create Data Folders\n\nHere we show the case for 2 groups.\n```bash\n    $mkdir -p /path/to/data/invindex_out/0\n    $mkdir -p /path/to/data/invindex_out/1\n    $mkdir -p /path/to/data/idf_out/0\n    $mkdir -p /path/to/data/idf_out/1\n    $mkdir -p /path/to/data/docs_out/0\n    $mkdir -p /path/to/data/docs_out/1\n    $mkdir -p /path/to/data/output/0\n    $mkdir -p /path/to/data/output/1\n```\n#### Task Tracker (for each worker node)\n```bash\n    cd /path/to/sea\n    export PYTHONPATH=$(pwd)\n    export TASK_TRACKER_PORT=30001\n    export JOB_TRACKER=\"http://[job_tracker]:23233\"\n    export IDF_FILE=/path/to/data/idf_out/0.out\n    python -m mapreduce.workers \u0026    \n    python -m client.task_tracker\n```\n#### Check Deployment\n```bash\n    $ curl http://[job_tracker]:23233/status\n    {\n        \"workers\": [\n        {\n        \"host\": \"http://172.22.80.150:30001\",\n        \"state\": 0,\n        \"last_hbt\": 1493739581.0538707,\n        \"owner_jid\": \"None\",\n        \"owner_tid\": \"None\"\n        },\n        {\n        \"host\": \"http://172.22.80.145:30001\",\n        \"state\": 0,\n        \"last_hbt\": 1493739580.8972585,\n        \"owner_jid\": \"None\",\n        \"owner_tid\": \"None\"\n        }\n        ],\n        \"current\": \"None\",\n        \"jobs\": []\n    }%\n```\n#### Create Job\n\nPlease run the following task one-by-one.\n\nReformat task\n```bash\n    $ curl http://[job_tracker]:23233/new?spec=/path/to/enwiki-mini.yaml\n    {\"status\": \"ok\"}%\n```\nInverted Index task\n```bash\n    $ curl http://[job_tracker]:23233/new?spec=/path/to/invindex-mini.yaml\n    {\"status\": \"ok\"}%\n```\nIn-Group IDF task\n```bash\n    $ curl http://[job_tracker]:23233/new?spec=/path/to/idf-mini.yaml\n    {\"status\": \"ok\"}%\n```\nGlobal IDF task\n```bash\n    $ curl http://[job_tracker]:23233/new?spec=/path/to/accu_idf-mini.yaml\n    {\"status\": \"ok\"}%\n    $ bzip2 -d /path/to/data/idf_out/0.out.bz2\n```\n**Need to decompress the IDF before next step.**\n\nDocument task\n```bash\n    $ curl http://[job_tracker]:23233/new?spec=/path/to/docs-mini.yaml\n    {\"status\": \"ok\"}%\n```\nIntegrate task\n```bash\n    $ curl http://[job_tracker]:23233/new?spec=/path/to/integrate-mini.yaml\n    {\"status\": \"ok\"}%\n```\nFinal result:\n```bash\n    (venv) jz2653@linax2[jobs]$ tree     /data/jz2653/mini-serval/\n    ├── 0\n    │   ├── reformatted_0.in.bz2\n    │   └── reformatted_1.in.bz2\n    ├── 1\n    │   ├── reformatted_0.in.bz2\n    │   └── reformatted_1.in.bz2\n    ├── docs_out\n    │   ├── 0\n    │   │   ├── 0.err\n    │   │   ├── 0.out.bz2\n    │   │   ├── 1.err\n    │   │   └── 1.out.bz2\n    │   └── 1\n    │       ├── 0.err\n    │       ├── 0.out.bz2\n    │       ├── 1.err\n    │       └── 1.out.bz2\n    ├── idf_out\n    │   ├── 0\n    │   │   ├── 0.err\n    │   │   └── 0.out.bz2\n    │   ├── 0.err\n    │   ├── 0.out\n    │   └── 1\n    │       ├── 0.err\n    │       └── 0.out.bz2\n    ├── invindex_out\n    │   ├── 0\n    │   │   ├── 0.err\n    │   │   ├── 0.out.bz2\n    │   │   ├── 1.err\n    │   │   └── 1.out.bz2\n    │   └── 1\n    │       ├── 0.err\n    │       ├── 0.out.bz2\n    │       ├── 1.err\n    │       └── 1.out.bz2\n    └── output\n        ├── 0\n        │   ├── docs_0.pkl.bz2\n        │   ├── docs_1.pkl.bz2\n        │   ├── indexes_0.pkl.bz2\n        │   ├── indexes_1.pkl.bz2\n        │   └── tfidf.pkl\n        └── 1\n            ├── docs_0.pkl.bz2\n            ├── docs_1.pkl.bz2\n            ├── indexes_0.pkl.bz2\n            ├── indexes_1.pkl.bz2\n            └── tfidf.pkl\n\n    14 directories, 36 files\n```\n### Queryer\n\n#### `etcd` service\n\nFirst, download `etcd` source code and compile.\n```bash\n    $ git clone https://github.com/coreos/etcd.git\n    $ cd etcd\n    $ ./build\n    $ ./bin/etcd\n```\n\u003e from `etcd` github page.\n\nStart `etcd` service\n```bash\n    $ export EXTERNAL_IP=[this host's external IP]\n    $ /path/to/bin/etcd \\\n        --advertise-client-urls=http://$(hostname):32379 \\\n        --listen-client-urls=http://127.0.0.1:2379,http://$EXTERNAL_IP:32379\n```\n#### Master\n```bash\n    $ cd /path/to/sea\n    $ etcdctl --endpoints http://linserv2.cims.nyu.edu:32379 set /misaki/n_srv 2\n    $ export ETCD_CLUSTER=[etcd_host]:32379\n    $ export MASTER_PORT=11112\n    $ python -m search.master\n```\n#### Host\n\nchange shard size to 2:\n```bash\n    $ vim /path/to/sea/search/manifest.py\n```\nmodify line 7 and 8 from:\n\n    N_INDEX_SRV = 8\n    N_DOC_SRV = 8\n\nto:\n\n    N_INDEX_SRV = 2\n    N_DOC_SRV = 2\n\nstart host:\n```bash\n    $ export ETCD_CLUSTER=[etcd_host]:32379\n    $ export DATA_BASE=/path/to/data/output\n    $ python -m search.start\n```\nopen browser:\n```bash\n    $ firefox http://[master host]:11112/\n```    \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashwinpn%2Fwikisea","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fashwinpn%2Fwikisea","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashwinpn%2Fwikisea/lists"}