{"id":19738974,"url":"https://github.com/fpopic/master-indexers","last_synced_at":"2026-06-08T01:31:38.366Z","repository":{"id":71942388,"uuid":"92337166","full_name":"fpopic/master-indexers","owner":"fpopic","description":"(Class) C++ script to preprocess input files (mapping ids to indices and vice versa) on a single core machine. ","archived":false,"fork":false,"pushed_at":"2017-07-06T22:07:31.000Z","size":11,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-11-23T01:03:09.348Z","etag":null,"topics":["cpp","id-to-index","indexing-engine","recommender-system"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fpopic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-24T21:13:20.000Z","updated_at":"2019-04-22T08:53:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"1222d4ae-cb72-4043-93ba-36917a13b687","html_url":"https://github.com/fpopic/master-indexers","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/fpopic/master-indexers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fpopic%2Fmaster-indexers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fpopic%2Fmaster-indexers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fpopic%2Fmaster-indexers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fpopic%2Fmaster-indexers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fpopic","download_url":"https://codeload.github.com/fpopic/master-indexers/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fpopic%2Fmaster-indexers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34044919,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-07T02:00:07.652Z","response_time":124,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpp","id-to-index","indexing-engine","recommender-system"],"created_at":"2024-11-12T01:15:50.316Z","updated_at":"2026-06-08T01:31:38.349Z","avatar_url":"https://github.com/fpopic.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"#### Usage:\n\nUsed to preprocess input files on a single machine (because it constructs shared lookup table, \ntherefore synchronisation and locking for mutable collection are needed on a multicore-distributed environment).\n \nIt maps ids to indices (e.g. userId =\u003e userIndex, itemId =\u003e itemIndex) and saves results to a file with it's lookup table. \n\nIndices are prefered over ids because linear algebra libraries operate with indices.\n\nScala time \u003e 45 min (only for item-item matrix)\\\nC++   time \u003c 5 min  (for both matrices)\n\nUnindexer is used to revert indices to ids after the recommendations are computed.\n\n#### Compile:\n\nusing g++:\n\n```\ng++ -static -std=c++11 -O3 indexer.cpp -o indexer\ng++ -static -std=c++11 -O3 unindexer.cpp -o unindexer\n```\n\n\nor with cmake:\n\n```\nmkdir cmake-build-debug\ncmake --build cmake-build-debug --target indexer -- -j 4\ncmake --build cmake-build-debug --target unindexer -- -j 4\n```\n\n#### Run indexer:\n\n```./indexer \u003citem_item_input\u003e \u003cuser_item_input\u003e```\n\nOutput files will be in the same folder as input with .indexed and .lookup suffix.\n\n#### Output:\n```\nHeader: itemId1,itemId2,a,b,c,d\nProcessed lines:1000000\n...\nProcessed lines:71000000\nOutput file with 71930771 lines saved.\nLookup file with 67052 entries saved.\nTime: 166s\n\nHeader: userId,date,itemId,quantity\nProcessed lines:1000000\n...\nProcessed lines:29000000\nOutput file with 29154707 lines saved.\nLookup file with 20998 entries saved.\nTime: 50s\n\nProcess finished with exit code 0\n```\n\n#### Run unindexer:\n\n```./unindexer \u003crecommendations_input\u003e \u003cusers_lookup_input\u003e \u003citems_lookup_input\u003e```\n\n\u003crecommendations_input\u003e file must be formated like:\n\n```\"userId:itemId1,itemId2,...,itemIdX\"```\n\nOutput files will be in the same folder as input with .unindexed suffix formated like:\n\n```\"userIndex:itemIndex1,itemIndex2,...,itemIndexX\"```\n \nwhere k is the number of top k recommendations for a user, so 0 \u003c= X \u003c= k\n\n#### Output:\n```\nProcessed lines:1000\n...\nProcessed lines:2000\nOutput file with 21225 lines saved.\nTime: 4s\n\nProcess finished with exit code 0\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffpopic%2Fmaster-indexers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffpopic%2Fmaster-indexers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffpopic%2Fmaster-indexers/lists"}