{"id":33179783,"url":"https://github.com/pranab/sifarish","last_synced_at":"2026-03-14T01:01:26.714Z","repository":{"id":65874329,"uuid":"2589264","full_name":"pranab/sifarish","owner":"pranab","description":"Content based and collaborative filtering based recommendation and personalization engine implementation on Hadoop and Storm","archived":false,"fork":false,"pushed_at":"2019-11-01T12:30:21.000Z","size":7878,"stargazers_count":332,"open_issues_count":6,"forks_count":130,"subscribers_count":63,"default_branch":"master","last_synced_at":"2024-04-16T17:39:01.295Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://pkghosh.wordpress.com","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pranab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-10-17T02:53:00.000Z","updated_at":"2024-04-15T08:52:59.000Z","dependencies_parsed_at":"2023-02-14T16:31:16.796Z","dependency_job_id":null,"html_url":"https://github.com/pranab/sifarish","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pranab/sifarish","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pranab%2Fsifarish","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pranab%2Fsifarish/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pranab%2Fsifarish/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pranab%2Fsifarish/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pranab","download_url":"https://codeload.github.com/pranab/sifarish/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pranab%2Fsifarish/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27778074,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-17T02:00:08.291Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-11-16T03:00:36.842Z","updated_at":"2025-12-17T05:46:56.165Z","avatar_url":"https://github.com/pranab.png","language":"Java","funding_links":[],"categories":["人工智能"],"sub_categories":[],"readme":"## Introduction\nSifarish is a suite of solutions for recommendation personalization implementaed on \nHadoop and Storm. Various  algorithms, including  feature similarity based recommendation \nand collaborative filtering based recommendation using social rating data are available\n\n## Philosophy\n* Providing complete business solutions, not just bunch of machine learning algorithms\n* Simple to use\n* Input output in CSV format\n* Metadata defined in simple JSON file\n* Extremely configurable with tons of configuration knobs\n\n## Getting Started\nPlease read ../resource/GentleIntroductionToSifarish.docx for a high level introduction\nand overview. The various tutorial documents in the resource directory are useful for\nrunning different example use cases.\n\n## Blogs\nThe following blogs of mine are good source of details of sifarish. These are the only source\nof detail documentation\n\n* http://pkghosh.wordpress.com/2011/10/26/similarity-based-recommendation-basics/\n* http://pkghosh.wordpress.com/2011/11/28/similarity-based-recommendation-hadoop-way/\n* http://pkghosh.wordpress.com/2011/12/15/similarity-based-recommendation-text-analytic/\n* http://pkghosh.wordpress.com/2012/04/21/socially-accepted-recommendation/\n* http://pkghosh.wordpress.com/2010/10/19/recommendation-engine-powered-by-hadoop-part-1/\n* http://pkghosh.wordpress.com/2010/10/31/recommendation-engine-powered-by-hadoop-part-2/\n* http://pkghosh.wordpress.com/2012/12/31/get-social-with-pearson-correlation/\n* http://pkghosh.wordpress.com/2012/09/03/from-item-correlation-to-rating-prediction/\n* http://pkghosh.wordpress.com/2014/02/10/from-explicit-user-engagement-to-implicit-product-rating/\n* http://pkghosh.wordpress.com/2014/04/14/making-recommendations-in-real-time/\n* http://pkghosh.wordpress.com/2014/05/26/popularity-shaken/\n* http://pkghosh.wordpress.com/2014/06/23/novelty-in-personalization/\n* http://pkghosh.wordpress.com/2014/09/10/realtime-trending-analysis-with-approximate-algorithms/\n* http://pkghosh.wordpress.com/2014/12/22/positive-feedback-driven-recommendation-rank-reordering/\n* https://pkghosh.wordpress.com/2015/01/20/diversity-in-personalization-with-attribute-diffusion/\n* https://pkghosh.wordpress.com/2015/03/22/customer-service-and-recommendation-system/\n\n## Content Similarity Based Recommendation\nIn the absence of social rating data, the only options is a feature similarity \nbased recommendation. Similarity is calculated based on distance between entities \nin a multi dimensional feature space. Some examples are - recommending jobs based \non user's resume - recommending products based on user profile. These\nsolutions are known as content based recommendation, because it's based innate \nfeatures of some entity.\n\nThere are two different solutions as follows\n1. Similarity between entities of different types (e.g. user profile and product)\n2. Similarity between entities of same type (e.g. product)\n\nAttribute meta data is defined in a json file. Both entities need not have the \nsame set of attributes. Mapping between attributes values from one entity to \nthe other can be defined in the config file.\n\nThe data type supported are numerical (integer), categorical, text, geo location, time. The \ndistance algorithms  can be chosed to be euclidian, manhattan or minkowski. The default algorithm \nis euclidian. \n\nThe distancs between different atrributes of different types are combined to find distance between \ntwo entity instances. Different weights can be assigned to the attributes to control the relative \nimportance of different attributes.\n\nThe tutorial ../resource/product_similarity_tutorial.txt is a good starting point. The relevant\nblogs are useful to understand the inner workings.\n\n\n## Social Interaction Data Based Recommendation\nThese solutions are based user behavior data with respect to some product \nor service. these algorithms are also known as collaborative filtering.  \n\nUser behavior data is defined in terms of some explicit rating by user \nor it's derived from user  behavior in the site. The essential  input to all these algorithms \nis a matrix of user and items. The value for a cell could be the ratingas an integer. It could \nalso be boolean,  if the user's interest in an item is expressed as a boolean\n\nThe tutorial ../resource/tutorial.txt is a good starting point. The relevant blogs are useful \nto understand the inner workings.\n\n\n## Cold Starting Recommenders\nThese solutions are used when enough social data is not avaialable. \n\n1. If data contains text attributes, use TextAnalyzer MR to convert text to token stream \n   using lucene\n2. Find similar items based on user profile. Use DiffTypeSimilarity MR\n3. Use TopMatches MR to find top n matches for a profile\n\n\n## Warm Starting Recommenders\nWhen limited amount of user behavior data is available, these solutuions are appropriate\n\n1. If data contains text attributes, use TextAnalyzer MR to convert text to token stream \n   using lucene\n2. Find similar items by pairing items with one another using SameTypeSimilarity MR\n3. Use TopMatches MR to find top n matches for a product\n\n\n## Recommenders with Fully engaged Users\nWhen significant of user behavior data is available, these soltions can be used. In \nthe order of  complexity, the choices are as follows. They are all based on social data\n\nThere two phases for collaborative filetering based recommendation using social data\n1. Find correlation between items 2. Predict rating based on items alreadyv rated and \nresult of 1\n\nThe process involved running multiple map reduce jobs. Some of them are optional. Please refer to the \ntutorial document tutorial.txt in the resource directory\n\n\n## Real Time Recommendation\nRecommendations can be made real time based on user's current behavior in a pre defined time\nwindow. The solution is based on Storm, although Hadoop gets used to compute item correlation\nmatrix from historical user behavior data.\n\n## Text Attribute\nFor content based recommendation being able to find match between text field is an important\nfactor. Text attributes are stemmed or normalized with Apache Lucene. Various languages, in addition\nto default of english are supported. They are german, french, italian, spanish, polish and brazilian\nportuguese. Text matching algorithms supported are cosine, jaccard and semantic. For semantic matching,\nRDF semantic graph is used \n\n## Complex Attributes\nFor content based recommendation, There is  support for structured fields e.g., Location, Time Window, \nEvent, Categorized Item etc. Many of these  provide contextual dimensions to recommendation. They \nare particularly relevant for recommendation in the mobile space\n\n## Novelty \nNovelty for an item can be computed at individual user level or the whole user community as a\nwhole. Novelty is blended into the final recommendation list by taking weighted average of\npredicted rating and novelty  \n\n## Diversilty \nBased on recent work in the academic world, I am working on implementing some  algorithms to introduce  \ndiversity in recommendation. Unlike novelty, diversity is group wise property. Diversity can be\ndefined either in terms item dissimilarity in a collaborative filtering sense or structural and content \nsense\n\n## Facted Match\nFor content based recommendation, faceted match is supported as faceted search in Solr.\nFaceted fields are specified through a configuration parameter\n\n## Dithering\nDithering effectively handles the problem users usually not browsing the first few items\nin a list. The dithering process shuffles the list little bit, every time recommended items \nare presented to the user.\n \n## Getting started\nPlease use the tutorial.txt file in the resource directory for batch mode recommendation \nprocessing. For real time recommendation please use the tutorial document there is a separate\ntutorial document realtime\\_recommendation\\_tutorial.txt\n\n## Integration with other recommndation systems\nIf you use Apache mahout or some thing else for recommendation, you can\nbring your basic recommendation output (userID, itemID, predictedRating) to\nsifarish for additional postprocessing to improve the quality of the output. They are\nlisted in the next section.\n\n## Post processing plugins\nJust accuracy from the CF algorithm is not enough for a good recommender. There\nare various post processing plugins are essential. They improve the quality of results. \nHere is the list. Sifarsh supports most them. Some are under development.\n\n* Business goal injection \n* Adding novelty \n* Adding diversity \n* Rank reordering for explicit positive feedback \n* Rank reordering for implicit negative feedback \n* Dithering\n\n## Configuration\nPlease refer to the wiki page for a detailed list of all configuration parameters\nhttps://github.com/pranab/sifarish/wiki/Configuration. Going through the tutorial documents\nin the resource directory, you can find sample configuration for various use cases.\n\n## Build\nPlease read jar\\_dependency.txt in the resource directory for build and run time dependency\n\nFor Hadoop 1\n* mvn clean install\n\nFor Hadoop 2 (non yarn), use the branch nuovo\n* git checkout nuovo\n* mvn clean install\n\nFor Hadoop 2 (yarn), use the branch nuovo\n* git checkout nuovo\n* mvn clean install -P yarn\n\n## Help\nPlease feel free to email me at pkghosh99@gmail.com\n\n## Contribution\nContributors are welcome. Please email me at pkghosh99@gmail.com\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpranab%2Fsifarish","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpranab%2Fsifarish","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpranab%2Fsifarish/lists"}