{"id":14982377,"url":"https://github.com/vinta/albedo","last_synced_at":"2025-08-09T05:26:13.208Z","repository":{"id":65974796,"uuid":"83229036","full_name":"vinta/albedo","owner":"vinta","description":"A recommender system for discovering GitHub repos, built with Apache Spark","archived":false,"fork":false,"pushed_at":"2020-06-11T18:56:45.000Z","size":453,"stargazers_count":174,"open_issues_count":1,"forks_count":39,"subscribers_count":22,"default_branch":"master","last_synced_at":"2024-12-07T00:03:28.387Z","etag":null,"topics":["apache-spark","elasticsearch","feature-engineering","machine-learning","python","recommender-system","scala"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vinta.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-02-26T17:57:30.000Z","updated_at":"2024-11-12T14:57:52.000Z","dependencies_parsed_at":"2023-02-19T18:15:34.495Z","dependency_job_id":null,"html_url":"https://github.com/vinta/albedo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vinta%2Falbedo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vinta%2Falbedo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vinta%2Falbedo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vinta%2Falbedo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vinta","download_url":"https://codeload.github.com/vinta/albedo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230511479,"owners_count":18237657,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","elasticsearch","feature-engineering","machine-learning","python","recommender-system","scala"],"created_at":"2024-09-24T14:05:18.328Z","updated_at":"2024-12-19T23:13:57.278Z","avatar_url":"https://github.com/vinta.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"Albedo\n======\n\nA recommender system for discovering GitHub repos, built with [Apache Spark](https://spark.apache.org/).\n\n**Albedo** is a fictional character in Dan Simmons's [Hyperion Cantos](https://en.wikipedia.org/wiki/Hyperion_Cantos) series. Councilor Albedo is the TechnoCore's AI advisor to the Hegemony of Man.\n\n## Setup\n\n```bash\n$ git clone https://github.com/vinta/albedo.git\n$ cd albedo\n$ make up\n```\n\n## Collect Data\n\nYou need to create your own `GITHUB_PERSONAL_TOKEN` on [your GitHub settings page](https://help.github.com/articles/creating-an-access-token-for-command-line-use/).\n\n```bash\n# get into the main container\n$ make attach\n\n# this step might take a few hours to complete\n# depends on how many repos you starred and how many users you followed\n$ (container) python manage.py migrate\n$ (container) python manage.py collect_data -t GITHUB_PERSONAL_TOKEN -u GITHUB_USERNAME\n# or\n$ (container) wget https://s3-ap-northeast-1.amazonaws.com/files.albedo.one/albedo.sql\n$ (container) mysql -h mysql -u root -p123 albedo \u003c albedo.sql\n\n# username: albedo\n# password: hyperion\n$ make run\n$ open http://127.0.0.1:8000/admin/\n```\n\n## Start a Spark Cluster\n\nYou could also create a Spark cluster on [Google Cloud Dataproc](https://cloud.google.com/dataproc/).\n\n```bash\n# start a local Spark cluster in Standalone mode\n$ make spark_start\n```\n\n## Use Popularity as the Recommendation Baseline\n\nSee [PopularityRecommenderBuilder.scala](src/main/scala/ws/vinta/albedo/PopularityRecommenderBuilder.scala) for complete code.\n\n```bash\n$ spark-submit \\\n    --master spark://localhost:7077 \\\n    --packages \"com.github.fommil.netlib:all:1.1.2,mysql:mysql-connector-java:5.1.41\" \\\n    --class ws.vinta.albedo.PopularityRecommenderTrainer \\\n    target/albedo-1.0.0-SNAPSHOT.jar\n# NDCG@30 = 0.002017744675282716\n```\n\n## Build the User Profile for Feature Engineering\n\nSee [UserProfileBuilder.scala](src/main/scala/ws/vinta/albedo/UserProfileBuilder.scala) for complete code.\n\n```bash\n$ spark-submit \\\n    --master spark://localhost:7077 \\\n    --packages \"com.github.fommil.netlib:all:1.1.2,mysql:mysql-connector-java:5.1.41\" \\\n    --class ws.vinta.albedo.UserProfileBuilder \\\n    target/albedo-1.0.0-SNAPSHOT.jar\n```\n\n## Build the Item Profile for Feature Engineering\n\nSee [RepoProfileBuilder.scala](src/main/scala/ws/vinta/albedo/RepoProfileBuilder.scala) for complete code.\n\n```bash\n$ spark-submit \\\n    --master spark://localhost:7077 \\\n    --packages \"com.github.fommil.netlib:all:1.1.2,mysql:mysql-connector-java:5.1.41\" \\\n    --class ws.vinta.albedo.RepoProfileBuilder \\\n    target/albedo-1.0.0-SNAPSHOT.jar\n```\n\n## Train an ALS Model for Candidate Generation\n\nSee [ALSRecommenderBuilder.scala](src/main/scala/ws/vinta/albedo/ALSRecommenderBuilder.scala) for complete code.\n\n```bash\n$ spark-submit \\\n    --master spark://localhost:7077 \\\n    --packages \"com.github.fommil.netlib:all:1.1.2,mysql:mysql-connector-java:5.1.41\" \\\n    --class ws.vinta.albedo.ALSRecommenderBuilder \\\n    target/albedo-1.0.0-SNAPSHOT.jar\n# NDCG@30 = 0.05209047292612741\n```\n\n## Build a Content-based Recommender for Candidate Generation\n\nElasticsearch's [More Like This](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html) API will do the tricks.\n\n```bash\n$ (container) python manage.py sync_data_to_es\n```\n\nSee [ContentRecommenderBuilder.scala](src/main/scala/ws/vinta/albedo/ContentRecommenderBuilder.scala) for complete code.\n\n```bash\n$ spark-submit \\\n    --master spark://localhost:7077 \\\n    --packages \"com.github.fommil.netlib:all:1.1.2,org.apache.httpcomponents:httpclient:4.5.2,org.elasticsearch.client:elasticsearch-rest-high-level-client:5.6.2,mysql:mysql-connector-java:5.1.41\" \\\n    --class ws.vinta.albedo.ContentRecommenderBuilder \\\n    target/albedo-1.0.0-SNAPSHOT.jar\n# NDCG@30 = 0.002559563451967487\n```\n\n## Train a Word2Vec Model for Text Vectorization\n\nSee [Word2VecCorpusBuilder.scala](src/main/scala/ws/vinta/albedo/Word2VecCorpusBuilder.scala) for complete code.\n\n```bash\n$ spark-submit \\\n    --master spark://localhost:7077 \\\n    --packages \"com.github.fommil.netlib:all:1.1.2,com.hankcs:hanlp:portable-1.3.4,mysql:mysql-connector-java:5.1.41\" \\\n    --class ws.vinta.albedo.Word2VecCorpusBuilder \\\n    target/albedo-1.0.0-SNAPSHOT.jar\n```\n\n## Train a Logistic Regression Model for Ranking\n\nSee [LogisticRegressionRanker.scala](src/main/scala/ws/vinta/albedo/LogisticRegressionRanker.scala) for complete code.\n\n```bash\n$ spark-submit \\\n    --master spark://localhost:7077 \\\n    --packages \"com.github.fommil.netlib:all:1.1.2,com.hankcs:hanlp:portable-1.3.4,mysql:mysql-connector-java:5.1.41\" \\\n    --class ws.vinta.albedo.LogisticRegressionRanker \\\n    target/albedo-1.0.0-SNAPSHOT.jar\n# NDCG@30 = 0.021114356461615493\n```\n\n## TODO\n\n- Build a recommender system with Spark: Factorization Machine\n- Build a recommender system with Spark: GDBT for Feature Learning\n- Build a recommender system with Spark: Item2Vec\n- Build a recommender system with Spark: PageRank and GraphX\n- Build a recommender system with Spark: XGBoost\n\n## Related Posts\n\n- [Build a recommender system with Spark: Implicit ALS](https://vinta.ws/code/build-a-recommender-system-with-pyspark-implicit-als.html)\n- [Build a recommender system with Spark: Content-based and Elasticsearch](https://vinta.ws/code/build-a-recommender-system-with-spark-content-based-and-elasticsearch.html)\n- [Build a recommender system with Spark: Logistic Regression](https://vinta.ws/code/build-a-recommender-system-with-spark-logistic-regression.html)\n- [Feature Engineering 特徵工程中常見的方法](https://vinta.ws/code/feature-engineering.html)\n- [Spark ML cookbook (Scala)](https://vinta.ws/code/spark-ml-cookbook-scala.html)\n- [Spark SQL cookbook (Scala)](https://vinta.ws/code/spark-sql-cookbook-scala.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvinta%2Falbedo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvinta%2Falbedo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvinta%2Falbedo/lists"}