{"id":21006276,"url":"https://github.com/derlin/bda-lsa-project","last_synced_at":"2026-04-13T07:08:53.717Z","repository":{"id":148622606,"uuid":"91235011","full_name":"derlin/bda-lsa-project","owner":"derlin","description":"BDA project - latent semantic analysis of Wikipedia with spark (SVD and LDA)","archived":false,"fork":false,"pushed_at":"2018-08-18T07:53:18.000Z","size":2639,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-01-20T10:31:09.252Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/derlin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-14T10:08:55.000Z","updated_at":"2024-06-22T02:46:04.000Z","dependencies_parsed_at":"2023-05-20T17:45:12.954Z","dependency_job_id":null,"html_url":"https://github.com/derlin/bda-lsa-project","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Fbda-lsa-project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Fbda-lsa-project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Fbda-lsa-project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Fbda-lsa-project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/derlin","download_url":"https://codeload.github.com/derlin/bda-lsa-project/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243428421,"owners_count":20289315,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T08:50:29.490Z","updated_at":"2025-12-30T10:22:58.579Z","avatar_url":"https://github.com/derlin.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![scaladoc](https://img.shields.io/badge/doc-scaladoc-blue.svg)](https://derlin.github.io/bda-lsa-project/api/index.html)\n[![gh-pages](https://img.shields.io/badge/doc-gh--pages-blue.svg)](https://derlin.github.io/bda-lsa-project)\n[![scaladoc](https://img.shields.io/badge/doc-wiki-blue.svg)](https://github.com/derlin/bda-lsa-project/wiki)\n![love](https://img.shields.io/badge/made%20with%20%E2%99%A5-in%20Switzerland-ff69b4.svg)\n\n# Table of Contents\n\n- [About this repository](#about-this-repository)\n  * [Structure](#structure)\n- [Building and Running](#building-and-running)\n  * [Building the jar](#building-the-jar)\n  * [Project structure](#project-structure)\n  * [Example of pipeline](#example-of-pipeline)\n- [Models available](#models-available)\n  * [SVD](#svd)\n  * [LDA](#lda)\n      - [ml.LDA](#mllda)\n      - [mllib.LDA](#mlliblda)\n  \n \u003csmall\u003e\u003ci\u003e\u003ca href='http://ecotrust-canada.github.io/markdown-toc/'\u003eTable of contents generated with markdown-toc\u003c/a\u003e\u003c/i\u003e\u003c/small\u003e\n \n# About this repository\n\n_Context_: BDA (Big Data Analytics), Master MSE, june 2017.\n\n_Authors_: Lucy Linder, Kewin Dousse, Davide Mazzolini, Christophe Blanquet.\n\nThis project is based on Chapter 6 of the book [_Advanced Analytics with Spark_](https://github.com/sryza/aas). It contains code and information on how to apply LSA techniques to the English Wikipedia articles corpus. \n\n## Structure\n\n- the [source code](src/main/scala) is made of multiple classes thoroughly commented and documented with scaladoc\n- the folder [spark-shell-scripts](spark-shell-scripts) contains scripts intended to be loaded into a spark-shell session. Once again, most of them are documented\n- READMEs are scattered at each level in order to help you understand the repository structure\n- the [wiki](https://github.com/derlin/bda-lsa-project/wiki) contains the results, notes, tips and tricks, how-to etc. This is a good place to start if you just want to see what we did.\n- the [gh-pages website](https://github.com/derlin/bda-lsa-project) contains other resources such as `all-steps-from-book` in html as well as the __scaladoc__ \n- a __presentation__ is also available in [PDF (other/bda-lsa-project-slides.pdf)](other/bda-lsa-project-slides.pdf) or in [google slides](https://docs.google.com/presentation/d/1QmcnOZX43cVA3acqbjQT5rEmPPhYmRVuP0GbZjslEhM/edit?usp=sharing)  \n\n# Building and Running\n\n## Building the jar\n\nThis project uses sbt. To create the jar, use:\n    \n    export JAVA_OPTS=\"-Xms256m -Xmx4g\"\n    sbt assembly\n\nThe jar is now available under `target/scala-2.11/bda-project-lsa-assembly-1.0.jar`.\n\n## Project structure\n\nThe project is made of multiple spark programs. Each program stores its output on disk, the actual location depending on the properties set in `config.properties`.\n \n A usual pipeline is:\n \n 1. convert wikidump XML into plain text (`bda.lsa.preprocessing.XmlToParquetWriter`)\n 2. create the vocabulary and the TF-IDF matrix (`bda.lsa.preprocessing.DocTermMatrixWriter`)\n 3. create one of the models (`bda.lsa.svd.RunSVD`, `bda.lsa.lda.mllib.RunLDA`, `bda.lsa.lda.ml.RunLDA`)\n \n At this point, the model is persisted somewhere and you can load it inside a spark-shell to interact with it. The classes `bda.lsa.svd.SVDQueryEngine`, `bda.lsa.lda.mllib.LDAQueryEngine` and `bda.lsa.lda.ml.LDAQueryEngine` implement useful queries to analyse the models. \n\n\n## Example of pipeline\n\n1. ensure you have a _wikidump_ somewhere to process.\n2. create the jar: \n\n        export JAVA_OPTS=\"-Xms256m -Xmx4g\"\n        sbt assembly\n        \n3. create a `config.properties` and add the following:\n\n        path.wikidump=wikidump-1500.xml # your xml \n        path.base=/tmp/spark-wiki/   # a base path to store the results\n        \n4. convert XML to text:  \n \n        spark-submit --class bda.lsa.preprocessing.XmlToParquetWriter \\\n            target/scala-2.11/bda-project-lsa-assembly-1.0.jar\n            \n    This will save the DataFrame `[(title: String, content: String) ]` in `path.base/wikidump-parquet`.\n    __important__: the job will fail if the aforementioned directory already exists !\n\n5. create the TF-IDF matrix:\n\n        spark-submit --class bda.lsa.preprocessing.DocTermMatrixWriter \\\n                target/scala-2.11/bda-project-lsa-assembly-1.0.jar  \\\n                \u003cnumTerms\u003e [\u003cpercent\u003e \u003cnumDocs\u003e]\n                \n   The only required argument is `numTerms`: this is the size of the vocabulary.\n    \n6. Run one of the models. For example, for svd:\n    \n        spark-submit --class bda.lsa.svd.RunSVD \\\n                       target/scala-2.11/bda-project-lsa-assembly-1.0.jar  \\\n                       \u003ck: default 100\u003e\n     \n   The parameter `k` is the number of topics to infer, default to 100.\n   \n7. open a spark-shell and load the model:\n\n        spark-shell --jars target/scala-2.11/bda-project-lsa-assembly-1.0.jar \n        val data = bda.lsa.getData(spark)\n        val model = bda.lsa.svd.RunSVD.loadModel(spark)\n        \n8. Optionally, use the querier in the shell:\n\n        val q = new bda.lsa.svd.SVDQueryEngine(model, data)\n   \n   \n   \n# Models available\n\n## SVD\n\nThe class `bda.lsa.svd.RunSVD` makes it easy to compute an SVD model.\n\nThe results are saved in `{base.path}/svd`. More information about SVD can be found in the [svd package README](/derlin/bda-lsa-project/blob/master/src/main/scala/bda/lsa/svd/Readme.md).\n\nAfter creating the model (see steps above), you can use the `bda.lsa.svd.SVDQueryEngine` to discover the results. From a spark-shell:\n\n```\nspark-shell --jars bda-project-lsa-assembly-1.0.jar\n\u003e val data = bda.lsa.getData(spark)\n\u003e val model = bda.lsa.svd.RunSVD.loadModel(spark)\n\u003e val q = new bda.lsa.svd.SVDQueryEngine(model, data)\n```\n\nSee the [wiki](/derlin/bda-lsa-project/wiki) for our results and conclusion.\n \n## LDA\n \n LDA models are available in two flavors: with _spark mllib_ and _spark ml_.\n \n  We focused on the mllib implementation, mostly because the `org.apache.spark.mllib.clustering.DistributedLDAModel` offer more utility methods than it's ml counterpart. Our ml implementation will creates the model, but does not offer a useful query engine.\n  \n#### ml.LDA\n\n \nTo run the model:\n\n    spark-submit --class bda.lsa.lda.ml.RunLDA \\\n          target/scala-2.11/bda-project-lsa-assembly-1.0.jar  \\\n          \u003ck: default 100\u003e  \u003cmaxIters: default 100\u003e \n          \nThe model is then saved to `{base.path}/ml-lda`.\n\nAfter creating the model (see steps above), you can use the `bda.lsa.lda.mllib.LDAQueryEngine` to discover the results. From a spark-shell:\n \n ```\n spark-shell --jars bda-project-lsa-assembly-1.0.jar\n \u003e val data = bda.lsa.getData(spark)\n \u003e val model = bda.lsa.lda.ml.RunLDA.loadModel(spark)\n \u003e val q = new bda.lsa.ml.LDAQueryEngine(model, data)\n ```\nNote that the query engine might not be the most efficient... Some queries take time !\n \n#### mllib.LDA\n \n \nTo run the model:\n\n    spark-submit --class bda.lsa.lda.mllib.RunLDA \\\n          target/scala-2.11/bda-project-lsa-assembly-1.0.jar  \\\n          \u003ck: default 100\u003e  \u003cmaxIters: default 100\u003e \u003calpha: default -1\u003e  \u003cbeta: default -1\u003e\n          \nThe model is then saved to `{base.path}/mllib-lda`.\n\nAfter creating the model (see steps above), you can use the `bda.lsa.lda.mllib.LDAQueryEngine` to discover the results. From a spark-shell:\n \n ```\n spark-shell --jars bda-project-lsa-assembly-1.0.jar\n \u003e val data = bda.lsa.getData(spark)\n \u003e val model = bda.lsa.lda.mllib.RunLDA.loadModel(spark)\n \u003e val q = new bda.lsa.lda.mllib.LDAQueryEngine(model, data)\n ```\n \n See the [wiki](/derlin/bda-lsa-project/wiki) for our results and conclusion.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderlin%2Fbda-lsa-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fderlin%2Fbda-lsa-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderlin%2Fbda-lsa-project/lists"}