https://github.com/microsoft/LightLDA

Scalable, fast, and lightweight system for large-scale topic modeling
https://github.com/microsoft/LightLDA

Last synced: about 1 month ago
JSON representation

Scalable, fast, and lightweight system for large-scale topic modeling

• Host: GitHub
• URL: https://github.com/microsoft/LightLDA
• Owner: microsoft
• License: mit
• Archived: true
• Created: 2015-09-11T02:42:49.000Z (over 9 years ago)
• Default Branch: master
• Last Pushed: 2020-12-28T12:12:12.000Z (almost 4 years ago)
• Last Synced: 2024-08-03T18:20:58.572Z (4 months ago)
• Language: C++
• Homepage: http://www.dmtk.io
• Size: 82 KB
• Stars: 840
• Watchers: 94
• Forks: 235
• Open Issues: 45
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-topic-models - lightLDA - C++ implementation using O(1) Metropolis-Hastings sampling [:page_facing_up:](https://arxiv.org/pdf/1412.1576.pdf) (Models / Latent Dirichlet Allocation (LDA) [:page_facing_up:](https://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf))

README

        
# LightLDA

LightLDA is a distributed system for large scale topic modeling. It implements a distributed sampler that enables very large data sizes and models. LightLDA improves sampling throughput and convergence speed via a fast O(1) metropolis-Hastings algorithm, and allows small cluster to tackle very large data and model sizes through model scheduling and data parallelism architecture. LightLDA is implemented with C++ for performance consideration.

We have sucessfully trained big topic models (with trillions of parameters) on big data (Top 10% PageRank values of Bing indexed page, containing billions of documents) in Microsoft. For more technical details, please refer to our [WWW'15 paper](http://www.www2015.it/documents/proceedings/proceedings/p1351.pdf). 

For documents, please view our website [http://www.dmtk.io](http://www.dmtk.io).

## Why LightLDA

The highlight features of LightLDA are

* **Scalable**: LightLDA can train models with trillions of parameters on big data with billions of documents, a scale previous implementations cann't handle. 

* **Fast**: The sampler can sample millions of tokens per second per multi-core node.

* **Lightweight**: Such big tasks can be trained with as few as tens of machines.

## Quick Start

Run ``` $ sh build.sh ``` to build lightlda.

Run ``` $ sh example/nytimes.sh ``` for a simple example.

## Reference

Please cite LightLDA if it helps in your research:

```

@inproceedings{yuan2015lightlda,

  title={LightLDA: Big Topic Models on Modest Computer Clusters},

  author={Yuan, Jinhui and Gao, Fei and Ho, Qirong and Dai, Wei and Wei, Jinliang and Zheng, Xun and Xing, Eric Po and Liu, Tie-Yan and Ma, Wei-Ying},

  booktitle={Proceedings of the 24th International Conference on World Wide Web},

  pages={1351--1361},

  year={2015},

  organization={International World Wide Web Conferences Steering Committee}

}

```

Microsoft Open Source Code of Conduct

------------

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [[email protected]](mailto:[email protected]) with any additional questions or comments.