Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/baidu/familia
A Toolkit for Industrial Topic Modeling
https://github.com/baidu/familia
lda nlp sentence-lda topic-modeling topic-models twe
Last synced: about 1 month ago
JSON representation
A Toolkit for Industrial Topic Modeling
- Host: GitHub
- URL: https://github.com/baidu/familia
- Owner: baidu
- License: bsd-3-clause
- Created: 2017-06-14T06:48:21.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2021-07-01T08:28:33.000Z (over 3 years ago)
- Last Synced: 2024-10-09T13:41:20.031Z (about 1 month ago)
- Topics: lda, nlp, sentence-lda, topic-modeling, topic-models, twe
- Language: C++
- Homepage:
- Size: 5.97 MB
- Stars: 2,638
- Watchers: 158
- Forks: 596
- Open Issues: 28
-
Metadata Files:
- Readme: README.EN.md
- License: LICENSE
Awesome Lists containing this project
README
[![Build Status][image-1]][1]
[![License][image-2]]()
**Familia** is an open-source project, which implements three popular topic models based on the large-scale industrial data. They are Latent Dirichlet Allocation(LDA)、SentenceLDA and Topical Word Embedding(TWE). In addition, **Familia** offers several tools including lda-infer and lda-query-doc-sim. **Familia** could be easily applied to many tasks, such as document classification, document clustering and personalized recommendation. Due to the high cost of model training, we will continue to release well-trained topic models based on the various types of large-scale data.
## News!!!
Recently, we launched the Familia's LDA model in [PaddleHub](https://github.com/PaddlePaddle/PaddleHub) 1.8 version. According to different datasets, we launched three LDA models: lda_news, lda_novel, lda_webpage.
PaddleHub is very convenient to use. Let's use lda_news as an example.
1. First, before using PaddleHub, you need to install the PaddlePaddle deep learning framework. For more installation instructions, please refer to [PaddlePaddle Quick Installation] (https://www.paddlepaddle.org.cn/install/quick).
2. Install Paddlehub: `pip install paddlehub`
3. lda_news model installation: `hub install lda_news`
4. API usage
``` python
import paddlehub as hub
lda_news = hub.Module(name="lda_news")
jsd, hd = lda_news.cal_doc_distance(doc_text1="今天的天气如何，适合出去游玩吗", doc_text2="感觉今天的天气不错，可以出去玩一玩了")
# jsd = 0.003109, hd = 0.0573171
lda_sim = lda_news.cal_query_doc_similarity(query='百度搜索引擎', document='百度是全球最大的中文搜索引擎、致力于让网民更便捷地获取信息，找到所求。百度超过千亿的中文网页数据库，可以瞬间找到相关的搜索结果。')
# LDA similarity = 0.06826
results = lda_news.cal_doc_keywords_similarity('百度是全球最大的中文搜索引擎、致力于让网民更便捷地获取信息，找到所求。百度超过千亿的中文网页数据库，可以瞬间找到相关的搜索结果。')
# [{'word': '百度', 'similarity': 0.12943492762349573},
# {'word': '信息', 'similarity': 0.06139783578769882},
# {'word': '找到', 'similarity': 0.055296603463188265},
# {'word': '搜索', 'similarity': 0.04270794098349327},
# {'word': '全球', 'similarity': 0.03773627056367886},
# {'word': '超过', 'similarity': 0.03478658388202199},
# {'word': '相关', 'similarity': 0.026295857219683725},
# {'word': '获取', 'similarity': 0.021313585287833996},
# {'word': '中文', 'similarity': 0.020187103312009513},
# {'word': '搜索引擎', 'similarity': 0.007092890537169911}]
```
More detailed introduction and usage can be found here: https://www.paddlepaddle.org.cn/hublist?filter=en_category&value=SemanticModel
## Introduction
The details of topic models implemented by **Familia** can be referred to [papers on topic models][3].
Generally, the applications adopting topic models are categorized into two parts: Semantic Representation and Semantic Matching.
- **Semantic Representation**
Topic models are able to mine hidden dimensions (topics) from document collection and generate semantic representations of documents. These generated semantic representations can be used as features for document classification, document content analysis, and CTR prediction.
- **Semantic Matching**
We offer two methods to compute semantic similarity between documents:
- Semantic similarity between short-long documents, which can be applied to keyword extraction and computing query-document semantic similarity.
- Semantic similarity between long-long documents, which can be applied to computing semantic similarity between user profile and news article.
More details can be referred to [**Familia Wiki**][4].
## Compilation
The required third parties include `gflags-2.0`，`glogs-0.3.4`，`protobuf-2.5.0`. The complier should support `C++11`, `g++ >= 4.8` and be compatible with linux and mac. The deps could be obtained and installed automatically by running the following script.
$ sh build.sh
## Download
$ cd model
$ sh download_model.sh
More details can be referred to [Models][5].
## Demo
**Familia** demo includes the following functions:
- **Semantic Representation**
utilize topic models to infer the topic distribution of the input document.
- **Semantic Matching**
compute semantic similarity between short-long or long-long documents.
- **Topic Show**
demonstrate top words under each topic for users’ better understanding.
More details can be referred to [Demos][6].
## Tips
* If libglog.so, libgflags.so and other dynamic libraries could not be found, please add third\_party to the environmental parameter `LD_LIBRARY_PATH`.
`export LD_LIBRARY_PATH=./third_party/lib:$LD_LIBRARY_PATH`
## Contact
[Github Issues][7]
{familia} at baidu.com
## Citation
The following article describes the Familia project and industrial cases powered by topic modeling. It bundles and translates the Chinese documentation of the website. We recommend citing this article as default.
Di Jiang, Yuanfeng Song, Rongzhong Lian, Siqi Bao, Jinhua Peng, Huang He and Hua Wu. 2018. [Familia: A Configurable Topic Modeling Framework for Industrial Text Engineering][8]. arXiv preprint arXiv:1808.03733.
@article{jiang2018familia,
author = {Di Jiang and Yuanfeng Song and Rongzhong Lian and Siqi Bao and Jinhua Peng and Huang He and Hua Wu},
title = {{Familia: A Configurable Topic Modeling Framework for Industrial Text Engineering}},
journal = {arXiv preprint arXiv:1808.03733},
year = {2018}
}
Further Reading: [Federated Topic Modeling][11]
## Copyright and License
Familia is provided under the [BSD-3-Clause License][9].
[1]: http://travis-ci.org/baidu/Familia
[3]: https://github.com/baidu/Familia/wiki/%E5%8F%82%E8%80%83%E6%96%87%E7%8C%AE
[4]: https://github.com/baidu/Familia/wiki
[5]: https://github.com/baidu/Familia/blob/master/model/README.md
[6]: https://github.com/baidu/Familia/wiki/Demo%E4%BD%BF%E7%94%A8%E6%96%87%E6%A1%A3
[7]: https://github.com/baidu/Familia/issues
[8]: https://arxiv.org/abs/1808.03733v2
[9]: LICENSE
[11]: https://github.com/baidu/Familia/blob/master/papers/FTM.pdf
[image-1]: https://travis-ci.org/baidu/Familia.svg?branch=master
[image-2]: https://img.shields.io/pypi/l/Django.svg