{"id":28780669,"url":"https://github.com/nlesc/cptm","last_synced_at":"2026-04-09T10:34:15.175Z","repository":{"id":145997557,"uuid":"53065279","full_name":"NLeSC/cptm","owner":"NLeSC","description":"Cross-Perspective Topic Modeling","archived":false,"fork":false,"pushed_at":"2017-10-27T13:36:15.000Z","size":211,"stargazers_count":11,"open_issues_count":2,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-04-14T04:29:36.156Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NLeSC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2016-03-03T16:24:06.000Z","updated_at":"2023-10-13T06:25:37.000Z","dependencies_parsed_at":null,"dependency_job_id":"1ae2488f-ee8b-4561-9da3-cb355193f6e3","html_url":"https://github.com/NLeSC/cptm","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/NLeSC/cptm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NLeSC%2Fcptm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NLeSC%2Fcptm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NLeSC%2Fcptm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NLeSC%2Fcptm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NLeSC","download_url":"https://codeload.github.com/NLeSC/cptm/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NLeSC%2Fcptm/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260415151,"owners_count":23005508,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-17T18:07:38.572Z","updated_at":"2026-04-09T10:34:15.170Z","avatar_url":"https://github.com/NLeSC.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![DOI](https://zenodo.org/badge/doi/10.5281/zenodo.47756.svg)](http://dx.doi.org/10.5281/zenodo.47756) [![Build Status](https://travis-ci.org/NLeSC/cptm.svg?branch=develop)](https://travis-ci.org/NLeSC/cptm.svg?branch=develop)\n\n# Cross-Perspective Topic Modeling\n\nA Gibbs sampler to do Cross-Perspective Topic Modeling, as described in\n\n\u003e Fang, Si, Somasundaram, \u0026 Yu (2012). Mining Contrastive Opinions on Political Texts using Cross-Perspective Topic Model. In proceedings of the fifth ACM international conference on Web Search and Data Mining. http://dl.acm.org/citation.cfm?id=2124306\n\n## Installation\n\nInstall prerequisites.\n\n    sudo apt-get install gfortran libopenblas-dev liblapack-dev\n\nClone the repository.\n\n    git clone git@github.com:NLeSC/cptm.git\n    cd cptm\n\nInstall the requirements (in virtual environment if desired).\n\n    pip install -r requirements.txt\n\nInstall cptm (compiles Cython code).\n\n    python setup.py install\n\nAdd the cptm directory to the `PYTHONPATH` (otherwise the scripts don't work).\n\n    export PYTHONPATH=$PYTHONPATH:.\n\nTests can be run with `nosetests` (don't forget to `pip install nose` if you're using a virtual environment).\n\n## Getting and preparing the Dutch parliamentary proceedings\n\nDownload the data.\n\n    ./get_data.sh /path/to/xml/data/dir\n\nCreate sets of text documents for different perspectives.\n\n    python folia2cpt_input.py /path/to/xml/data/dir /path/to/perspectives/dir\n\nThe script expects the directory structure generated by the `get_data.sh`\nscript. When the script finishes, there will be different directories in the\nperspectives dir. Each directory is a division of the data using different\nperspectives. The following perspectives are generated:\n\n* `gov_opp`: Government vs. Opposition. The division is based on the data in\n`data/dutch_coalitions.csv`\n* `parties`: a perspective for each political party found in the data (noisy)\n* `cabinets`: a perspective for each cabinet (based on the data\n`data/dutch_coalitions.csv`)\n* `cabinets-gov_opp`: a perspective for each cabinet divided by\nGovernment/Opposition (based on the data in `data/dutch_coalitions.csv`)\n\n## Running experiments\n\nExperiments are configured using a json object. An example json file can be\ncopied from `data/config.json.example`\n\n    cp data/config.json.example /path/experiment.json\n\nThe json object looks like:\n\n    {\n        \"inputData\": \"/path/to/input/data/*\",\n        \"outDir\": \"/path/to/output/directory/{}\",\n        \"testSplit\": 20,\n        \"nIter\": 200,\n        \"beta\": 0.02,\n        \"beta_o\": 0.02,\n        \"expNumTopics\": [20, 40, 60, 80, 100, 120, 140, 160, 180, 200],\n        \"nProcesses\": 3,\n        \"nTopics\": 100,\n        \"topicLines\": [0],\n        \"opinionLines\": [1, 2, 3],\n        \"sampleEstimateStart\": 80,\n        \"sampleEstimateEnd\": 199,\n        \"minFreq\": 5,\n        \"removeTopTF\": 100,\n        \"removeTopDF\": 100\n    }\n\nOptions\n\n* `inputData`: directory containing data separated by perspective (should end with \\*),\ne.g., `/path/to/perspectives/perspective/*` (where `perspective` is one of `gov_opp`,\n`parties`, `cabinets`, `cabinets-gov_opp`)  \n* `outDir`: directory where parameter estimates and other results will be saved (should\nend with `{}`)  \n* `testSplit`: percentage of the data used for calculating perplexity  \n* `nIter`: number of sampling iterations  \n* `beta`: beta parameter (topics)  \n* `beta_o`: beta parameter (opinions)  \n* `expNumTopics`: list of numbers of topics (e.g., `[20, 30]` means two experiments\nwill be run, one for # topics = 20 and one for # topics = 30\n(script: `experiment_number_of_topics.py`))\n* `nProcesses`: the number of processes the script can use (experiments will be run\nin parallel if possible)  \n* `nTopics`: the number of topics for which results will be calculated (scripts:\n`experiment_calculate_perplexity.py`, `experiment_calculate_perspective_jsd.py`,\n`experiment_find_contrastive_opinions.py`, `experiment_generate_results_nb.py`,\n`experiment_get_results.py`, `experiment_jsd_opinions.py`, and\n`experiment_number_of_topics.py`.  \n* `topicLines`: line number(s) in input files containing topic words  \n* `opinionLines`: line number(s) in input files containing opinion words  \n* `sampleEstimateStart`: the relevant parameters are estimated from the samples that\nare saved during each iteration. `sampleEstimateStart` is the iteration number where\nto start estimating  \n* `sampleEstimateEnd`: the relevant parameters are estimated from the samples that\nare saved during each iteration. `sampleEstimateEnd` is the last iteration number\nthat is used to calculate results (\u003c`nIter`).\n* `minFreq`: minimal term frequency (terms occuring less frequently will be\nremoved from the vocabularies)  \n* `removeTopTF`: the number of terms removed from the vocabularies based on term\nfrequency (terms are ordered by term frequency, next the top X is removed)  \n* `removeTopDF`: the number of terms removed from the vocabularies based on\ndocument frequency (terms are ordered by document frequency, next the top X is\nremoved)  \n\n### Experiment scripts\n\nFirst, run an experiment with different numbers of topics:\n\n    python cptm/experiment_number_of_topics.py /path/to/experiment.json\n\nNext, calculate opinion perplexity:\n\n    python cptm/experiment_calculate_perplexity.py /path/to/experiment.json\n\nTo generate an iPython notebook to inspect the results of an experiment:\n\n    python cptm/experiment_generate_results_nb.py   /path/to/dir/with/results/ experimentName /path/to/resulting/notebook.ipynb\n\nThe notebook helps to determine the 'optimal' number of topics for the data and\nto choose appropriate `sampleEstimateStart` and `sampleEstimateEnd`. These\nparameters are required to generate estimates of `theta`, `phi topics`, and\n`phi opionions`.\n\nSet the `nTopics`, `sampleEstimateStart`, and `sampleEstimateEnd` parameters in\nthe experiment configuration file. Next, generate esitmates of `theta`,\n`phi topics`, and `phi opionions`:\n\n    python cptm/experiment_get_results.py /path/to/experiment.json\n\nNow you can go back to the iPython notebook to have a look at the topics and\nopinions.\n\nThe notebook prints the top 5 topic words for all topics and the top 5 of\ncorresponding opinion words for each perspective. By default, the topics are\nordered by topic number. They can also be ordered by Jensen-Shannon divergence\nof the opinions. That requires calculating the Jensen-Shannon divergences:\n\n    python cptm/experiment_jsd_opinions.py /path/to/experiment.json\n\n[Fang et al. 2012] describes contrastive opinion modeling, a method to\ndetermine opinions for individual topic words. To do contrastive opinion\nmodeling for all topic words (and save the results on disk), run:\n\n    python cptm/experiment_find_contrastive_opinions.py /path/to/experiment.json\n    [-p \u003clist of perspectives\u003e] [-o /path/to/output]\n\nThe `\u003clist of perspectives\u003e` should be formatted like: `\"['Kok II-ChristenUnie',\n'Kok II-CDA', 'Kok II-LPF', 'Kok II-PvdA', 'Kok II-SGP', 'Kok II-D66',\n'Kok II-GroenLinks', 'Kok II-VVD', 'Kok II-SP']\"` (including the double quotes).\n\nThere are some additional scripts:\n\n* `experiment_calculate_perspective_jsd.py`\ncalculates the pairwise average jsd between perspectives for all topics:\n\n    python experiment_calculate_perspective_jsd.py experiment.json\n\n* `experiment_prune_samples.py` removes saved parameter samples (generated by the\nGibbs sampler) for certain iterations. Before, the Gibbs sampler saved estimates\nfor all iterations. However, because this took to much disk space, now the\nsampler only saves every tenth estimate. The `experiment_prune_samples` script\nremoves samples for results generated with an old version of the sampler:\n\n    python experiment_prune_samples.py /path/to/experiment.json\n\n* `experiment_manifesto.py` calculates opinion word perplexity per document for\na set of text documents. The corpus is not divided in perspectives. (This script\nis used to estimate the likelihood of party manifestos given opinions for the\ndifferent perspectives (party manifestos come from the manifesto project)) First\nrun `manifestoproject2cptm_input.py` to create a cptm corpus that can be used\nas input:\n\n    python experiment_manifesto.py \u003cexperiment.json\u003e \\\u003cinput dir\u003e \\\u003coutput dir\u003e\n\n* `experiment_theta_for_texts_perspectives.py` extracts a document/topic\nmatrix for a set of text documents. The corpus is not divided in perspectives.\nThis script is used to calculate theta for the CAP vragenuurtje data. First\nrun `tabular2cptm_input.py` to create a cptm corpus that can be used\nas input:\n\n    python experiment_theta_for_texts_perspectives.py \u003cexperiment.json\u003e \\\u003cinput dir\u003e \\\u003coutput dir\u003e\n\n* `experiment_corr_pca_ches.py` calculate correlations between PCA projections\nand CHES rankings:\n\n    python experiment_corr_pca_ches.py \u003cexperiment.json\u003e \u003cinpt ches data\u003e [-o /path/to/output]\n\n\n* `experiment_cptcorpus_count_words.py` counts the number of topic and opinion\nwords in the corpus:\n\n    python experiment_cptcorpus_count_words.py \u003cexperiment.json\u003e\n\n\n### Other scripts\n\n* `corpusstatistics.py`\n\nPrints some corpus statistics (such as the number of documents in the dataset).\n\n    python corpusstatistics.py \u003cpath to raw data files\u003e \u003cexperiment.json\u003e\n\n* `folia_party_names.py`\n\nExtract names of political parties from the Folia files.\n\n    python folia_party_names.py \u003cpath to raw data files\u003e\n\n* `generateCPTCorpus.py`\n\nScript that generates a (synthetic) corpus to test the CPT model. This script is\nused in the tests.\n\n    Usage: python generateCPTCorpus.py \u003cout dir\u003e\n\n* `manifestoproject2cptm_input.py`\n\nCreate input files in cptm format from manifesto project csv files\n\n    python manifestoproject2cptm_input.py \u003cinput dir\u003e \u003coutput dir\u003e\n\nThe input dir should contain the manifesto project cvs files.\n\n* `tabular2cptm_input.py`\n\nScript that converts a field in a tabular data file to cptm input files.\n\n    python tabular2cpt_input.py \u003ccsv of excel file\u003e \u003cfull text field name\u003e\n\u003cdir out\u003e\n\n## cptm functionality\n\n### Saving CPTCorpus to disk\n\n    from CPTCorpus import CPTCorpus\n\n    corpus = CPTCorpus(files, testSplit=20)\n    corpus.save('/path/to/corpus.json')\n\n### Loading CPTCorpus from disk\n\n    from CPTCorpus import CPTCorpus\n\n    corpus2 = CPTCorpus.load('/path/to/corpus.json')\n\n---\nCopyright Netherlands eScience Center.\n\nDistributed under the terms of the Apache2 license. See LICENSE for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnlesc%2Fcptm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnlesc%2Fcptm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnlesc%2Fcptm/lists"}