{"id":18243966,"url":"https://github.com/rybesh/topics","last_synced_at":"2025-04-04T13:31:03.316Z","repository":{"id":39978912,"uuid":"343259594","full_name":"rybesh/topics","owner":"rybesh","description":"workflow for topic modeling a personal library of PDFs","archived":false,"fork":false,"pushed_at":"2023-03-10T05:57:30.000Z","size":74,"stargazers_count":3,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2023-04-18T01:41:55.267Z","etag":null,"topics":["mallet","pdf","pyldavis","topic-modeling"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rybesh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-01T01:45:54.000Z","updated_at":"2023-01-25T16:07:08.000Z","dependencies_parsed_at":"2023-02-14T09:16:38.095Z","dependency_job_id":null,"html_url":"https://github.com/rybesh/topics","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rybesh%2Ftopics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rybesh%2Ftopics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rybesh%2Ftopics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rybesh%2Ftopics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rybesh","download_url":"https://codeload.github.com/rybesh/topics/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223143815,"owners_count":17094721,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mallet","pdf","pyldavis","topic-modeling"],"created_at":"2024-11-05T09:04:26.047Z","updated_at":"2024-11-05T09:04:26.590Z","avatar_url":"https://github.com/rybesh.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Topic model a library of PDFs\n\nThis is a Unix-style workflow for [topic modeling](https://www.youtube.com/watch?v=gN2x_KjJI1o) a personal\nlibrary of PDFs using [MALLET](http://mallet.cs.umass.edu). It should work on macOS and Linux.\n\nIf you find something in these instructions unclear, or something\ndoesn't work for you, or you have a suggestion for improving this\nworkflow, please [file an issue](https://github.com/rybesh/topics/issues).\n\n## Prerequisites\n\nThese instructions assume some basic familiarity with the macOS\nterminal or Linux shell.\n\nDepending on how many PDFs you have, you may need a computer that has\na good amount of RAM and a CPU with multiple cores. On my 2019 laptop\nwith a 8 core Intel CPU and 16GB of RAM, building and visualizing a\n50-topic model of around 5,000 PDFs, having a total of about 20\nmillion words, takes about 15 minutes. Building and visualizing a\n200-topic model takes about an hour. (Most of that time is spent\ngenerating the visualization, not building the topic model.)\n\nIf you have more PDFs than that, or don't have that many cores or that\nmuch RAM, you might want to run this on an on-demand cloud compute\nserver—but doing that is beyond the scope of these instructions.\n\nWherever you run this, you will need recent versions of the following\nsoftware: [GNU make](https://www.gnu.org/software/make/) (version 4.3 or later), [Python](https://www.python.org) (version\n3.8 or later), [pdftotext](https://en.wikipedia.org/wiki/Pdftotext), [git](https://git-scm.com), and [ant](https://ant.apache.org).\n\n(It probably will work with an earlier version of Python 3, but I\nhaven't tested this. If you want to try it with an earlier version,\nset the value of `NEED_PYTHON_VERSION` in the [Makefile](https://github.com/rybesh/topics/blob/main/Makefile#L8) to that\nversion, and then please let me know if it works by [filing an\nissue](https://github.com/rybesh/topics/issues).)\n\nOn macOS, all of these can be installed using [Homebrew](https://brew.sh). After\ninstalling Homebrew, execute the following command in the terminal to\ninstall all the prerequisites:\n\n```\nbrew install make python3 poppler git ant\n```\n\n(`poppler` is the name of the package that installs `pdftotext`.)\n\nOn Ubuntu Linux 20.10 (Groovy Gorilla) or later, these prerequisites\ncan be installed using `apt-get`:\n\n```\nsudo apt-get install make python3 python3-venv poppler-utils git ant\n```\n\nNote that on Ubuntu you must install `python3-venv` in addition to\n`python3`.\n\n## Make a local copy of this repository\n\nEither [clone this repository](https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/cloning-a-repository), or [download the ZIP file](https://github.com/rybesh/topics/archive/main.zip) of\nthis repository and unzip it someplace. Either way you should end up\nwith a directory called `topics` on your computer.\n\n## Collect your PDFs\n\nVisit the `topics` directory in your terminal or shell:\n\n```\ncd /wherever/you/put/topics\n```\n\nCreate a subdirectory called `pdf`:\n\n```\nmkdir pdf\n```\n\nThen you can put all your PDFs in this subdirectory. If you don't want\nto disturb the organization of your PDFs, you can leave them where\nthey are and use [symbolic links](https://wiki.debian.org/SymLink) to point to them. For example, if\nyou keep your PDFs in `~/Documents/Papers` and `~/Desktop/TO-READ`\n\n```\ncd pdf\nln -s ~/Documents/Papers\nln -s ~/Desktop/Papers\n```\n\nThe directory structure inside `pdf` (or inside the directories\npointed to by symbolic links within `pdf`) doesn't matter. Any files\nending in `.pdf`, no matter how deeply nested, will be found and\nindexed and all other files will be ignored.\n\nIf there are PDFs you want to exclude from your topic models, you can\ncreate a file called `exclude.txt` in the `topics` directory, and list\npatterns there matching any filenames you want to exclude. For example\nif your `exclude.txt` file contained:\n\n```\nmichel foucault\ndraft\n```\n\n… then any PDFs with filenames containing the name “Michel Foucault”\nor the word “draft” will be excluded. You can also use [basic regular\nexpressions](https://www.gnu.org/software/grep/manual/html_node/Basic-vs-Extended.html). Matching is case-insensitive.\n\n## Build your topic models\n\nAssuming you’re still visiting the `topics` directory in your terminal\nor shell, you can build a topic model with 50 topics with the command:\n\n```\n./build 50-topics\n```\n\nThis will\n\n1. convert all your PDFs to plain text files in a subdirectory called `txt`\n1. download and install the [MALLET](http://mallet.cs.umass.edu/) topic modeling software\n1. build a topic model with 50 topics\n1. create an interactive visualization of the topics using [pyLDAvis](https://github.com/bmabey/pyLDAvis)\n1. create a set of web pages for browsing the model\n\n`50` can be changed to whatever you want; to build a model with 111\ntopics use the command:\n\n```\n./build 111-topics\n```\n\n“Hyperparameter optimization” is a MALLET option that allows some\ntopics to be more prominent than others, which can sometimes result in\nbetter models. To build an optimized model, put the word `-optimized`\nafter the number of topics you want:\n\n```\n./build 50-optimized-topics\n```\n\nYou can build multiple topic models with one command (each will be\nbuilt sequentially):\n\n```\n./build 50-topics 50-optimized-topics 100-topics 100-optimized-topics\n```\n\nBy default MALLET will use 12G of RAM; if you don't have that much (or\nyou have more) you can specify how much RAM to use:\n\n```\nMEMORY=8g ./build 50-topics\n```\n\nNote that the less RAM MALLET uses, the more time topic modeling will\ntake (and the more it uses, the less time it will take).\n\n## Browse your topic models\n\nRun a local web server for browsing the topic models with the command:\n\n```\nmake serve\n```\n\nThis will run a server on http://localhost:5555, which you can visit\nwith a web browser. If something else is already running on port 5555,\nyou can change the port:\n\n```\nPORT=12345 make serve\n```\n\nWhen you visit http://localhost:5555 you'll see links to each of the\nmodels you've built. For each model, you can see\n\n* lists of the top documents per topic\n* a visualization of the topic structure\n* diagnostics tool for troubleshooting the model\n\nThe lists of the top documents per topic show the strength of\nassociation with the topic in a column along the left side of the\npage. Clicking on a document in the list will open it in the\nbrowser. For documents that are in the list of top documents for more\nthan one topic, links to the other topics appear after the link to the\ndocument.\n\nYou can find a specific document in the models you've built by using\nthe `topics` shell script in the `topics` directory. Give it the path\nto a PDF, and it will print the URL of and (if it can) open in a\nbrowser every top documents list in which that document appears:\n\n```\n./topics pdf/Desktop/TO-READ/an-interesting-article.pdf\n```\n\nNote that if you change the port on which the local web server runs\nfrom the default of `5555`, you'll need to rebuild the shell scripts\nfor finding specific documents. Assuming you've built topic models\nwith 50, 100, and 200 topics, you can delete and rebuild the shell\nscripts as follows:\n\n```\nrm -f topics 50-topics 100-topics 200-topics\n./build 50-topics 100-topics 200-topics\n```\n\nFor an explanation of the topic structure visualization, see the\n[pyLDAvis documentation](https://pyldavis.readthedocs.io/en/latest/readme.html#usage).\n\nFor an explanation of the diagnostics tool, see the [MALLET\ndocumentation](http://mallet.cs.umass.edu/diagnostics.php).\n\n## Rebuild your topic models\n\nIf you add new PDFs to your PDF directories, you'll need to rebuild\nyour topic models. To do this use the command:\n\n```\n./rebuild 50-optimized-topics\n```\n\n**This will delete all existing models** and then build an optimized\n50-topic model.\n\nIf you only want to rebuild a specific model, without deleting all\nexisting models, you can delete just that model and then build it:\n\n```\n./delete 50-topics\n./build 50-topics\n```\n\nNote that if you've added new PDFs, only the models that are rebuilt\nwill incorporate them, so most of the time you're going to want to use\n`rebuild` to rebuild all of your models.\n\nThese commands (`rebuild` or `delete` followed by `build`) will only\nconvert the new PDFs to plain text, skipping any PDFs that have\nalready been converted. If you want to start completely from scratch,\nincluding re-converting all your PDFs and re-installing MALLET and\npyLDAvis, use the command:\n\n```\nmake superclean 50-topics\n```\n\nNote that **you will not get the same set of topics** when you rebuild\nyour models! This is expected: topic modeling involves random\nsampling, which produces different (but comparable) results each time.\n\nFor exploring a library of PDFs, this is a feature, not a bug: you can\nrebuild your topic models several times, looking to see what kinds of\ninteresting clusters are turned up.\n\nHowever, if you need reproducibility, when you build your topic models\nyou can specify a [random seed](https://en.wikipedia.org/wiki/Random_seed) for MALLET to use as follows:\n\n```\nRANDOM_SEED=7 ./build 50-topics\n```\n\nThe value of `RANDOM_SEED` can be any positive integer (not zero). \n\nIf you use the same `RANDOM_SEED` value every time you build the\nmodels, you should get the same results each time.\n\n## Troubleshooting\n\nSome PDFs, such as scanned PDFs that have not had OCR run on them, may\nnot have any readable text in them, meaning that the plain text\nversions of these files will be empty. You can check if this is the\ncase for any of your PDF files by running this command:\n\n```\nmake wordcounts.csv\n```\n\nThis will generate a [CSV](https://simple.wikipedia.org/wiki/Comma-separated_values) file listing the word count for each\nplain text file created from your PDFs. Open the CSV in the\nspreadsheet software of your choice and sort by word count. Files with\nzero or very few words should be checked to see if the PDF needs to\nhave OCR run on it.\n\nYou may find that one or more of your topics seems to consist of\ngibberish. This is usually a sign that OCR has failed, producing a\nbunch of unreadable symbols instead of readable text. This is often\ndue to a problem with embedded fonts in the PDF file. Using Adobe\nAcrobat Pro DC to [convert fonts to outlines](https://www.copperbottomdesign.com/blog/converting-fonts-to-outlines) before running OCR\ncan fix this.\n\nYou can use the diagnostics tool to find other kinds of “problematic”\ntopics—see the [MALLET documentation](http://mallet.cs.umass.edu/diagnostics.php).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frybesh%2Ftopics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frybesh%2Ftopics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frybesh%2Ftopics/lists"}