{"id":34017119,"url":"https://github.com/terencicp/mastodon-topics","last_synced_at":"2026-04-21T12:02:55.436Z","repository":{"id":327848267,"uuid":"1107608638","full_name":"terencicp/mastodon-topics","owner":"terencicp","description":"Topic modeling of Mastodon posts using BERTopic and LLM summarization. Results can be viewed in a Streamlit app.","archived":false,"fork":false,"pushed_at":"2025-12-08T05:35:13.000Z","size":329,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-09T22:56:09.595Z","etag":null,"topics":["bertopic","mastodon","streamlit","summarization","topic-modeling"],"latest_commit_sha":null,"homepage":"https://mastodon-topics.streamlit.app/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/terencicp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-01T11:24:05.000Z","updated_at":"2025-12-08T05:35:16.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/terencicp/mastodon-topics","commit_stats":null,"previous_names":["terencicp/mastodon-topics"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/terencicp/mastodon-topics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terencicp%2Fmastodon-topics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terencicp%2Fmastodon-topics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terencicp%2Fmastodon-topics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terencicp%2Fmastodon-topics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/terencicp","download_url":"https://codeload.github.com/terencicp/mastodon-topics/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terencicp%2Fmastodon-topics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27707004,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-13T02:00:09.769Z","response_time":147,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bertopic","mastodon","streamlit","summarization","topic-modeling"],"created_at":"2025-12-13T14:26:52.202Z","updated_at":"2025-12-13T14:26:53.046Z","avatar_url":"https://github.com/terencicp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mastodon topic modeling\n\nThis project implements an end-to-end pipeline for discovering and summarizing popular topics on the Mastodon social network. It combines BERTopic for topic modeling with large language models to generate interpretable summaries, providing daily insights into what the Mastodon community is discussing.\n\nView the results at [mastodon-topics.streamlit.app](https://mastodon-topics.streamlit.app/)\n\nView the development notebooks at [github.com/terencicp/social-network-topic-modeling](https://github.com/terencicp/social-network-topic-modeling)\n\n## Architecture\n\nThe pipeline consists of three main components:\n- **pipeline/collect.py**: Runs continuously to collect Mastodon posts\n- **pipeline/run_pipeline.py**: Runs once daily to generate topic summaries\n- **app/streamlit.py**: Used to visualize the results of run_pipeline.py\n\n![Pipeline diagram](./pipeline-diagram.svg)\n\nThe main files are:\n\n**collect.py** - Data collection\n   - Fetches public posts from mastodon.social API\n   - Anonymizes data and stores it in MongoDB\n\n**preprocess.py** - Data preprocessing\n   - Filters english-language posts\n   - Aggregates textual content from multiple fields\n\n**model.py** - Topic modeling\n   - Generates Gemma-300m embeddings\n   - Trains BERTopic models with optimized parameters\n\n**summarize.py** - LLM summarization\n   - Selects top topics by popularity\n   - Generates structured summaries using Qwen3 8b\n\n**streamlit.py** - Streamlit app\n   - Displays topic summaries\n\n## Requirements\n\n### System dependencies\n- Python 3.11\n- MongoDB\n- Ollama\n\n### Python dependencies\n- Pipeline: pipeline/requirements.txt\n- Streamlit: app/requirements.txt\n\n### Environment variables\n- MASTODON_HASHING_KEY for data anonymization\n- HUGGINGFACE_TOKEN from your HuggingFace account for Gemma embeddings\n\n## Acknowledgments\n\nThis project builds upon the work of several amazing open-source projects and research teams:\n\n- **[Mastodon](https://joinmastodon.org/)**: decentralized social network\n- **[Gemma embeddings](https://ai.google.dev/gemma)**: embedding models from Google DeepMind\n- **[BERTopic](https://maartengr.github.io/BERTopic/)**: topic modeling framework by Maarten Grootendorst\n- **[Qwen3](https://qwen.ai/)**: LLM from Alibaba Cloud\n\nPlease make a donation to Mastodon if you use its API to offset the server costs.\n\n## Contact\n\nLinkedIn: [Terenci Claramunt](https://www.linkedin.com/in/terenci/)\n\nThis project was developed as part of my Applied Data Science Degree thesis.\n\nFeel free to reach out for questions about the methodology or implementation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fterencicp%2Fmastodon-topics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fterencicp%2Fmastodon-topics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fterencicp%2Fmastodon-topics/lists"}