{"id":19162798,"url":"https://github.com/centre-for-humanities-computing/newsfluxus","last_synced_at":"2026-04-06T06:33:00.767Z","repository":{"id":40959445,"uuid":"269298423","full_name":"centre-for-humanities-computing/newsFluxus","owner":"centre-for-humanities-computing","description":null,"archived":false,"fork":false,"pushed_at":"2023-07-06T22:21:17.000Z","size":3590,"stargazers_count":5,"open_issues_count":2,"forks_count":4,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-09-09T23:59:32.557Z","etag":null,"topics":["digital-humanities","information-theory-and-code","newspaper-analysis"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/centre-for-humanities-computing.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-06-04T08:08:06.000Z","updated_at":"2023-09-13T02:37:35.000Z","dependencies_parsed_at":"2025-04-20T15:15:14.950Z","dependency_job_id":null,"html_url":"https://github.com/centre-for-humanities-computing/newsFluxus","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/centre-for-humanities-computing/newsFluxus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centre-for-humanities-computing%2FnewsFluxus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centre-for-humanities-computing%2FnewsFluxus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centre-for-humanities-computing%2FnewsFluxus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centre-for-humanities-computing%2FnewsFluxus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/centre-for-humanities-computing","download_url":"https://codeload.github.com/centre-for-humanities-computing/newsFluxus/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centre-for-humanities-computing%2FnewsFluxus/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31463014,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-05T21:22:52.476Z","status":"online","status_checked_at":"2026-04-06T02:00:07.287Z","response_time":112,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["digital-humanities","information-theory-and-code","newspaper-analysis"],"created_at":"2024-11-09T09:13:11.800Z","updated_at":"2026-04-06T06:33:00.752Z","avatar_url":"https://github.com/centre-for-humanities-computing.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NewsFluxus #\n\nTool for modelling change and persistence in newspaper content. For an exposition of the underlying method see [Persistent News: The Information Dynamics of Nordic Newspapers](https://centre-for-humanities-computing.github.io/Nordic-Digital-Humanities-Laboratory/portfolio/news_c19_method/) and for design see [News-fluxus design specification](https://github.com/centre-for-humanities-computing/newsFluxus).\n\nPublications:\n\n- K. L. Nielbo, R. B. Baglini, P. B. Vahlstrup, K. C. Enevoldsen, A. Bechmann, and A. Roepstorff, “News Information Decoupling: An Information Signature of Catastrophes in Legacy News Media,” arXiv:2101.02956 [cs].\n\n## Prerequisites\n\nFor running in virtual environment (recommended) and assuming python3.7+ is installed.\n\n```bash\n$ sudo pip3 install virtualenv\n$ virtualenv -p /usr/bin/python3.7 venv\n$ source venv/bin/activate\n```\n\n## Installation\n\nClone repository and install requirements\n\n```bash\n$ git clone https://github.com/centre-for-humanities-computing/newsFluxus.git\n$ pip3 install -r requirements.txt\n```\n\n### GPU acceleration\n\nCurrently the requirements file installs `torch` and `torchvision` without support for GPU acceleration. If you want to use your accelerator(-s) comment out `torch` and `torchvision` in the requirements file, uninstall with pip (if relevant), and run `pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html` for your desired CUDA version (in this case 11.0+).\n\n### Install Mallet\nClone and install Mallet (plus dependencies)\n```sh\n$ sudo apt-get install default-jdk\n$ sudo apt-get install ant\n$ git clone git@github.com:mimno/Mallet.git\n$ cd Mallet/\n$ ant\n```\nChange path the local mallet installation in `src/tekisuto/models/latentsemantics.py`\n\n#### Test Mallet wrapper\n```bash\n\u003e\u003e\u003e from gensim.test.utils import common_corpus, common_dictionary\n\u003e\u003e\u003e from gensim.models.wrappers import LdaMallet\n\n\u003e\u003e\u003e path_to_mallet_binary = \"/path/to/mallet/binary\"\n\u003e\u003e\u003e model = LdaMallet(path_to_mallet_binary, corpus=common_corpus, num_topics=20, id2word=common_dictionary)\n```\n\n### Download language resources\n```bash\n$ python downloader.py --langauge \u003clanguage-code\u003e\n# ex. for Danish langauge resources\n$ python downloader.py --language da\n```\nAnd you will be prompted for location to store data, just use default. To find language codes see [Stanza](https://stanfordnlp.github.io/stanza/available_models.html)\n\n#### Test Stanza Installation\n```\n\u003e\u003e\u003e import stanza\n\n\u003e\u003e\u003e nlp = stanza.Pipeline(lang=\"da\")\n\u003e\u003e\u003e doc = nlp(\"Rap! rap! sagde hun, og så rappede de sig alt hvad de kunne, og så til alle sider under de grønne blade, og moderen lod dem se så meget de ville, for det grønne er godt for øjnene.\")\n\u003e\u003e\u003e doc.sentences[0].print_dependencies()\n```\n\n### Train model and extract signal\n```bash\n$ bash main.sh\n```\n\nAnd individually\n\n```bash\n$ python src/bow_mdl.py --dataset \u003cpath-to-dataset\u003e --language \u003clanguage-code\u003e --bytestore \u003cfrequency-of-backup\u003e --sourcename \u003cname-of-dataset\u003e --estimate \"\u003cstart stop step\u003e\" --verbose \u003cfrequency-of-log\u003e\n$ python src/signal_extraction.py --model \u003cpath-to-serialized-model\u003e\n# ex. for Danish sample\n$ python bow_mdl.py --dataset ../dat/sample.ndjson --language da --bytestore 100 --estimate \"20 50 10\" --sourcename sample --verbose 100\n$ python python src/signal_extraction.py --model mdl/da_sample_model.pcl\n```\n\n### Research use-case\nRequires `matplotlib`\n```bash\n$ python src/news_uncertainty.py --dataset mdl/da_sample_signal.json --window 7 --figure \"fig\"\n```\nresulting visualizations in `fig/`\n\n## Contributing\n\n1. Fork it!\n2. Create your feature branch: `git checkout -b my-new-feature`\n3. Commit your changes: `git commit -am 'Add some feature'`\n4. Push to the branch: `git push origin my-new-feature`\n5. Submit a pull request :smiling_imp:\n\n## Versioning\n\n| Edition | Date | Comment |\n| --- | --- | --- |\n| v1.0 | June 04 2020 | Launch |\n| v1.1 | January 14 2020 | New NLP pipeline |\n\n## Authors\nKristoffer L. Nielbo\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details\n\n## Acknowledgments\n[Stopwords ISO](https://github.com/stopwords-iso) for their multilingual collection of stopwords.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcentre-for-humanities-computing%2Fnewsfluxus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcentre-for-humanities-computing%2Fnewsfluxus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcentre-for-humanities-computing%2Fnewsfluxus/lists"}