{"id":19162869,"url":"https://github.com/centre-for-humanities-computing/fabula_pipeline","last_synced_at":"2026-06-24T00:31:03.871Z","repository":{"id":208821991,"uuid":"688452840","full_name":"centre-for-humanities-computing/fabula_pipeline","owner":"centre-for-humanities-computing","description":null,"archived":false,"fork":false,"pushed_at":"2024-01-25T13:50:34.000Z","size":1121,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-09T23:59:51.927Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/centre-for-humanities-computing.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-07T11:34:31.000Z","updated_at":"2025-06-30T11:24:35.000Z","dependencies_parsed_at":"2023-11-23T13:42:58.829Z","dependency_job_id":"19d0616f-27e7-41f3-a083-5f48b899c16d","html_url":"https://github.com/centre-for-humanities-computing/fabula_pipeline","commit_stats":null,"previous_names":["centre-for-humanities-computing/fabula_pipeline"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/centre-for-humanities-computing/fabula_pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centre-for-humanities-computing%2Ffabula_pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centre-for-humanities-computing%2Ffabula_pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centre-for-humanities-computing%2Ffabula_pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centre-for-humanities-computing%2Ffabula_pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/centre-for-humanities-computing","download_url":"https://codeload.github.com/centre-for-humanities-computing/fabula_pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centre-for-humanities-computing%2Ffabula_pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34712578,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-23T02:00:07.161Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T09:13:23.502Z","updated_at":"2026-06-24T00:31:03.848Z","avatar_url":"https://github.com/centre-for-humanities-computing.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fabula_pipeline\n\n[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/centre-for-humanities-computing/fabula_pipeline/blob/main/src/pipe_notebook.ipynb)\n\n\n## tl;dr\nThis pipeline takes a folder containing books in .txt format and outputs a JSON-file containing the different literary features developed / used by the Fabula-NET team at Aarhus University.\nCurrently only English and Danish are supported. \n\nFeatures given no matter the language: \n\n- Stylometrics:\n    - word count, mean word length, MSTTR\n    - mean sentence length, GZIPR, BZIPR\n- Bigram entropy and word entropy\n- Sentiment arc:\n    - mean \u0026 standard deviation\n    - mean sentiment for each segment (arc divided into 20 segments)\n    - mean sentiment for first 10%, mean sentiment for last 10% \n    - difference in sentiment between last 10% and the rest of the sentiment arc\n    - Approximate entropy\n    - Hurst\n\nFeatures given only for English: \n- Readability:\n    - flesch grade, flesch ease, smog, ari, dale chall new\n- Roget Categories\n\n\nAdditionally, for each text a CSV-file is produced that contains the token attributes from SpaCy which is saved in a folder called 'spacy_books/' within the specified output folder.\n\n\n## Getting Started\n\nInstall all requirements for the pipeline script.\n\n```bash\npip install -r requirements.txt\n```\n\nThe pipeline scripts assumes that the books, which should go through the pipeline, are in individual .txt-files with unique filenames, e.g.,: \n\n    book_files\n        \\_ 00000001.txt\n        \\_ 00000002.txt\n        \\_ 00000003.txt\n        ..\n        ..\n\n\n## How to run the pipeline\n\nThe pipeline has five command-line argument:\n1. input directory (`--in_dir`)\n2. output directory (`--out_dir`)\n3. language (`-lang`)\n4. sentiment method (`--sentiment_method` or `sent`)\n5. ucloud (`--ucloud`)\n\n\n`--in_dir` should point to the folder containing the book files. \n\n`--out_dir` specifies the folder the book features and SpaCy attributes are saved to. It defaults to a folder called `output/` (which will be created if it does not exist already), but can be used to point to a different folder.\n\n`-lang` specifies which language the books are in. For now only `english` and `danish` are supported. `english` is the deafult option.\n\n`--sentiment_method` specifies which method should be used for sentiment analysis for each sentence in the book. For now `afinn`, `vader`, `syuzhet`, and `avg_vader_syuzhet` are supported. The last options takes the mean of the two different methods for each sentence. If language is set to English, the sentiment method has to be `afinn`. \n\n`--ucloud` is a flag that is used if the code is run on the cloud computing service UCloud. This is because the Syuzhet sentiment analysis does not currently work on UCloud. \n\n\nTo run the pipeline, go into the run_pipe.sh and set the command-line arguments to the desired values. Afterwards, the script can be run like this:\n\n```bash\nbash run_pipe.sh\n```\n\nThe pipeline can of course also be run regularly in the command-line: \n\n```bash\npython3 src/pipeline.py --in_dir your/data/path/ --out_dir your/out/path/ -lang \"danish\"\n```\n\n## Output\nThe pipeline script will create a JSON-file in the `--out_dir` folder called `books_features.json`.\n\nIt is a nested dictionary, where the top-level key is the filename wihtout the file extension, and the value is a dictionary where the keys are the name of the features and the values are the results for the features. \nE.g.,\n\n    {\"00000001\": \n        {word_count: int, \n            average_wordlen: float,\n            msttr: float,\n            average_sentlen: float,\n            gzipr: float,\n            bzipr: float,\n            bigram_entropy: float,\n            flesch_grade: float,\n            mean_sentiment: float,\n            std_sentiment: float,\n            mean_sentiment_per_segment: list,\n            mean_sentiment_first_ten_percent: float,\n            mean_sentiment_last_ten_percent: float,\n            difference_lastten_therest: float,\n            approximate_entropy: list,\n            hurst: float,\n            flesch_ease: float,\n            smog: float,\n            ari: float,\n            dale_chall_new: float,\n            arc: list\n            }\n        }\n\nIt will also create a folder called spacy_books/ within the `--out_dir` folder, where a CSV-file for each book is saved containing the SpaCy token attributes for each token in the book. \n \n## Future implementations \n\n- Making an optional argument, that specifies whether goodreads features should be run as well\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcentre-for-humanities-computing%2Ffabula_pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcentre-for-humanities-computing%2Ffabula_pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcentre-for-humanities-computing%2Ffabula_pipeline/lists"}