https://github.com/kata-ai/indosum
A benchmark dataset for Indonesian text summarization.
https://github.com/kata-ai/indosum
indonesian indonesian-language natural-language-processing text-summarization
Last synced: 4 months ago
JSON representation
A benchmark dataset for Indonesian text summarization.
- Host: GitHub
- URL: https://github.com/kata-ai/indosum
- Owner: kata-ai
- License: apache-2.0
- Created: 2018-09-12T05:41:33.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2019-03-20T02:33:59.000Z (over 6 years ago)
- Last Synced: 2025-04-14T06:46:15.914Z (6 months ago)
- Topics: indonesian, indonesian-language, natural-language-processing, text-summarization
- Language: Python
- Homepage: https://github.com/kata-ai/indosum
- Size: 102 KB
- Stars: 77
- Watchers: 6
- Forks: 15
- Open Issues: 0
-
Metadata Files:
- Readme: README.rst
- License: LICENSE.txt
Awesome Lists containing this project
README
Text Summarization
++++++++++++++++++
This repository contains the code for our work:
Kurniawan, K., & Louvan, S. (2018). IndoSum: A New Benchmark Dataset for Indonesian Text Summarization. In 2018 International Conference on Asian Language Processing (IALP) (pp. 215–220). Bandung, Indonesia: IEEE. https://doi.org/10.1109/IALP.2018.8629109
Requirements
============
Create a virtual environment from ``environment.yml`` file using conda::
$ conda env create -f environment.yml
To run experiments with NeuralSum [CL16]_, Tensorflow is also required.
Dataset
=======
Get the dataset from https://drive.google.com/file/d/1OgYbPfXFAv3TbwP1Qcwt_CC9cVWSJaco/view.
Preprocessing for NeuralSum
---------------------------
For NeuralSum, the dataset should be further preprocessed using ``prep_oracle_neuralsum.py``::
$ ./prep_oracle_neuralsum.py -o neuralsum train.01.jsonl
The command will put the oracle files for NeuralSum under ``neuralsum`` directory. Invoke the script with ``-h/--help`` to see its other options.
Running experiments
===================
The scripts to run the experiments are named ``run_.py``. For instance, to run an experiment using LEAD, the script to use is ``run_lead.py``. All scripts use `Sacred `_ so you can invoke each with ``help`` command to see its usage. The experiment configurations are fully documented. Run ``./run_.py print_config`` to print all the available configurations and their docs.
Training a model
----------------
To train a model, for example the naive Bayes model, run ``print_config`` command first to see the available configurations::
$ ./run_bayes.py print_config
This command will give an output something like::
INFO - summarization-bayes-testrun - Running command 'print_config'
INFO - summarization-bayes-testrun - Started
Configuration (modified, added, typechanged, doc):
cutoff = 0.1 # proportion of words with highest TF-IDF score to be considered important words
idf_path = None # path to a pickle file containing the IDF dictionary
model_path = 'model' # where to load or save the trained model
seed = 313680915 # the random seed for this experiment
corpus:
dev = None # path to dev oracle JSONL file
encoding = 'utf-8' # file encoding
lower = True # whether to lowercase words
remove_puncts = True # whether to remove punctuations
replace_digits = True # whether to replace digits
stopwords_path = None # path to stopwords file, one per each line
test = 'test.jsonl' # path to test oracle JSONL file
train = 'train.jsonl' # path to train oracle JSONL file
eval:
delete_temps = True # whether to delete temp files after finishes
on = 'test' # which corpus set the evaluation should be run on
size = 3 # extract at most this number of sentences as summary
summ:
path = 'test.jsonl' # path to the JSONL file to summarize
size = 3 # extract at most this number of sentences as summary
INFO - summarization-bayes-testrun - Completed after 0:00:00
So, to train the model on a train corpus in ``/tmp/train.jsonl`` and save the model to ``/tmp/models/bayes.model``, simply run::
$ ./run_bayes.py train with corpus.train=/tmp/train.jsonl model_path=/tmp/models/bayes.model
Evaluating a model
------------------
Evaluating an unsupervised model is simple. For example, to evaluate a LEAD-N summarizer::
$ ./run_lead.py evaluate with corpus.test=/tmp/test.jsonl
This command will print an output like this::
INFO - run_experiment - Running command 'evaluate'
INFO - run_experiment - Started
INFO - read_jsonl - Reading test JSONL file from /tmp/test.jsonl
INFO - evaluate - References directory: /var/folders/p9/4pp5smf946q9xtdwyx792cn40000gn/T/tmp7jct3ede
INFO - evaluate - Hypotheses directory: /var/folders/p9/4pp5smf946q9xtdwyx792cn40000gn/T/tmpnaqoav4o
INFO - evaluate - ROUGE scores: {'ROUGE-1-R': 0.71752, 'ROUGE-1-F': 0.63514, 'ROUGE-2-R': 0.62384, 'ROUGE-2-F': 0.5502, 'ROUGE-L-R': 0.70998, 'ROUGE-L-F': 0.62853}
INFO - evaluate - Deleting temporary files and directories
INFO - run_experiment - Result: 0.63514
INFO - run_experiment - Completed after 0:00:11
Evaluating a trained model is done similarly with ``model_path`` configuration is set to the path to the saved model.
Setting up Mongodb observer
---------------------------
Sacred allows the experiments to be observed and saved to a Mongodb database. The experiment scripts above can readily be used for this, simply set two environment variables ``SACRED_MONGO_URL`` and ``SACRED_DB_NAME`` to your Mongodb authentication string and database name (to save the experiments into) respectively. Once set, the experiments will be saved to the database. Use ``-u`` flag when invoking the experiment script to disable saving.
Reproducing results
-------------------
All best configurations obtained from tuning on the development set are saved as Sacred's named configurations. This makes it easy to reproduce our results. For instance, to reproduce our LexRank result on fold 1, simply run::
./run_lexrank.py evaluate with tuned_on_fold1 corpus.test=test.01.jsonl
Since the best configuration is named as ``tuned_on_fold1``, the command above will use that configuration and evaluate the model on the test set. In general, all run scripts have ``tuned_on_foldX`` named configuration, where ``X`` is the fold number. For ``run_neuralsum.py`` though, there are other named configurations, namely ``emb300_on_foldX`` and ``fasttext_on_foldX``, referring to the scenario of using word embedding size of 300 and fastText pretrained embedding respectively. Some run scripts do not have such named configurations; that is because their hyperparameters were not tuned/they do not have any.
License
=======
Apache License, Version 2.0.
Citation
========
If you're using our code or dataset, please cite::
@inproceedings{kurniawan2018,
place={Bandung, Indonesia},
title={IndoSum: A New Benchmark Dataset for Indonesian Text Summarization},
url={https://ieeexplore.ieee.org/document/8629109},
DOI={10.1109/IALP.2018.8629109},
booktitle={2018 International Conference on Asian Language Processing (IALP)},
publisher={IEEE},
author={Kurniawan, Kemal and Louvan, Samuel},
year={2018},
month={Nov},
pages={215-220}
}
.. [CL16] Cheng, J., & Lapata, M. (2016). Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (pp. 484–494). Berlin, Germany: Association for Computational Linguistics. Retrieved from http://www.aclweb.org/anthology/P16-1046