{"id":22706354,"url":"https://github.com/kmkurn/msc-project","last_synced_at":"2025-03-29T20:42:28.897Z","repository":{"id":68133749,"uuid":"100644698","full_name":"kmkurn/msc-project","owner":"kmkurn","description":"Exploring Recurrent Neural Network Grammars for Parsing Low-Resource Languages. MSc Project. University of Edinburgh.","archived":false,"fork":false,"pushed_at":"2017-08-21T13:27:53.000Z","size":1753,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-04T21:41:59.039Z","etag":null,"topics":["natural-language-processing","neural-network","syntactic-parser"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kmkurn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-08-17T20:56:48.000Z","updated_at":"2020-12-08T08:42:24.000Z","dependencies_parsed_at":"2023-03-13T20:28:12.040Z","dependency_job_id":null,"html_url":"https://github.com/kmkurn/msc-project","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kmkurn%2Fmsc-project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kmkurn%2Fmsc-project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kmkurn%2Fmsc-project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kmkurn%2Fmsc-project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kmkurn","download_url":"https://codeload.github.com/kmkurn/msc-project/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246243564,"owners_count":20746307,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["natural-language-processing","neural-network","syntactic-parser"],"created_at":"2024-12-10T10:08:23.703Z","updated_at":"2025-03-29T20:42:28.885Z","avatar_url":"https://github.com/kmkurn.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Exploring Recurrent Neural Network Grammars for Parsing Low-Resource Languages\n\nThis repository contains an implementation for my final project [*Exploring Recurrent Neural Network Grammars for Parsing Low-Resource Languages*](http://kkurniawan.com/assets/msc-thesis.pdf), MSc in Artificial Intelligence, University of Edinburgh. The implementation extended the [original recurrent neural network grammars by Dyer et al. (2016)](https://github.com/clab/rnng/). Before put under this repository, their code was forked and modified under [this repository](http://github.com/kmkurn/rnng). Any changes from the original code are thus can be viewed in that repository's commit history. This `README` file is also largely based on theirs.\n\n## Prerequisites\n\n * Python 2.7\n * A C++ compiler supporting the [C++11 language standard](https://en.wikipedia.org/wiki/C%2B%2B11)\n * [Boost](http://www.boost.org/) libraries\n * [Eigen](http://eigen.tuxfamily.org) (latest development release)\n * [CMake](http://www.cmake.org/)\n * [EVALB](http://nlp.cs.nyu.edu/evalb/) (latest version; please put the EVALB folder under `src` directory)\n * [brown-cluster](https://github.com/percyliang/brown-cluster) (version 1.3)\n\n## Python package installation\n\nThe `pypkg` package needs to be installed. Run `pip install -e .` from the project root directory.\n\n## Build instructions\n\n    mkdir src/build\n    cd src/build\n    cmake -DEIGEN3_INCLUDE_DIR=/path/to/eigen ..\n    make\n\nBy default, pretrained word embeddings feature for the generative model is enabled. To disable it, pass `-DDEFINE_ENABLE_PRETRAINED=OFF` option to `cmake`. You can also pass `-j [number of threads]` option to enable multithreading for your compilation.\n\n## Input file format\n\nSee `sample_input_english.txt` (English Penn Treebank) and `sample_input_indonesian.txt` (Indonesian Treebank)\n\n## Oracles\n\nThe oracle-generation scripts convert a bracketed parse tree into a sequence of actions. The script also converts singletons in the training set and unknown words in the dev and test set into the appropriate `UNK` tokens.\n\n### Obtaining the oracle for the discriminative model\n\nFor English\n\n    python src/get_oracle.py [training file] [training file] \u003e train.oracle\n    python src/get_oracle.py [training file] [dev file] \u003e dev.oracle\n    python src/get_oracle.py [training file] [test file] \u003e test.oracle\n\nFor Indonesian\n\n    python src/get_oracle_id.py [training file] [training file] \u003e train.oracle\n    python src/get_oracle_id.py [training file] [dev file] \u003e dev.oracle\n    python src/get_oracle_id.py [training file] [test file] \u003e test.oracle\n\n### Obtaining the oracle for the generative model\n\nFor English\n\n    python src/get_oracle_gen.py [training file] [training file] \u003e train_gen.oracle\n    python src/get_oracle_gen.py [training file] [dev file] \u003e dev_gen.oracle\n    python src/get_oracle_gen.py [training file] [test file] \u003e test_gen.oracle\n\nFor Indonesian\n\n    python src/get_oracle_gen_id.py [training file] [training file] \u003e train_gen.oracle\n    python src/get_oracle_gen_id.py [training file] [dev file] \u003e dev_gen.oracle\n    python src/get_oracle_gen_id.py [training file] [test file] \u003e test_gen.oracle\n\n## Discriminative model\n\nThe discriminative variant of the RNNG is used as a proposal distribution for decoding the generative model (although it can also be used for decoding on its own).\n\n### Training the discriminative model\n\nTo train the discriminative model **without** character embeddings, run the command below\n\n    build/nt-parser/nt-parser --cnn-mem 1700 -x -T [training oracle file] -d [dev oracle file] -C [original dev file (PTB bracketed format, see sample_input_english.txt)] -P -t --pretrained_dim [dimension of pretrained word embeddings] -w [pretrained word embeddings file] --lstm_input_dim 128 --hidden_dim 128 -D [dropout rate] --model_dir [directory to save the model] \u003e log.txt\n\nTo use character embeddings, run the command using `build/nt-parser/nt-parser-char` binary instead, and add additional options `--char_embeddings_model addition --separate_unk_embeddings`. Both commands **MUST** be run from the `src` directory.\n\nIf pretrained word embeddings are not used, remove the `--pretrained_dim` and `-w` options.\n\nThis command will train the discriminative model with early stopping: every 25 parameter updates, the model is evaluated on the dev file, and the training will be stopped if there is no improvement after 10 evaluations. The training log is printed to `log.txt` (including information of the filename which the model is saved to, which is used for decoding with the `-m` option below).\n\nRun the command with `-h` option to see all the available options.\n\n### Decoding with discriminative model\n\nTo decode using the discriminative model **without** character embeddings, run the command below\n\n    build/nt-parser/nt-parser --cnn-mem 1700 -x -T [training_oracle_file] -p [test_oracle_file] -C [original_test_file (PTB bracketed format, see sample_input_english.txt)] -P --pretrained_dim [dimension of pretrained word embeddings] -w [pretrained word embeddings file] --lstm_input_dim 128 --hidden_dim 128 -m [path to model parameter file] \u003e output.txt\n\nTo use character embeddings, run the command using `build/nt-parser/nt-parser-char` binary instead, and add additional options `--char_embeddings_model addition --separate_unk_embeddings`. Both commands **MUST** be run from the `src` directory.\n\nThe output will be stored in `/tmp/parse/parser_test_eval.xxxx.txt` and the parser will output F1 score calculated with `EVALB` with options specified in `COLLINS.prm` file. The parameter file (following the `-m` in the command above) can be obtained from `log.txt`.\n\nIf training was done using pretrained word embeddings (by specifying the `-w` and `--pretrained_dim` options) or POS tags (`-P` option), then decoding must also use that same options.\n\nRun the command with `-h` option to see all the available options.\n\n## Generative model\n\nIn (Dyer et al., 2016), the generative model achieved state of the art results, and decoding is done using sampled trees from the trained discriminative model.\n\n### Training word clusters\n\nWord clusters are trained with Brown clustering algorithm, whose implementation is available [here](https://github.com/percyliang/brown-cluster). You should install the implementation on your machine. To train the clusters, first run the command to get the unkified words from an oracle file\n\n    python scripts/get_unkified_from_oracle.py [training_oracle_file] \u003e unkified.txt\n\nThen, do Brown clustering on the output file\n\n    python scripts/do_brown_cluster.py --outdir cluster-dir unkified.txt\n\nThe clusters are available in `cluster-dir/paths` file. Invoke any of the scripts above with `-h` or `--help` flag to see all the available options.\n\n### Training the generative model\n\nTo train the generative model **without** character embeddings, run the command below\n\n    build/nt-parser/nt-parser-gen -x -T [training oracle generative] -d [dev oracle generative] -t --clusters [path to clusters file, or clusters-train-berk.txt for PTB full dataset] --pretrained_dim [dimension of pretrained word embeddings] -w [pretrained word embeddings file] --input_dim 256 --lstm_input_dim 256 --hidden_dim 256 -D [dropout rate] --model_dir [directory to save the model] \u003e log_gen.txt\n\nTo use character embeddings, run the command using `build/nt-parser/nt-parser-gen-char` binary instead, and add additional options `--char_embeddings_model addition --separate_unk_embeddings`. Both commands **MUST** be run from the `src` directory.\n\nIf pretrained word embeddings are not used, remove the `--pretrained_dim` and `-w` options.\n\nThis command will train the generative model with early stopping; every 25 parameter updates, the model is evaluated on the dev file, and the training will be stopped if there is no improvement after 10 evaluations. The training log is printed to `log_gen.txt`, including information on where the parameters of the model is saved to, which is used for decoding later.\n\n### Decoding with the generative model\n\nDecoding with the generative model requires sample trees from the trained discriminative model. All the commands below must be run from the `src` directory.\n\n#### Sampling trees from the discriminative model\n\nTo sample trees from the discriminative model **without** character embeddings, run the command below\n\n    build/nt-parser/nt-parser --cnn-mem 1700 -x -T [training oracle file] -p [test oracle file] -C [original test file (PTB bracketed format, see sample_input_english.txt)] -P --pretrained_dim [dimension of pretrained word embeddings] -w [pretrained word embeddings file] --lstm_input_dim 128 --hidden_dim 128 -m [parameter file of the trained discriminative model] --alpha [flattening coefficient] -s 100 \u003e test-samples.props\n\nTo use character embeddings, run the command using `build/nt-parser/nt-parser-char` binary instead, and add additional options `--char_embeddings_model addition --separate_unk_embeddings`. Both commands **MUST** be run from the `src` directory.\n\nIf pretrained word embeddings are not used, remove the `--pretrained_dim` and `-w` options.\n\nImportant parameters:\n\n * s = # of samples (all reported results used 100)\n * alpha = flattening coefficient (value not exceeding 1 is sensible; may be better to tune this on dev set)\n\n#### Prepare samples for likelihood evaluation\n\n    utils/cut-corpus.pl 3 test-samples.props \u003e test-samples.trees\n\n#### Evaluate joint likelihood under generative model\n\nTo evaluate the samples using the generative model **without** character embeddings, run the command below\n\n    build/nt-parser/nt-parser-gen -x -T [training oracle generative] --clusters [path to clusters file, or clusters-train-berk.txt for PTB full dataset] --pretrained_dim [dimension of pretrained word embeddings] -w [pretrained word embeddings file] --input_dim 256 --lstm_input_dim 256 --hidden_dim 256 -p test-samples.trees -m [parameters file from the trained generative model, see log_gen.txt] \u003e test-samples.likelihoods\n\nTo use character embeddings, run the command using `build/nt-parser/nt-parser-gen-char` binary instead, and add additional options `--char_embeddings_model addition --separate_unk_embeddings`.\n\nIf pretrained word embeddings are not used, remove the `--pretrained_dim` and `-w` options.\n\n#### Estimate marginal likelihood (final step to get language modeling ppl)\n\n    utils/is-estimate-marginal-llh.pl 2416 100 test-samples.props test-samples.likelihoods \u003e llh.txt 2\u003e rescored.trees\n\n * 100 = # of samples\n * 2416 = # of sentences in test set\n * `rescored.trees` will contain the reranked samples\n\nThe file `llh.txt` would contain the final language modeling perplexity after marginalization (see the last lines of the file)\n\n#### Compute generative model parsing accuracy (final step to get parsing accuracy from the generative model)\n\n    utils/add-fake-preterms-for-eval.pl rescored.trees \u003e rescored.preterm.trees\n    utils/replace-unks-in-trees.pl [Discriminative oracle for the test file] rescored.preterm.trees \u003e hyp.trees\n    utils/remove_dev_unk.py [gold trees on the test set (same format as sample_input_english.txt)] hyp.trees \u003e hyp_final.trees\n    EVALB/evalb -p COLLINS.prm [gold trees on the test set (same format as sample_input_english.txt)] hyp_final.trees \u003e parsing_result.txt\n\nThe file `parsing_result.txt` contains the final parsing accuracy using EVALB.\n\n## License\n\nThis software is released under the terms of the [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0).\n\n## Changes\n\nSeveral major changes has been added from the [original implementation by Dyer et al.](https://github.com/clab/rnng/):\n\n- Early stopping (`src/nt-parser/nt-parser.cc`, `src/nt-parser/nt-parser-gen.cc`)\n- Pretrained word embeddings feature of the generative model (`src/nt-parser/nt-parser-gen.cc`, `src/CMakeLists.txt`)\n- RNNGs with character embeddings (`src/nt-parser/nt-parser-char.cc`, `src/nt-parser/nt-parser-gen-char.cc`, `src/nt-parser/embeddings.h`, `src/nt-parser/embeddings.cc`, `src/nt-parser/CMakeLists.txt`)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkmkurn%2Fmsc-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkmkurn%2Fmsc-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkmkurn%2Fmsc-project/lists"}