{"id":16941312,"url":"https://github.com/msiemens/hyphenn-de","last_synced_at":"2025-03-22T12:32:44.824Z","repository":{"id":48508837,"uuid":"91207287","full_name":"msiemens/HypheNN-de","owner":"msiemens","description":"A neural network hyphenator for the German language","archived":false,"fork":false,"pushed_at":"2023-10-25T19:48:56.000Z","size":94,"stargazers_count":43,"open_issues_count":3,"forks_count":6,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-10-14T21:09:19.308Z","etag":null,"topics":["german-language","hyphenation","keras","machine-learning","neural-network","rust","tensorflow"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/msiemens.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-05-13T22:55:30.000Z","updated_at":"2024-05-20T13:46:16.000Z","dependencies_parsed_at":"2022-08-31T12:10:25.891Z","dependency_job_id":null,"html_url":"https://github.com/msiemens/HypheNN-de","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msiemens%2FHypheNN-de","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msiemens%2FHypheNN-de/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msiemens%2FHypheNN-de/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msiemens%2FHypheNN-de/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/msiemens","download_url":"https://codeload.github.com/msiemens/HypheNN-de/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221829525,"owners_count":16887618,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["german-language","hyphenation","keras","machine-learning","neural-network","rust","tensorflow"],"created_at":"2024-10-13T21:09:15.577Z","updated_at":"2024-10-28T12:52:53.520Z","avatar_url":"https://github.com/msiemens.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HypheNN-de\n\nA neural network that hyphenates German words. Following _B. Fritzke and\nC. Nasahl, \"A neural network that learns to do hyphenation\"_, it uses a window\nof 8 characters and determines whether the word can be hyphenated after position\n4 of the current window. Currently the network achieves a success rate of 99.2 %.\n\nThe network input is encoded using one-hot encoding the character set and outputs\na single value indicating the probability of a hyphenation at the current position\nbeing valid.\n\n\n## Dependencies\n\n- Python 3 with TensorFlow and Keras installed\n- Rust\n\n\n## Download \u0026 prepare data\n\n1. Download Wiktionary dump from https://dumps.wikimedia.org/dewiktionary/latest/ (look for `dewiktionary-{timestamp}-pages-articles.xml.bz2`) and unpack it\n2. Compile and run `prepare_data`:\n    - `$ cd prepare_data`\n    - `$ cargo run --release ../data/dewiktionary-*-pages-articles.xml ../wordlist.txt`\n\nNotes: There's a fair amount of post-processing happening to cleanup the data. The whole process may work with other languages, but the data cleanup probably will need some adaption.\nNote: The preprocessing randomizes the order of entries.\n\n\n## Train\n\nTo train the network, run:\n\n```\n$ python train.py\nUsing TensorFlow backend.\nBuilding model...\nDone\n\nTraining model...\n2017-05-13 21:58:40.012420: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:857] OS X does not support NUMA - returning NUMA node zero\n2017-05-13 21:58:40.012582: I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] Found device 0 with properties: \nname: Quadro K2000\nmajor: 3 minor: 0 memoryClockRate (GHz) 0.954\npciBusID 0000:02:00.0\nTotal memory: 2.00GiB\nFree memory: 1001.96MiB\n2017-05-13 21:58:40.012595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:927] DMA: 0 \n2017-05-13 21:58:40.012600: I tensorflow/core/common_runtime/gpu/gpu_device.cc:937] 0:   Y \n2017-05-13 21:58:40.012610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:996] Creating TensorFlow device (/gpu:0) -\u003e (device: 0, name: Quadro K2000, pci bus id: 0000:02:00.0)\nEpoch 1/50\n1253797/1253797 [==============================] - 7s - loss: 0.0450 - acc: 0.9468       \nEpoch 2/50\n1253797/1253797 [==============================] - 7s - loss: 0.0218 - acc: 0.9753       \n...\nEpoch 48/50\n1253797/1253797 [==============================] - 7s - loss: 0.0076 - acc: 0.9920      \nEpoch 49/50\n1253797/1253797 [==============================] - 7s - loss: 0.0076 - acc: 0.9919       \nEpoch 50/50\n1253797/1253797 [==============================] - 7s - loss: 0.0076 - acc: 0.9920      \nDone\nTime: 0:06:05.55\n```\n\nThe network weights are stored in `data/model.h5`\n\n## Validate\n\n```\n$ python validate.py\nUsing TensorFlow backend.\nBuilding model...\nDone\n\n2017-05-14 00:50:01.683992: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:857] OS X does not support NUMA - returning NUMA node zero\n2017-05-14 00:50:01.684174: I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] Found device 0 with properties: \nname: Quadro K2000\nmajor: 3 minor: 0 memoryClockRate (GHz) 0.954\npciBusID 0000:02:00.0\nTotal memory: 2.00GiB\nFree memory: 720.18MiB\n2017-05-14 00:50:01.684190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:927] DMA: 0 \n2017-05-14 00:50:01.684195: I tensorflow/core/common_runtime/gpu/gpu_device.cc:937] 0:   Y \n2017-05-14 00:50:01.684206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:996] Creating TensorFlow device (/gpu:0) -\u003e (device: 0, name: Quadro K2000, pci bus id: 0000:02:00.0)\nValidating model...\n1253536/1253797 [============================\u003e.] - ETA: 0s   \nDone\nResult: [0.0073155245152076989, 0.99240706430147785]\nTime: 0:00:40.17\n```\n\nNote: The first value is the mean square error, the second value is the\nachieved accuracy.\n\n## Run\n\n\n```\n$ python predict.py Silbentrennung\nUsing TensorFlow backend.\nBuilding model...\nDone\n\n2017-05-14 00:49:08.660959: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:857] OS X does not support NUMA - returning NUMA node zero\n2017-05-14 00:49:08.661122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] Found device 0 with properties: \nname: Quadro K2000\nmajor: 3 minor: 0 memoryClockRate (GHz) 0.954\npciBusID 0000:02:00.0\nTotal memory: 2.00GiB\nFree memory: 738.18MiB\n2017-05-14 00:49:08.661136: I tensorflow/core/common_runtime/gpu/gpu_device.cc:927] DMA: 0 \n2017-05-14 00:49:08.661141: I tensorflow/core/common_runtime/gpu/gpu_device.cc:937] 0:   Y \n2017-05-14 00:49:08.661152: I tensorflow/core/common_runtime/gpu/gpu_device.cc:996] Creating TensorFlow device (/gpu:0) -\u003e (device: 0, name: Quadro K2000, pci bus id: 0000:02:00.0)\nInput: Silbentrennung\nHyphenation: Sil·ben·tren·nung\n```\n\n\n## Tensorflow build notes\n\nSetup: See https://gist.github.com/Mistobaan/dd32287eeb6859c6668d\n\nConfigure:\n\n```\nPYTHON_BIN_PATH=$(which python) CUDA_TOOLKIT_PATH=\"/usr/local/cuda\" CUDNN_INSTALL_PATH=\"/usr/local/cuda\" TF_UNOFFICIAL_SETTING=1 TF_NEED_CUDA=1 TF_CUDA_COMPUTE_CAPABILITIES=\"3.0\" TF_CUDNN_VERSION=\"6\" TF_CUDA_VERSION=\"8.0\" TF_CUDA_VERSION_TOOLKIT=8.0 ./configure\n```\n\nNote: Use defaults everywhere.\n\nBuild:\n\n```\nbazel build -c opt --copt=-mavx --copt=-msse4.2 --config=cuda //tensorflow/tools/pip_package:build_pip_package\n```\n\nIf encountering problems with `Library not loaded: @rpath/libcudart.8.0.dylib`,\nfollow http://stackoverflow.com/a/40007947/997063.\n\nIf encountering problems regarding `-lgomp`, replace `-lgomp` with\n`-L/usr/local/Cellar/llvm/4.0.0/lib/libiomp5.dylib` in\n`third_party/gpus/cuda/BUILD.tpl` or comment out this line.\n\n\n## References\n\n- http://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/\n- http://blog.aloni.org/posts/backprop-with-tensorflow/\n- B. Fritzke and C. Nasahl, \"A neural network that learns to do hyphenation,\" IJCNN-91-Seattle International Joint Conference on Neural Networks, Seattle, WA, USA, 1991, pp. 960 vol.2-. (doi: 10.1109/IJCNN.1991.155602)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsiemens%2Fhyphenn-de","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmsiemens%2Fhyphenn-de","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsiemens%2Fhyphenn-de/lists"}