https://github.com/hpprc/mecab-parallel-experiment
Parallel processing for a single large text file with MeCab
https://github.com/hpprc/mecab-parallel-experiment
mecab nlp parallel python
Last synced: 3 months ago
JSON representation
Parallel processing for a single large text file with MeCab
- Host: GitHub
- URL: https://github.com/hpprc/mecab-parallel-experiment
- Owner: hppRC
- Created: 2022-11-15T11:22:28.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2022-11-16T02:57:38.000Z (over 2 years ago)
- Last Synced: 2025-02-10T13:37:32.725Z (5 months ago)
- Topics: mecab, nlp, parallel, python
- Language: Python
- Homepage: https://zenn.dev/hpp/articles/64466d9476fb2b
- Size: 6.84 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
| M1 MacBook pro (10 cores) | mean | std | min | median | max | improvement |
| --------------------------- | ------- | ----- | ------- | ------- | ------- | ----------- |
| naive | 162.568 | 0.895 | 161.700 | 162.386 | 164.721 | x1.000 |
| serial_read | 42.620 | 0.620 | 42.148 | 42.428 | 44.357 | x3.814 |
| parallel_read | 35.527 | 0.106 | 35.345 | 35.511 | 35.732 | x4.576 |
| parallel_read_write | 20.826 | 0.255 | 20.598 | 20.732 | 21.523 | x7.806 |
| parallel_read_write_at_once | 20.678 | 0.146 | 20.493 | 20.645 | 21.036 | x7.862 |
| Ubuntu Server (40 cores, NFS) | mean | std | min | median | max | improvement |
| ----------------------------- | ------- | ------- | ------- | ------- | ------- | ----------- |
| naive | 444.190 | 47.739 | 404.738 | 429.977 | 579.257 | x1.000 |
| serial_read | 350.161 | 207.625 | 139.832 | 276.194 | 749.342 | x1.269 |
| parallel_read | 278.844 | 229.585 | 61.472 | 179.473 | 719.069 | x1.593 |
| parallel_read_write | 52.463 | 2.525 | 49.139 | 51.760 | 56.053 | x8.467 |
| parallel_read_write_at_once | 56.320 | 11.061 | 46.864 | 52.601 | 82.210 | x7.887 |