{"id":27301558,"url":"https://github.com/paithiov909/gibasa","last_synced_at":"2025-07-12T13:36:39.469Z","repository":{"id":42029366,"uuid":"415636523","full_name":"paithiov909/gibasa","owner":"paithiov909","description":"A plain ‘Rcpp’ wrapper of ‘MeCab'","archived":false,"fork":false,"pushed_at":"2025-03-18T13:30:50.000Z","size":60228,"stargazers_count":16,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-19T09:58:55.318Z","etag":null,"topics":["mecab","pos-tagging","r","r-package","rcpp"],"latest_commit_sha":null,"homepage":"https://paithiov909.github.io/gibasa/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paithiov909.png","metadata":{"files":{"readme":"README.Rmd","changelog":"NEWS.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-10-10T16:15:33.000Z","updated_at":"2025-04-09T10:19:32.000Z","dependencies_parsed_at":"2024-02-22T06:28:31.252Z","dependency_job_id":"207c7085-08f2-4c6e-9c0f-b4087b01b184","html_url":"https://github.com/paithiov909/gibasa","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"purl":"pkg:github/paithiov909/gibasa","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paithiov909%2Fgibasa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paithiov909%2Fgibasa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paithiov909%2Fgibasa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paithiov909%2Fgibasa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paithiov909","download_url":"https://codeload.github.com/paithiov909/gibasa/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paithiov909%2Fgibasa/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265002565,"owners_count":23696076,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mecab","pos-tagging","r","r-package","rcpp"],"created_at":"2025-04-12T01:46:48.845Z","updated_at":"2025-07-12T13:36:39.450Z","avatar_url":"https://github.com/paithiov909.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\"\n)\npkgload::load_all(export_all = FALSE)\n```\n\n# gibasa\n\n\u003c!-- badges: start --\u003e\n[![gibasa status badge](https://paithiov909.r-universe.dev/badges/gibasa)](https://paithiov909.r-universe.dev/gibasa)\n![GitHub](https://img.shields.io/github/license/paithiov909/gibasa)\n[![R-CMD-check](https://github.com/paithiov909/gibasa/workflows/R-CMD-check/badge.svg)](https://github.com/paithiov909/gibasa/actions)\n[![codecov](https://codecov.io/gh/paithiov909/gibasa/branch/main/graph/badge.svg)](https://app.codecov.io/gh/paithiov909/gibasa)\n[![CRAN logs badge](https://cranlogs.r-pkg.org/badges/gibasa)](https://cran.r-project.org/package=gibasa)\n\u003c!-- badges: end --\u003e\n\n## Overview\n\ngibasa is a plain 'Rcpp' wrapper for 'MeCab', a morphological analyzer for CJK text.\n\nPart-of-speech tagging with morphological analyzers is useful for processing CJK text data. This is because most words in CJK text are not separated by whitespaces and `tokenizers::tokenize_words` may split them into wrong tokens.\n\nThe main goal of gibasa package is to provide an alternative to `tidytext::unnest_tokens` for CJK text data. For this goal, gibasa provides three main functions: `gibasa::tokenize`, `gibasa::prettify`, and `gibasa::pack`.\n\n![flowchart of a text analysis that combines gibasa and other packages](man/figures/tidytext_fig5_1_mod.drawio.png)\n\n- `gibasa::tokenize` takes a TIF-compliant data.frame of corpus, returning tokens as format that known as 'tidy text data', so that users can replace `tidytext::unnest_tokens` with it for tokenizing CJK text.\n- `gibasa::prettify` turns tagged features into columns.\n- `gibasa::pack` takes a 'tidy text data', typically returning space-separated corpus.\n\n## Installation\n\nYou can install binary package via [CRAN](https://cran.r-project.org/package=gibasa) or [r-universe](https://paithiov909.r-universe.dev/gibasa).\n\n```r\n## Install gibasa from r-universe repository\ninstall.packages(\"gibasa\", repos = c(\"https://paithiov909.r-universe.dev\", \"https://cloud.r-project.org\"))\n\n## Or build from source package\nSys.setenv(MECAB_DEFAULT_RC = \"/fullpath/to/your/mecabrc\") # if necessary\nremotes::install_github(\"paithiov909/gibasa\")\n```\n\nTo use gibasa package requires the [MeCab](https://taku910.github.io/mecab/) library and its dictionary installed and available.\n\nIn case using Linux or macOS, you can install them with their package managers, or build and install from the source by yourself.\n\nIn case using Windows, use installer [built for 64bit](https://github.com/ikegami-yukino/mecab/releases/tag/v0.996.2). Note that gibasa requires a UTF-8 dictionary, not a Shift-JIS one.\n\nAs of v0.9.4, gibasa looks at the file specified by the environment variable `MECABRC` or the file located at `~/.mecabrc`. If the MeCab dictionary is in a different location than the default, create a mecabrc file and specify where the dictionary is located.\n\nFor example, to install and use the [ipadic](https://pypi.org/project/ipadic/) from PyPI, run:\n\n```sh\n$ python3 -m pip install ipadic\n$ python3 -c \"import ipadic; print('dicdir=' + ipadic.DICDIR);\" \u003e ~/.mecabrc\n```\n\n## Usage\n\n### Tokenize sentences\n\n```{r}\nres \u003c- gibasa::tokenize(\n  data.frame(\n    doc_id = seq_along(gibasa::ginga[5:8]),\n    text = gibasa::ginga[5:8]\n  ),\n  text,\n  doc_id\n)\nres\n```\n\n### Prettify output\n\n```{r}\ngibasa::prettify(res)\ngibasa::prettify(res, col_select = 1:3)\ngibasa::prettify(res, col_select = c(1, 3, 5))\ngibasa::prettify(res, col_select = c(\"POS1\", \"Original\"))\n```\n\n### Pack output\n\n```{r}\nres \u003c- gibasa::prettify(res)\ngibasa::pack(res)\n\ndplyr::mutate(\n  res,\n  token = dplyr::if_else(is.na(Original), token, Original),\n  token = paste(token, POS1, sep = \"/\")\n) |\u003e\n  gibasa::pack() |\u003e\n  head(1L)\n```\n\n### Change dictionary\n\nIPA, UniDic, [CC-CEDICT-MeCab](https://github.com/ueda-keisuke/CC-CEDICT-MeCab), and [mecab-ko-dic](https://bitbucket.org/eunjeon/mecab-ko-dic/src/master/) schemes are supported.\n\n```{r}\n## UniDic 2.1.2\ngibasa::tokenize(\"あのイーハトーヴォのすきとおった風\", sys_dic = file.path(\"mecab/unidic-lite\")) |\u003e\n  gibasa::prettify(into = gibasa::get_dict_features(\"unidic26\"))\n\n\n## CC-CEDICT\ngibasa::tokenize(\"它可以进行日语和汉语的语态分析\", sys_dic = file.path(\"mecab/cc-cedict\")) |\u003e\n  gibasa::prettify(into = gibasa::get_dict_features(\"cc-cedict\"))\n\n\n## mecab-ko-dic\ngibasa::tokenize(\"하네다공항한정토트백\", sys_dic = file.path(\"mecab/mecab-ko-dic\")) |\u003e\n  gibasa::prettify(into = gibasa::get_dict_features(\"ko-dic\"))\n```\n\n## Build dictionaries\n\n### Build a system dictionary\n\n```{r}\n## build a new ipadic in temporary directory\nbuild_sys_dic(\n  dic_dir = file.path(\"mecab/ipadic-eucjp\"), # replace here with path to your source dictionary\n  out_dir = tempdir(),\n  encoding = \"euc-jp\" # encoding of source csv files\n)\n\n## copy the 'dicrc' file\nfile.copy(file.path(\"mecab/ipadic-eucjp/dicrc\"), tempdir())\n\ndictionary_info(sys_dic = tempdir())\n```\n\n### Build a user dictionary\n\n```{r}\n## write a csv file and compile it into a user dictionary\nwriteLines(\n  c(\n    \"月ノ,1290,1290,4579,名詞,固有名詞,人名,姓,*,*,月ノ,ツキノ,ツキノ\",\n    \"美兎,1291,1291,8561,名詞,固有名詞,人名,名,*,*,美兎,ミト,ミト\"\n  ),\n  con = (csv_file \u003c- tempfile(fileext = \".csv\"))\n)\nbuild_user_dic(\n  dic_dir = file.path(\"mecab/ipadic-eucjp\"),\n  file = (user_dic \u003c- tempfile(fileext = \".dic\")),\n  csv_file = csv_file,\n  encoding = \"utf8\"\n)\n\ntokenize(\"月ノ美兎は箱の中\", sys_dic = tempdir(), user_dic = user_dic)\n```\n\n## License\n\nGPL (\u003e=3).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaithiov909%2Fgibasa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpaithiov909%2Fgibasa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaithiov909%2Fgibasa/lists"}