{"id":20119164,"url":"https://github.com/harrystevens/tokenize-file","last_synced_at":"2025-08-07T05:26:38.385Z","repository":{"id":57144226,"uuid":"119973034","full_name":"HarryStevens/tokenize-file","owner":"HarryStevens","description":"Read a file, tokenize it, and spit out a handy JSON.","archived":false,"fork":false,"pushed_at":"2018-02-22T05:58:49.000Z","size":147,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-04-25T17:21:33.116Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/tokenize-file","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HarryStevens.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-02-02T11:20:30.000Z","updated_at":"2018-02-02T12:51:30.000Z","dependencies_parsed_at":"2022-09-06T08:54:12.073Z","dependency_job_id":null,"html_url":"https://github.com/HarryStevens/tokenize-file","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HarryStevens%2Ftokenize-file","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HarryStevens%2Ftokenize-file/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HarryStevens%2Ftokenize-file/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HarryStevens%2Ftokenize-file/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HarryStevens","download_url":"https://codeload.github.com/HarryStevens/tokenize-file/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241564447,"owners_count":19982958,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T19:14:39.635Z","updated_at":"2025-03-02T19:44:26.807Z","avatar_url":"https://github.com/HarryStevens.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tokenize-file\n[![Build Status](https://travis-ci.org/HarryStevens/tokenize-file.svg?branch=master)](https://travis-ci.org/HarryStevens/tokenize-file) [![Coverage Status](https://coveralls.io/repos/github/HarryStevens/tokenize-file/badge.svg?branch=master)](https://coveralls.io/github/HarryStevens/tokenize-file?branch=master)\n\nRead a file, tokenize it, and spit out a handy JSON.\n\n## Installation\n\n```bash\nnpm i tokenize-file -S\n```\n\n## Example\n\n```js\nvar tokenizeFile = require(\"tokenize-file\");\n\ntokenizeFile(\"path/to/file.txt\", tokens =\u003e {\n  console.log(tokens.filter(d =\u003e !d.stop_word \u0026\u0026 d.pos !== \"N\"));\n});\n```\n\n## API\n\n\u003ca name=\"tokenizeFile\" href=\"#tokenizeFile\"\u003e#\u003c/a\u003e \u003cb\u003etokenizeFile\u003c/b\u003e(\u003cem\u003epath/to/file_name\u003c/em\u003e, \u003cem\u003ecallback\u003c/em\u003e)\n\nRead a file, tokenize it, and spit out the JSON of the tokens. The tokenized data is passed as an array of objects to the callback function. In the array, each token is an object, represented as:\n```\n{\n  value: \"String\", // the token\n  count: Number, // the number of times it appears in the file\n  pos: \"String\" // the token's Penn Treebank POS tag,\n  stop_word: Boolean // whether the token value is a stop word, which can be filtered out in some analyses\n}\n```\n\n`tokenizeFile` can read any type of file supported by [textract](https://github.com/dbashford/textract):\n* HTML, HTM\n* ATOM, RSS\n* Markdown\n* XML, XSL\n* PDF\n* DOC, DOCX\n* ODT, OTT (experimental, feedback needed!)\n* RTF\n* XLS, XLSX, XLSB, XLSM, XLTX\n* CSV\n* ODS, OTS\n* PPTX, POTX\n* ODP, OTP\n* ODG, OTG\n* PNG, JPG, GIF\n* DXF\n* `application/javascript`\n* All `text/*` mime-types.\n\nThe POS tags are:\n\n| POS Tag | Description | Example |\n| --- | --- | --- |\n| CC | coordinating conjunction | and |\n| CD | cardinal number | 1, third |\n| DT | determiner | the |\n| EX | existential there | there is |\n| FW | foreign word | d’hoevre |\n| IN | preposition/subordinating conjunction | in, of, like |\n| JJ | adjective | big |\n| JJR | adjective, comparative | bigger |\n| JJS | adjective, superlative | biggest |\n| LS | list marker | 1) |\n| MD | modal | could, will |\n| NN | noun, singular or mass | door |\n| NNS | noun plural | doors |\n| NNP | proper noun, singular | John |\n| NNPS | proper noun, plural | Vikings |\n| PDT | predeterminer | both the boys |\n| POS | possessive ending | friend‘s |\n| PRP | personal pronoun | I, he, it |\n| PRP$ | possessive pronoun | my, his |\n| RB | adverb | however, usually, naturally, here, good |\n| RBR | adverb, comparative | better |\n| RBS | adverb, superlative | best |\n| RP | particle | give  up  |\n| TO | to | to go, to him |\n| UH | interjection | uhhuhhuhh |\n| VB | verb, base form | take |\n| VBD | verb, past tense | took |\n| VBG | verb, gerund/present participle | taking |\n| VBN | verb, past participle | taken |\n| VBP | verb, sing. present, non-3d | take |\n| VBZ | verb, 3rd person sing. present | takes |\n| WDT | wh-determiner | which |\n| WP | wh-pronoun | who, what |\n| WP$ | possessive wh-pronoun | whose |\n| WRB | wh-abverb | where, when |","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharrystevens%2Ftokenize-file","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fharrystevens%2Ftokenize-file","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharrystevens%2Ftokenize-file/lists"}