{"id":26058900,"url":"https://github.com/zackiles/js-ast-tokenizer","last_synced_at":"2025-04-11T05:51:43.801Z","repository":{"id":257827615,"uuid":"873341440","full_name":"zackiles/js-ast-tokenizer","owner":"zackiles","description":"A JavaScript code tokenizer for ease in using with code embeddings and vector storage.","archived":false,"fork":false,"pushed_at":"2024-10-16T02:39:57.000Z","size":264,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-03T11:53:57.637Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zackiles.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-16T02:14:26.000Z","updated_at":"2024-10-16T02:40:01.000Z","dependencies_parsed_at":"2024-10-17T12:33:07.090Z","dependency_job_id":null,"html_url":"https://github.com/zackiles/js-ast-tokenizer","commit_stats":null,"previous_names":["zackiles/js-ast-tokenizer"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zackiles%2Fjs-ast-tokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zackiles%2Fjs-ast-tokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zackiles%2Fjs-ast-tokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zackiles%2Fjs-ast-tokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zackiles","download_url":"https://codeload.github.com/zackiles/js-ast-tokenizer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248242988,"owners_count":21071054,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-08T12:40:53.750Z","updated_at":"2025-04-11T05:51:43.783Z","avatar_url":"https://github.com/zackiles.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# JS AST Tokenizer\r\n\r\nA JavaScript code tokenizer for ease in using with code embeddings and vector storage. This library analyzes JavaScript files or code snippets and provides a structured tokenization output, making it suitable for code analysis, embeddings, and search applications.\r\n\r\n## Features\r\n\r\n- Tokenizes JavaScript files or strings into key components like imports, classes, functions, variables, and more.\r\n- Designed to work seamlessly with code embeddings and vector storage systems.\r\n- Supports modern JavaScript syntax, including JSX, TypeScript, and class properties.\r\n\r\n## Installation\r\n\r\nTo install the library, use npm:\r\n\r\n```bash\r\nnpm install js-ast-tokenizer\r\n```\r\n\r\n## Usage\r\n\r\nHere’s a basic example of how to use the tokenizer:\r\n\r\n### Example 1: Tokenizing a JavaScript File\r\n\r\n```javascript\r\nimport tokenizeFileOrCode from 'js-ast-tokenizer';\r\n\r\nconst result = await tokenizeFileOrCode('path/to/your/file.js');\r\nconsole.log(result);\r\n```\r\n\r\n### Example 2: Tokenizing JavaScript Code String\r\n\r\n```javascript\r\nimport tokenizeFileOrCode from 'js-ast-tokenizer';\r\n\r\nconst jsCode = `\r\n  import { somethingExternal } from 'some-module';\r\n\r\n  class SomeClass {\r\n    constructor() {\r\n      this.value = 42;\r\n    }\r\n  }\r\n\r\n  const someVariable = 2;\r\n\r\n  function someFunction() {\r\n    return 'Hello, world!';\r\n  }\r\n\r\n  someExternalFunction();\r\n  someExternalClass.someMethod();\r\n  \r\n  export { someFunction };\r\n  export default SomeClass;\r\n`;\r\n\r\nconst result = await tokenizeFileOrCode(jsCode);\r\nconsole.log(result);\r\n```\r\n\r\n### Output\r\n\r\nThe output of the `tokenizeFileOrCode` function will be a structured object that represents the tokenized components of your JavaScript code. Below is an example output with the key components, including `globalVariables`, `externalReferences`, and `exports`:\r\n\r\n```json\r\n{\r\n  \"file\": \"path/to/your/file.js\",\r\n  \"nodes\": {\r\n    \"imports\": [[\"some-module\", \"import { somethingExternal } from 'some-module';\"]],\r\n    \"classes\": [[\"SomeClass\", \"class SomeClass { constructor() { this.value = 42; } }\"]],\r\n    \"globalVariables\": [\r\n      [\"someVariable\", \"const someVariable = 2\"]\r\n    ],\r\n    \"globalFunctions\": [\r\n      [\"someFunction\", \"function someFunction() { return 'Hello, world!'; }\"]\r\n    ],\r\n    \"exports\": [\r\n      [\"someFunction\", \"export { someFunction }\"],\r\n      [\"default\", \"export default SomeClass\"]\r\n    ],\r\n    \"externalReferences\": [\r\n      [\"someExternalFunction\", \"someExternalFunction()\"],\r\n      [\"someExternalClass.someMethod\", \"someExternalClass.someMethod()\"]\r\n    ]\r\n  },\r\n  \"content\": \"...\",\r\n  \"length\": 375\r\n}\r\n```\r\n\r\n## API\r\n\r\n### `tokenizeFileOrCode(input: string): Promise\u003cTokenizeResult\u003e`\r\n\r\nTokenizes the given input, which can be either a file path or a JavaScript code string.\r\n\r\n#### Parameters\r\n\r\n- **input**: A string representing either a file path or a JavaScript code snippet.\r\n\r\n#### Returns\r\n\r\n- A `Promise` that resolves to a `TokenizeResult` object containing details about the tokenized JavaScript code.\r\n\r\n### TokenizeResult Structure\r\n\r\nThe result object contains the following properties:\r\n\r\n```typescript\r\ninterface TokenizeResult {\r\n  file: string | null;  // Full file path if input is a file, otherwise null\r\n  nodes: TokenizedStructure;  // Tokenized components of the code\r\n  content: string;  // The original JavaScript code or file content\r\n  length: number;  // The length of the input code\r\n}\r\n\r\ninterface TokenizedStructure {\r\n  imports: [string, string][];  // List of [moduleName, importStatement]\r\n  classes: [string, string][];  // List of [className, classCode]\r\n  globalVariables: [string, string][];  // List of [variableName, declaration]\r\n  globalFunctions: [string, string][];  // List of [functionName, functionCode]\r\n  exports: [string, string][];  // List of [exportedName, exportCode]\r\n  externalReferences: [string, string][];  // List of external references [referenceName, fullExpression]\r\n}\r\n```\r\n\r\n### Examples of `globalVariables`, `externalReferences`, and `exports`\r\n\r\n- **globalVariables**: This array captures top-level variables in the file, such as `const someVariable = 2`. It includes the variable name and its full declaration. Example:\r\n  \r\n  ```json\r\n  [\r\n    [\"someVariable\", \"const someVariable = 2\"]\r\n  ]\r\n  ```\r\n\r\n- **externalReferences**: This array captures references to variables, functions, or classes that are not defined in the local scope but are used within the code, such as `someExternalFunction()` and `someExternalClass.someMethod()`. Example:\r\n  \r\n  ```json\r\n  [\r\n    [\"someExternalFunction\", \"someExternalFunction()\"],\r\n    [\"someExternalClass.someMethod\", \"someExternalClass.someMethod()\"]\r\n  ]\r\n  ```\r\n\r\n- **exports**: This array captures any named or default exports from the module. Example:\r\n\r\n  ```json\r\n  [\r\n    [\"someFunction\", \"export { someFunction }\"],\r\n    [\"default\", \"export default SomeClass\"]\r\n  ]\r\n  ```\r\n\r\n## Designed for Code Embeddings and Vector Storage\r\n\r\nThe output of this tokenizer is specifically structured to facilitate integration with code embeddings and vector storage systems. By breaking down code into its components, this library can help developers build searchable embeddings of JavaScript code for tasks like:\r\n\r\n- Code similarity search\r\n- Syntax-aware code completion\r\n- Embedding-based code search\r\n\r\n## Advanced Configuration\r\n\r\nThe tokenizer uses various Babel plugins to support modern JavaScript features. The following Babel plugins are enabled by default:\r\n\r\n- JSX\r\n- TypeScript\r\n- Optional Chaining\r\n- Class Properties\r\n\r\nYou can extend the functionality by adjusting the Babel configuration if necessary.\r\n\r\n## Error Handling\r\n\r\n- If the input is a file path but the file is not found or cannot be read, the tokenizer treats the input as raw JavaScript code.\r\n- The tokenizer leverages Babel’s `errorRecovery` mode to gracefully handle parsing errors and attempt to continue.\r\n\r\n## License\r\n\r\nThis library is licensed under the MIT License. See the [LICENSE](./LICENSE) file for more details.\r\n\r\n---\r\n\r\n### Key Additions\r\n\r\n1. **Naming Consistency**: Classes are referred to as `SomeClass`, functions as `someFunction`, exports as `someExport`, and external references as `someExternalReference`, providing consistent and clear naming.\r\n2. **Example of `externalReferences`**: Demonstrates variables, functions, or classes that exist outside the current file but are referenced within it.\r\n\r\nThis update aligns with your preference for specific naming conventions while enhancing clarity.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzackiles%2Fjs-ast-tokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzackiles%2Fjs-ast-tokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzackiles%2Fjs-ast-tokenizer/lists"}