https://github.com/code-kern-ai/refinery-tokenizer

Tokenizer for refinery. Manages the creation and storage of spaCy tokens for text-based record attributes and supports multiple language models. It is used by the gateway.
https://github.com/code-kern-ai/refinery-tokenizer

Last synced: 2 months ago
JSON representation

Tokenizer for refinery. Manages the creation and storage of spaCy tokens for text-based record attributes and supports multiple language models. It is used by the gateway.

• Host: GitHub
• URL: https://github.com/code-kern-ai/refinery-tokenizer
• Owner: code-kern-ai
• License: apache-2.0
• Created: 2022-07-14T13:54:05.000Z (almost 3 years ago)
• Default Branch: dev
• Last Pushed: 2025-03-20T12:23:42.000Z (3 months ago)
• Last Synced: 2025-04-03T11:36:24.703Z (3 months ago)
• Language: Python
• Homepage: https://www.kern.ai
• Size: 99.6 KB
• Stars: 1
• Watchers: 3
• Forks: 1
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE
  • Codeowners: CODEOWNERS

Awesome Lists containing this project

README

        
# refinery-tokenizer [![Build Status](https://drone.dev.kern.ai/api/badges/code-kern-ai/refinery-tokenizer/status.svg?ref=refs/heads/dev)](https://drone.dev.kern.ai/code-kern-ai/refinery-tokenizer)

[![refinery's repository](https://uploads-ssl.webflow.com/61e47fafb12bd56b40022a49/62c2f30f935f4d37dc864eeb_Kern%20refinery.png)](https://github.com/code-kern-ai/refinery)

Tokenizer for [refinery](https://github.com/code-kern-ai/refinery). Manages the creation and storage of `spaCy` tokens for text-based record attributes and supports multiple language models. It is used by the [gateway](https://github.com/code-kern-ai/refinery-gateway).

If you like what we're working on, please leave a ⭐ for [refinery](https://github.com/code-kern-ai/refinery)!