{"id":20074188,"url":"https://github.com/furk4neg3/ibm_nlp_tokenization_and_classification","last_synced_at":"2026-05-12T18:12:04.356Z","repository":{"id":262087560,"uuid":"886183098","full_name":"furk4neg3/IBM_NLP_Tokenization_and_Classification","owner":"furk4neg3","description":"This repository contains my implementations of tokenization techniques and document classification as part of an IBM AI lab project.","archived":false,"fork":false,"pushed_at":"2024-11-10T12:29:29.000Z","size":1375,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-13T00:33:46.963Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/furk4neg3.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-10T12:27:41.000Z","updated_at":"2024-11-10T12:29:32.000Z","dependencies_parsed_at":"2024-11-10T13:28:37.188Z","dependency_job_id":"c76e24a2-d5f8-456b-a102-c9350b444f41","html_url":"https://github.com/furk4neg3/IBM_NLP_Tokenization_and_Classification","commit_stats":null,"previous_names":["furk4neg3/ibm_nlp_tokenization_and_classification"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/furk4neg3%2FIBM_NLP_Tokenization_and_Classification","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/furk4neg3%2FIBM_NLP_Tokenization_and_Classification/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/furk4neg3%2FIBM_NLP_Tokenization_and_Classification/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/furk4neg3%2FIBM_NLP_Tokenization_and_Classification/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/furk4neg3","download_url":"https://codeload.github.com/furk4neg3/IBM_NLP_Tokenization_and_Classification/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241504321,"owners_count":19973219,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T14:49:51.187Z","updated_at":"2026-05-12T18:11:59.300Z","avatar_url":"https://github.com/furk4neg3.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# IBM NLP Tokenization and Document Classification Lab\n\nThis repository contains my implementations of tokenization techniques and document classification as part of an IBM AI lab project. The exercises cover fundamental NLP preprocessing methods, including word and subword tokenization, as well as applying classification models on processed text data using Hugging Face's Transformers library.\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Technologies Used](#technologies-used)\n- [Project Details](#project-details)\n- [Key Learnings](#key-learnings)\n- [References](#references)\n\n## Overview\n\nTokenization and document classification are essential steps in natural language processing. This project implements tokenization techniques such as:\n\n- Word tokenization\n- Subword tokenization (BPE, WordPiece)\n- Sentence-level tokenization\n\nAdditionally, a classification model is applied to the tokenized text data, demonstrating how NLP models process and classify documents based on their content.\n\n## Technologies Used\n\n- **Python**: Primary programming language\n- **Hugging Face Transformers**: Library for model tokenization and NLP tasks\n- **Jupyter Notebook**: For documenting and demonstrating code execution\n\n## Project Details\n\n1. **Tokenization Techniques**:\n   - **Word Tokenization**: Tokenizers that split text into individual words.\n   - **Subword Tokenization**: Implements Byte Pair Encoding (BPE) and WordPiece methods.\n   - **Sentence Tokenization**: Tokenizes text at the sentence level, useful for tasks requiring sentence-level analysis.\n\n2. **Document Classification**:\n   - Applied a classification model to categorize text documents based on their content.\n   - Trained a Hugging Face Transformer model to perform document classification, providing hands-on experience with text analysis.\n\n## Key Learnings\n\n- Practical understanding of various tokenization methods and their importance in NLP.\n- Hands-on experience with document classification using Hugging Face’s Transformers library.\n- Exposure to foundational NLP workflows, including data preparation and model training.\n\n## References\n\n- [IBM AI Engineering Professional Certificate](https://www.coursera.org/professional-certificates/ai-engineer?)\n- [Generative AI Engineering with LLMs Specialization](https://www.coursera.org/specializations/generative-ai-engineering-with-llms)\n- [Gen AI Foundational Models for NLP \u0026 Language Understanding](https://www.coursera.org/learn/gen-ai-foundational-models-for-nlp-and-language-understanding?specialization=generative-ai-engineering-with-llms)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffurk4neg3%2Fibm_nlp_tokenization_and_classification","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffurk4neg3%2Fibm_nlp_tokenization_and_classification","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffurk4neg3%2Fibm_nlp_tokenization_and_classification/lists"}