{"id":25857802,"url":"https://github.com/fareedkhan-dev/most-powerful-nlp-library","last_synced_at":"2025-09-07T01:34:21.800Z","repository":{"id":214438193,"uuid":"736323680","full_name":"FareedKhan-dev/Most-powerful-NLP-library","owner":"FareedKhan-dev","description":"Gemini, as capable as GPT-4, provides a free API with limited access. I tested it with the help of prompt engineering and found that it can solve almost any NLP task you want to tackle.","archived":false,"fork":false,"pushed_at":"2024-01-19T04:56:10.000Z","size":110,"stargazers_count":34,"open_issues_count":0,"forks_count":9,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-04T10:32:49.799Z","etag":null,"topics":["api","gemini","large-language-models","llm","nlp","nlp-library","preprocessing","python"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FareedKhan-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-12-27T15:40:37.000Z","updated_at":"2025-03-12T09:07:34.000Z","dependencies_parsed_at":"2024-01-04T05:37:46.157Z","dependency_job_id":"b08cd685-206b-4b18-ae0e-d98b8f47457e","html_url":"https://github.com/FareedKhan-dev/Most-powerful-NLP-library","commit_stats":{"total_commits":5,"total_committers":1,"mean_commits":5.0,"dds":0.0,"last_synced_commit":"aa7d16fa5f0a50a1378eaa675d05944a892784af"},"previous_names":["fareedkhan-dev/most-powerful-nlp-library"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FareedKhan-dev%2FMost-powerful-NLP-library","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FareedKhan-dev%2FMost-powerful-NLP-library/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FareedKhan-dev%2FMost-powerful-NLP-library/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FareedKhan-dev%2FMost-powerful-NLP-library/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FareedKhan-dev","download_url":"https://codeload.github.com/FareedKhan-dev/Most-powerful-NLP-library/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FareedKhan-dev%2FMost-powerful-NLP-library/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259080945,"owners_count":22802396,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","gemini","large-language-models","llm","nlp","nlp-library","preprocessing","python"],"created_at":"2025-03-01T19:17:54.602Z","updated_at":"2025-06-10T13:04:21.278Z","avatar_url":"https://github.com/FareedKhan-dev.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Header Image)](https://i.ibb.co/thJLdDt/Group-76.png)\n\n\nConsider one of the most stressful scenarios that most coders face, dealing with large text data that requires cleaning. When using regex, you need to define different sets of patterns to remove text, and even then, you may not be sure if there is any new garbage data that needs removal. Tasks like these can be stressful for developers because of the time and effort they have to invest, and there’s still uncertainty about whether the new data requires the same procedural coding or not.\n\nThe recent trends of Large Language Models (LLMs), whether open source or closed source, have given us a new dimension of how text data can be handled. Since LLMs can analyze text data more quickly than us and can intelligently understand data to a considerable extent, similar to the way we understand it, why don’t we perform our NLP tasks using LLMs? This could automate the process and make the coder’s life less stressful.\n\n## The Core Concept Driving My Library\n\nI tried out several open-source LLMs like [Mistral 8x7b](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) or [LLAMA-2–70b-chat-hf](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf), but they never met my expectations. Although good for question answering and text generation, they fall short when it comes to NLP tasks. On the other hand, [ChatGPT](https://openai.com/blog/chatgpt) exceeded my expectations, but it requires a paid API access to perform NLP tasks on our custom dataset. [Gemini](https://deepmind.google/technologies/gemini/), as capable as GPT-4, provides a free API with limited access. I tested it with the help of prompt engineering and found that it can solve almost any NLP task you want to tackle.\n\nHere is a simple visual illustration of how I have used Gemini multi-model to perform NLP tasks on my dataset:\n\n\u003cimg src=\"https://miro.medium.com/v2/resize:fit:1400/format:webp/1*BmW2FzclMY-pghvFuMTNDg.png\" alt=\"Visual illustration of how my library works\"\u003e\n\n\n## list of features avaliable in my library\n\nfor task related to preprocessing\n\n| Preprocessing Function          | Description                                                                                              |\n|-------------------------|----------------------------------------------------------------------------------------------------------|\n| tokenize_text           | Given an input sentence, perform tokenization and return a list of tokens.                               |\n| lemmatize_text          | Given an input sentence, perform Lemmatization and return the lemmatized sentence.                         |\n| stem_text               | Given an input sentence, perform Stemming and return the stemmed sentence.                                 |\n| extract_patterns        | Given an input sentence, extract specified patterns and return a list of extracted patterns.               |\n| clean_text              | Given an input sentence, clean the text without changing words.                                           |\n| remove_html_tags        | Given an input sentence, remove specified HTML tags.                                                      |\n| replace_text            | Given an input sentence, replace specified words using replacement rules.                                  |\n| extract_embeddings      | Given a list of input texts, generate embeddings using a specified model.                                  |\n\nfor task not related to preprocessing\n\n| Core NLP Function          | Description                                                                                              |\n|-------------------------|----------------------------------------------------------------------------------------------------------|\n| analyze_sentiment       | Perform sentiment analysis on the input text and return the sentiment category.                           |\n| classify_topic          | Perform Topic Classification on the input text and return the topic category.                               |\n| detect_spam             | Perform spam detection on the input text and return the spam category.                                      |\n| detect_ner              | Perform NER detection on the input text and return entities in the format word: entity.                    |\n| detect_pos              | Perform POS detection on the input text and return words with their respective POS tags.                   |\n| translate_text          | Translate the input text from a source language to a target language.                                       |\n| summarize_text          | Summarize the input text to the specified length.                                                          |\n| answer_question         | Answer a given question based on the input text.                                                           |\n| generate_text           | Generate a text of specified length based on a prompt.                                                     |\n| perform_srl             | Perform Semantic Role Labeling (SRL) detection on the input text and return predicates and roles.          |\n| recognize_intent        | Perform Intent Recognition detection on the input text and return the detected intent.                    |\n| paraphrasing_detection  | Determine if two sentences are paraphrases and return 'yes' or 'no' with an optional explanation.         |\n\n\n**Understanding File Structure**\n\nYou can skip this step as it is for later use only if you want to understand the library and how it works. Here is the file structure of this library.\n\n    main_directory/\n    |-- for_beginner/\n    |   |-- preprocessing.ipynb\n    |   |-- core_nlp.ipynb\n    |-- pre_processing.py\n    |-- core_nlp.py\n    |-- code_file.ipyb  # Containing example of each functions\n\nA “**for_beginner**” folder containing two Jupyter notebook files with code blocks for each NLP task that will make it easier for you to understand how this library is working. While both Python files are going to be used to call as modules and use them for your requested task.\n\npre_processing.py contains functions that are used for preprocessing our text, such as clean_text, remove_html_tags, etc., while core_nlp contains functions that are useful for handling text data and performing different tasks, such as summarize_text, translate_text, etc.\n\n**Installing the Library**\n\nFirst, you need to clone my GitHub repository.\n```bash\ngit clone https://github.com/FareedKhan-dev/Most-powerful-NLP-library.git\n```\nIf you don’t have Git installed on your machine, you can download the repository as a ZIP file.\n\n![[download repository as zip file from github link](https://github.com/FareedKhan-dev/Most-powerful-NLP-library)](https://cdn-images-1.medium.com/max/4676/1*82qE07mfI1EUX38ZkuaVzA.png)\n\nOnce you have cloned the repository, you need to install the required dependencies that allow you to work with the Gemini API.\n```bash\n# Install the Google Generative AI library\npip install -q -U google-generativeai\n```\n\n**Initiating the Library**\n\nIn the previous step, we installed the required dependencies and cloned our NLP library. Now, we need to import the necessary library that will fetch Gemini LLM API calls and instantiate the required API key.\n```python    \n# Import the Google Generative AI library\nimport google.generativeai as genai\n\n# Initialize the GenerativeModel with 'gemini-pro'\nmodel = genai.GenerativeModel('gemini-pro')\n\n# Configure the library with your API key\ngenai.configure(api_key=\"Your-API-key\")\n```\n\nYou can obtain your API key from [here](https://makersuite.google.com/app/apikey). Once you have the key, proceed to the next step.\n\n\n\n**Tokenization Example**\n\n\n```python\nfrom pre_processing import tokenize_text\n\nuser_input = '''The cats are running and playing in the gardens, while the dogs are barking loudly and chasing their tails'''\n\nmy_output = tokenize_text(user_input, model)\n\nprint(type(my_output), my_output)\n```\n\n    \u003cclass 'list'\u003e ['The', 'cats', 'are', 'running', 'and', 'playing', 'in', 'the', 'gardens', ',', 'while', 'the', 'dogs', 'are', 'barking', 'loudly', 'and', 'chasing', 'their', 'tails']\n    \n\n**Lemmatization Example**\n\n\n```python\nfrom pre_processing import lemmatize_text\n\n# Assuming 'your_model' is the instance of your model\nuser_input = '''The cats are running and playing in the gardens, while the dogs are barking loudly and chasing their tails'''\nlemmatized_sentence = lemmatize_text(user_input, model)\nprint(lemmatized_sentence)\n```\n\n    The cat be run and play in the garden, while the dog be bark loud and chase their tail\n    \n\n**Stemming Example**\n\n\n```python\nfrom pre_processing import stem_text\n\nuser_input = '''The cats are running and playing in the gardens, while the dogs are barking loudly and chasing their tails'''\n\nstemmed_sentence = stem_text(user_input, model)\n\nprint(stemmed_sentence)\n```\n\n    the cat ar run and play in the garden, whil the dog ar bark loud and chas their tail\n    \n\n**Pattern Matching Example**\n\n\n```python\nfrom pre_processing import extract_patterns\n\nuser_input = '''The phone number of fareed khan is 123-456-7890 and 523-456-7892. Please call for assistance and email me at x123@gmail.com'''\n\n# You can add more patterns here separated by commas\npattern_matching = '''emal, phone number, name'''\n\nextracted_patterns = extract_patterns(user_input, pattern_matching, model)\n\nprint(extracted_patterns)\n```\n\n    ['123-456-7890', '523-456-7892', 'x123@gmail.com', 'fareed khan']\n    \n\n**Text Cleaning Example**\n\n\n```python\nfrom pre_processing import clean_text\n\nuser_input = '''faree$$@$%d khan will arrive at 9:00 AM. He will@%$ 1meet you at the airport. He will be driving a black BMW. His license plate is 123-456-7890.'''\n\ncleaned_text = clean_text(user_input, model)\n\nprint(cleaned_text)\n\n```\n\n    Fareed Khan will arrive at 9:00 AM. He will meet you at the airport. He will be driving a black BMW. His license plate is 123-456-7890.\n    \n\n**HTML tags removal Example**\n\n```python\nfrom pre_processing import remove_html_tags\n\nuser_input = '''\u003cp\u003eThis is \u003cb\u003ebold\u003c/b\u003e and \u003ci\u003eitalic\u003c/i\u003e text.\u003c/p\u003e'''\n\n# You can add more tags here separated by commas\nhtml_tags = '''\u003cp\u003e, \u003cb\u003e, \u003ci\u003e''' \n\ncleaned_text = remove_html_tags(user_input, html_tags, model)\n\nprint(cleaned_text)\n\n```\n\n    This is bold and italic text.\n    \n\n**Replace text Example**\n\n\n```python\nfrom pre_processing import replace_text\n\nuser_input = '''I like cats, but I don't like dogs.'''\n\n# You can add more rules here separated by commas\nreplacement_rules = '''all animals to rabbits'''\n\nmodified_text = replace_text(user_input, replacement_rules, model)\n\nprint(modified_text)\n```\n\n    I like rabbits, but I don't like dogs.\n    \n\n**Generate Embedding Vectors Example**\n\n\n```python\nfrom pre_processing import extract_embeddings\n\nuser_input = [\"cats are running and playing in the gardens\", \"dogs are barking loudly and chasing their tails\"]\n\n# extract_embeddings() takes a list of strings as input\nmodified_text = extract_embeddings(user_input)\n\n# print first 10 values of embedding vector the first sentence\nmodified_text['embedding'][0][:10]\n```\n\n\n\n\n    [0.0195884,\n     0.024218114,\n     -0.029704109,\n     -0.05665759,\n     -0.011961627,\n     -0.026998892,\n     -0.024396203,\n     -0.021466378,\n     0.021265924,\n     -0.0027763597]\n\n\n\n**Text Classification:**\n   - Sentiment Analysis\n   - Topic Classification\n   - Spam Detection\n\n\n```python\nfrom core_nlp import analyze_sentiment\n\nuser_input = \"I love to play football, but today I am feeling very sad. I do not want to play football today.\"\n\n# You can add more categories here separated by commas (Default: positive, negative, neutral)\ncategory = \"positive, negative, neutral\"\n\nsentiment_result = analyze_sentiment(input_text=user_input, category=category, explanation=True, model=model)\nprint(sentiment_result)\n```\n\n    **Category: Negative**\n    \n    **Short Explanation:**\n    \n    The overall sentiment of the text is negative. The author expresses a love for football but then goes on to say that they are feeling very sad and do not want to play football today. This indicates a negative sentiment towards the activity of playing football.\n    \n\n\n```python\nfrom core_nlp import classify_topic\n\nuser_input = \"I love to play football, but today I am feeling very sad. I do not want to play football today.\"\n\n# You can add more topics here separated by commas (Default: story, horror, comedy)\ntopics = \"topics are: story, horror, comedy\"\n\ntopic_result = classify_topic(input_text=user_input, topics=topics, explanation=True, model=model)\n\nprint(topic_result)\n\n```\n\n    Topic: Story\n    Explanation: The input text is a story about a person who loves to play football but is feeling sad and does not want to play today. The text does not contain any elements of horror or comedy, so the topic is classified as \"story\".\n    \n\n**Spam Detection Example**\n\n\n```python\nfrom core_nlp import detect_spam\n\nuser_input = \"you have just won $14000, claim this award here at this link.\"\n\n# You can add more categories here separated by commas (Default: spam, not spam, unknown)\ncategory = 'spam, not_spam, unknown'\n\nspam_result = detect_spam(input_text=user_input, category=category, explanation=True, model=model)\n\nprint(spam_result)\n```\n\n    spam\n    \n    Explanation: The message contains the promise of a large monetary reward, which is a classic tactic used by spammers to attract attention and entice people to click on the link. The use of the word \"claim\" also indicates the sender's desire to obtain personal information from the recipient, which is another common goal of spammers.\n    \n\n**NER Detection Example**\n\n\n```python\nfrom core_nlp import detect_ner\n\nuser_input = \"I will meet you at the airport sharp 12:00 AM.\"\n\n# You can add more categories here separated by commas (Default: erson, location, date, number ... cardinal)\nner_tags = 'person, location, date, number, organization, time, money, percent, facility, product, event, language, law, ordinal, misc, quantity, cardinal'\n\nner_result = detect_ner(input_text=user_input, ner_tags=ner_tags, model=model)\nprint(ner_result)\n\n```\n\n    airport: facility\n    12:00 AM: time\n    \n\n**POS Tagging Example**\n\n\n```python\nfrom core_nlp import detect_pos\n\nuser_input = \"I will meet you at the airport sharp 12:00 AM.\"\n\n# you can add more categories here separated by commas (Default: NOUN, 'noun, verb, ..., cashtag_phrase, entity_phrase')\npos_tags = 'noun, verb, adjective, adverb, pronoun, preposition, conjunction, interjection, determiner, cardinal, foreign, number, date, time, ordinal, money, percent, symbol, punctuation, emoticon, hashtag, email, url, mention, phone, ip, cashtag, entity, noun_phrase, verb_phrase, adjective_phrase, adverb_phrase, pronoun_phrase, preposition_phrase, conjunction_phrase, interjection_phrase, determiner_phrase, cardinal_phrase, foreign_phrase, number_phrase, date_phrase, time_phrase, ordinal_phrase, money_phrase, percent_phrase, symbol_phrase, punctuation_phrase, emoticon_phrase, hashtag_phrase, email_phrase, url_phrase, mention_phrase, phone_phrase, ip_phrase, cashtag_phrase, entity_phrase'\n\npos_result = detect_pos(input_text=user_input, pos_tags=pos_tags, model=model)\n\nprint(pos_result)\n\n```\n\n    I: pronoun\n    will: verb\n    meet: verb\n    you: pronoun\n    at: preposition\n    the: determiner\n    airport: noun\n    sharp: adverb\n    12:00: time\n    AM: time\n    .: punctuation\n    \n\n**Machine Translation Example**\n\n\n```python\nfrom core_nlp import translate_text\n\nuser_input = \"I will meet you at the airport sharp 12:00 AM.\"\n\nsource_language = \"english\"\n\ntarget_language = \"spanish\"\n\ntranslation_result = translate_text(user_input, source_language, target_language, model)\n\nprint(translation_result)\n```\n\n    Te encontraré en el aeropuerto en punto de las 12:00 AM.\n    \n\n**Text Summarization Example**\n\n\n```python\nfrom core_nlp import summarize_text\n\nuser_input = \"I will meet you at the airport sharp 12:00 AM.\"\n\nsummary_length = \"medium\" # short, medium, long\n\nsummary_result = summarize_text(user_input, summary_length, model)\n\nprint(summary_result)\n\n```\n\n    You are requested to meet at the airport promptly at midnight.\n    \n\n**Question Answering Example**\n\n\n```python\nfrom core_nlp import answer_question\n\nquestion_text = \"Is it possible that an ant can kill a lion?\"\n\nanswer_result = answer_question(question_text, model=model)\n\nprint(answer_result)\n```\n\n    No, it is not possible for an ant to kill a lion.\n    \n\n**Text Generation Example**\n\n\n```python\nfrom core_nlp import generate_text\n\nprompt_text = \"poem on a friendship between a cat and a mouse\"\n\ngeneration_length = \"short\"\n\ngenerated_text = generate_text(prompt_text, generation_length, model)\n\nprint(generated_text)\n```\n\n    In a tale of unique bond, so true,\n    A cat and a mouse, friendship grew.\n    Amidst the world of chase and prey,\n    Their hearts entwined in a different way.\n    \n    The cat, playful and sleek and sly,\n    The mouse, nimble and bright of eye,\n    Met one day in the corner old,\n    Where stories and secrets were untold.\n    \n    They talked and laughed, they shared their dreams,\n    Of chasing stars and moonbeams.\n    No longer bound by predator or prey,\n    They found a bond that would never sway.\n    \n    Together they'd explore the night,\n    Underneath the silver moonlight.\n    A dance of shadows, soft and sweet,\n    Where differences were obsolete.\n    \n    They'd share their meals, they'd share their home,\n    A friendship that would forever roam.\n    In a world of chaos, a gentle grace,\n    A cat and a mouse, in harmony's embrace.\n    \n\n**Semantic Role Labeling (SRL) Example**\n\n\n```python\nfrom core_nlp import perform_srl\n\nuser_input = \"tornado is approaching the city, please take shelter\"\n\nsrl_result = perform_srl(user_input, model)\n\nprint(srl_result)\n```\n\n    Predicate: approach\n    Roles:\n    - Agent: tornado\n    - Theme: city\n    \n\n**Intent Recognition Example**\n\n\n```python\nfrom core_nlp import recognize_intent\n\nuser_input = \"tornado is approaching the city, please take shelter\"\n\nintent_result = recognize_intent(user_input, model)\n\nprint(intent_result)\n```\n\n    Intent: Emergency alert\n    \n\n**Paraphrasing Detection Example**\n\n\n```python\nfrom core_nlp import paraphrasing_detection\n\nuser_input = ['''The sun sets in the west every evening.''','''Every evening, the sun goes down in the west.''']\n\nintent_result = paraphrasing_detection(input_text=user_input, explanation=True, model=model)\n\nprint(intent_result)\n```\n\n    yes\n    Both sentences express the same idea that the sun sets in the west every evening. They use different words to convey the same meaning, such as \"sets\" and \"goes down\" for the verb and \"every evening\" for the temporal modifier.\n    \n## Handling Large Data\n\nUp until now, we’ve worked with relatively small text data, like short sentences. If you need to handle larger text, while I haven’t implemented it yet, one approach is to break your text data into chunks and process it accordingly. Here’s an example of how to work with a bigger dataset.\n\n    # Example text dataset\n    text_dataset = \"some_big_text_file.txt\"\n    \n    # Break the text into sentences based on full stops\n    sentences = text_dataset.split('. ')\n    \n    # some ner_tags you have defined\n    ner_tags = \"person, organization ...\"\n    \n    # Applying NER on it\n    for i, sentence in enumerate(sentences):\n        print(f\"Sentence {i + 1}:\")\n        \n        # Applying NER on each sentence\n        detect_ner(input_text=sentence, ner_tags=ner_tags, model=model)\n\nAnother approach to handling larger data is to break it into more extensive chunks, for example, 500 sentences per chunk, to preserve dataset information. If you want to apply the text_summarization task, you can then provide the summaries of each chunk in a combined manner to generate one detailed summary for the entire text.\n\n![Visual Illustration of how to handle large text data](large_text_working.png)\n\nThere are several ways to handle big data, but the approaches I’ve just shared are among the most common and practical.\n\n**Customizing the Library**\n\nCustomizing the library involves including your own functions, and a well-crafted prompt is essential for making your customized functions work. For instance, if you want to create a paraphrasing-checking function, you need to start with a prompt for the paraphrasing task.\n```python\n# Question to be asked for determining paraphrasing\nquestion = f'''Given the input text, determine if two sentences are paraphrases of each other.\nSentence 1: {user_input[0]}\nSentence 2: {user_input[1]}\nAnswer must be 'yes' or 'no'.\n{explanation}\n'''\n```\n\nWhen creating a customized function, it’s crucial to explain the expected output from Gemini to maintain consistency across runs. Additionally, defining the answer format is essential; for instance, in tokenization, you may specify that the output format should be a list. To achieve this, you can later convert the string representation to an actual list using the ast Python library. In the paraphrasing task, the rest of the prompt remains relatively constant, with changes depending on how many sentences you want to input—I've considered two in this example.\n\nOnce you create your prompt, you can build a function on top of it.\n```python\n# function for paraphrase detection\ndef paraphrasing_detection(input_text, explanation, model):\n\n    # Check if explanation is required\n    explanation_text = 'short explanation: ' if explanation else 'no explanation'\n\n    # Question to be asked for determining paraphrasing\n    question = f'''Given the input text, determine if two sentences are paraphrases of each other.\n    Sentence 1: {input_text[0]}\n    Sentence 2: {input_text[1]}\n    Answer must be 'yes' or 'no'.\n    {explanation_text}\n    '''\n\n    # Generate response\n    response = model.generate_content(question)\n    return response.text.strip()\n```\nYou can easily call that function on top of your text data.\n```pytho\n# Import the paraphrasing_detection function from the core_nlp module\nfrom core_nlp import paraphrasing_detection\n\n# User input text\nuser_input = ['''The sun sets in the west every evening.''', '''Every evening, the sun goes down in the west.''']\n\n# Perform paraphrasing detection using the specified model\nintent_result = paraphrasing_detection(user_input, explanation=True, model=model)\n\n# Print the paraphrasing detection result\nprint(intent_result)\n```\n\n```bash\n##### OUTPUT OF ABOVE CODE #####\n\nAnswer: yes\nShort Explanation: Both sentences express the same idea that the sun \nsets in the west  every evening. They use different words to convey \nthe same meaning,  such as \"sets\" and \"goes down\" for the verb and \n\"every evening\" for temporal modifier.\n\n##### OUTPUT OF ABOVE CODE #####\n```\n\n**What’s Next**\n\nThere are many more features introduced in this library. This is just a glimpse of how LLMs reshape NLP tasks and simplify the handling of text data. Explore the full potential by checking out my [GitHub repository](https://github.com/FareedKhan-dev/Most-powerful-NLP-library), which includes features like generating embeddings for cosine similarity, text summarization, and more. Feel free to adapt the library for your specific domain, whether it’s medical or any other. I hope you enjoy reading this blog.\n\nIf you want to build your own LLM from scratch or understand the mathematical aspects of transformers, you can refer to my other blogs:\n* [**Solving Transformer by Hand: A Step-by-Step Math Example**\n*Performing numerous matrix multiplications to solve the encoder and decoder parts of the transformer*levelup.gitconnected.com](https://levelup.gitconnected.com/understanding-transformers-from-start-to-end-a-step-by-step-math-example-16d4e64e6eb1)\n* [**Building a Million-Parameter LLM from Scratch Using Python**\n*A Step-by-Step Guide to Replicating LLaMA Architecture*levelup.gitconnected.com](https://levelup.gitconnected.com/building-a-million-parameter-llm-from-scratch-using-python-f612398f06c2)\n\n\u003chr\u003e\n\n**License**\nMIT License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffareedkhan-dev%2Fmost-powerful-nlp-library","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffareedkhan-dev%2Fmost-powerful-nlp-library","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffareedkhan-dev%2Fmost-powerful-nlp-library/lists"}