{"id":26142500,"url":"https://github.com/vicotrbb/pylexitext","last_synced_at":"2025-04-14T02:06:42.974Z","repository":{"id":51218989,"uuid":"314664556","full_name":"vicotrbb/Pylexitext","owner":"vicotrbb","description":"Pylexitext is a python library that aggregates a series of NLP methods, text analysis, content converters and other usefull stuff.","archived":false,"fork":false,"pushed_at":"2021-05-19T22:48:06.000Z","size":175,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-14T02:06:32.125Z","etag":null,"topics":["data-science","machine-learning","nlp","python","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vicotrbb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-20T20:41:09.000Z","updated_at":"2022-04-06T18:58:41.000Z","dependencies_parsed_at":"2022-09-05T13:40:22.285Z","dependency_job_id":null,"html_url":"https://github.com/vicotrbb/Pylexitext","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vicotrbb%2FPylexitext","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vicotrbb%2FPylexitext/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vicotrbb%2FPylexitext/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vicotrbb%2FPylexitext/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vicotrbb","download_url":"https://codeload.github.com/vicotrbb/Pylexitext/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248809044,"owners_count":21164896,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","machine-learning","nlp","python","python3"],"created_at":"2025-03-11T03:56:43.403Z","updated_at":"2025-04-14T02:06:42.935Z","avatar_url":"https://github.com/vicotrbb.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pylexitext\r\n\r\n\u003cimg src=\"https://img.shields.io/github/issues/vicotrbb/pylexitext\"\u003e \u003cimg src=\"https://img.shields.io/github/workflow/status/vicotrbb/Pylexitext/Python%20application\"\u003e \u003cimg src=\"https://img.shields.io/github/commit-activity/w/vicotrbb/Pylexitext\"\u003e \u003cimg src=\"https://img.shields.io/pypi/dm/pylexitext\"\u003e\r\n\r\nPylexitext is a python library that aggregates a series of NLP methods, text analysis, content converters and other usefull stuff.\r\n\r\n## Supported languages\r\n\r\n- English\r\n\r\n## How to use\r\n\r\nFirst you need to install the library using pip.\r\n\r\n```\r\npip install pylexitext\r\n```\r\n\r\nPylexitext uses a main object called `text` that wrapps all the text functions and some helpers to perform aditional functions.\r\nA basic functionality would looks like this:\r\n\r\n```\r\nfrom pylexitext import text\r\n\r\nsample = text.Text('\u003cYOUR TEXT\u003e')\r\nsample.describe()\r\n```\r\n\r\nThis script will load the pylexitext object with your text, perform all the pre-processing and then, with the `describe()` method, return to you a dict with some proprierties of your text.\r\n\r\nWith the text:\r\n\r\n```\r\nBest hello world ever made by a Developer.\r\n```\r\n\r\nThe output would be:\r\n\r\n```\r\n{'text_size': 42, 'total_words': 8, 'char_count': 35, 'non_stop_words': ['best', 'hello', 'world', 'ever', 'made', 'developer.'], 'stop_words': ['by', 'a'], 'stop_words_number': 2, 'unique_terms': {'made', 'hello', 'ever', 'best', 'developer.', 'world'}, 'unique_words': 6, 'sentences': ['best hello world ever made by a developer', ''], 'number_senteces': 2, 'lexical_diversity': 100.0, 'frequency_distribution': FreqDist({'best': 1, 'hello': 1, 'world': 1, 'ever': 1, 'made': 1, 'developer.': 1}), 'total_syllables': 13, 'total_polysyllables': 1, 'flesch_reading_ease_score': 65.13749999999999, 'flesch_kincaid_grade_level_score': 5.145, 'smog_score': 7.168621630094336, 'gunning_fog_index_score': 15.7}\r\n```\r\n\r\nThose are all the proprierties described by pylexitext:\r\n\r\n- Text size\r\n- Number of words\r\n- List of stopwords\r\n- Characteres count\r\n- List of words wout/ stopwords\r\n- Number of words wout/ stopwords\r\n- Number of present stopwords\r\n- Unique words\r\n- Number of unique words\r\n- Number of sentences\r\n- Lexical diversity (%)\r\n- Total syllables\r\n- Total polysyllables\r\n- Flesch reading ease score\r\n- Flesch kincaid grade level score\r\n- Smog score\r\n- Gunning fog index score(Not ready!)\r\n\r\n## Create a summary from your text\r\n\r\nPylexitext can create summaries of your texts using sentences ranking, generating and joining chunks. By default the number of chunks generated are 3.\r\n\r\nUsually, this function don't work well for small texts and if your text is big, you should generate more chunks(improving the final result).\r\n\r\n```\r\nfrom pylexitext import text\r\n\r\nsample = text.Text('\u003cYOUR BIG TEXT\u003e')\r\nsample.summarize(top_n=5)\r\n```\r\n\r\n## Part-of-speech(POS) tagging\r\n\r\nUsing NLTK, Pylexitext can perform a grammatical tagging which is the process of marking up a word in a text (corpus) as corresponding to a particular part of speech.\r\n\r\nThe embedded parameter is used to join the tag and the word, if False, the result will be a tuple.\r\n\r\n```\r\nfrom pylexitext import text\r\n\r\nsample = text.Text('Best hello world ever made by a Developer.')\r\nsample.speech_tagging(embedded=True)\r\n```\r\n\r\nOutput:\r\n\r\n```\r\n['best_JJS', 'hello_NN', 'world_NN', 'ever_RB', 'made_VBN', 'by_IN', 'a_DT', 'developer_NN', '._.']\r\n```\r\n\r\n## Generation of ngrams\r\n\r\nPylexitext can extracts ngrams from the text, which is a list of n(default=3) words from the text.\r\n\r\nThere is also a method `bigrams_extraction`, that extracts a bigram(2 words) by default.\r\n\r\n```\r\nfrom pylexitext import text\r\n\r\nsample = text.Text('Best hello world ever made by a Developer.')\r\nsample.ngrams_extraction(n=3)\r\n```\r\n\r\noutput:\r\n\r\n```\r\n[['best', 'hello', 'world'], ['hello', 'world', 'ever'], ['world', 'ever', 'made'], ['ever', 'made', 'by'], ['made', 'by', 'a'], ['by', 'a', 'developer']]\r\n```\r\n\r\n## Text stemming\r\n\r\nText stemming is a normalization method to return inflacted words to it's morphological original form.\r\n\r\nEx: fishing, fished, and fisher -\u003e fish\r\n\r\n```\r\nfrom pylexitext import text\r\n\r\nsample = text.Text(\"I'm coding it to be the best application.\")\r\nsample.stemming()\r\n```\r\n\r\noutput:\r\n\r\n```\r\ni'm code it to be the best application.\r\n```\r\n\r\n## Text Lexical Graph generation \u0026 plotting\r\n\r\nPylexitext can generate a lexical graph from the cleaned raw text at the Text object, this graph represents all the possible connections between words, being unique words as vertex and the connections as edges.\r\n\r\n```\r\nfrom pylexitext import text\r\n\r\nsample = text.Text(\"I'm coding it to be the best application.\")\r\nsample.lexical_graph()\r\n\r\n# {'im': ['coding'], 'coding': ['it'], 'it': ['to'], 'to': ['be'], 'be': ['the'] , 'the': ['best'], 'best': ['application'], 'application': []}\r\n```\r\n\r\nAs a visualization resource, you can easily plot the generated graph using the **lexical_graph_plot** method, that creates a pyploy graph for you.\r\n\r\n```\r\nfrom pylexitext import text\r\n\r\nsample = text.Text(\"I'm coding it to be the best application.\")\r\nsample.lexical_graph_plot()\r\n```\r\n\r\nThis method can be used as static from the **pylexitext.plots** as well.\r\n\r\n## Text Normalization\r\n\r\nText normalization is a series of techniques used to \"clean\" the text to it's most base level, trying to reduce the randomness os the text. Usually, this type of method is used to pre-process text before use on NLP/ML models.\r\n\r\n```\r\nfrom pylexitext import text\r\n\r\nsample = text.Text(\"I'm coding it to be the best application.\")\r\nsample.normalization()\r\n```\r\n\r\noutput:\r\n\r\n```\r\nim code best applic\r\n```\r\n\r\n## Static methods\r\n\r\nPylexitext has some usefull static methods for text processment and normalization, that can be used without define a main Text object.\r\n\r\nThose methods are:\r\n\r\n```\r\nfrom pylexitext.text import remove_numbers, remove_punctuation, remove_extra_whitespace_tabs, remove_non_unicode, noise_remoaval\r\n\r\nremove_numbers('Hi1 I'm    Victor Ceñía')\r\n# Hi I'm    Victor Ceñía\r\n\r\nremove_punctuation('Hi I'm    Victor Ceñía')\r\n# Hi Im    Victor Ceñía\r\n\r\nremove_numbers('Hi Im    Victor Ceñía')\r\n# Hi Im Victor Ceñía\r\n\r\nremove_non_unicode('Ceñía')\r\n# Hi Im Victor Cea\r\n\r\nnoise_removal('Hi1 I'm    Victor Ceñía')\r\n# hi Im victor cea\r\n```\r\n\r\n### Sentence similarity\r\n\r\nSentence similarity static method uses levenshtein distance method to compoare and calculate the similarity of two sentences.\r\n\r\n```\r\nfrom pylexitext.text import sentence_similarity\r\n\r\nsentence_similarity('hello beautiful world', 'hello world')\r\n# 0.8598892366800223\r\n\r\n# You can get the output in 0-100% as well:\r\nsentence_similarity('hello beautiful world', 'hello world', percentage_base=True)\r\n# 85.99\r\n```\r\n\r\n## Engines\r\n\r\nPylexitext provides a series of usefull Text Engines.\r\n\r\n### Search Engine\r\n\r\nThe search engine provide an easy way to search for a query string match on list of documents, this documents should be \"summaries\" from bigger documents, that, one time\r\nfound on the documents portifolio, could lead to the complete original document.\r\n\r\nThe engine object itself hold the documents, and handle all the search protocol to find the desired document. A document search will have a relevance score, that quantifies how much the document matches with the search query. The match score have a treshold of 0 by default, that can be changed if needed, adding more confidence to the search, but reducing the number of results.\r\n\r\nBellow an example of how to create the Engine object:\r\n\r\n```\r\nfrom pylexitext.engines import SearchEngine\r\n\r\ndocuments = [\r\n  '''At Scale You Will Hit Every Performance Issue I used to think I knew a bit about performance scalability and how to keep things trucking when you hit large amounts of data Truth is I know diddly squat on the subject since the most I have ever done is read about how its done To understand how I came about realising this you need some background''',\r\n  '''Richard Stallman to visit Australia Im not usually one to promote events and the like unless I feel there is a genuine benefit to be had by attending but this is one stands out Richard M Stallman the guru of Free Software is coming Down Under to hold a talk You can read about him here Open Source Celebrity to visit Australia''',\r\n  '''MySQL Backups Done Easily One thing that comes up a lot on sites like Stackoverflow and the like is how to backup MySQL databases The first answer is usually use mysqldump This is all fine and good till you start to want to dump multiple databases You can do this all in one like using the all databases option however this makes restoring a single database an issue since you have to parse out the parts you want which can be a pain'''\r\n]\r\n\r\ndocument =  '''The Great Benefit of Test Driven Development Nobody Talks About The feeling of productivity because you are writing lots of code Think about that for a moment Ask any developer who wants to develop why they became a developer One of the first things that comes up is I enjoy writing code This is one of the things that I personally enjoy doing Writing code any code especially when its solving my current problem makes me feel productive It makes me feel like Im getting somewhere Its empowering'''\r\n\r\n# By default, match_threshold is 0.0\r\nengine = SearchEngine(match_threshold=0.2)\r\n\r\n# Adds a single document to the Engine portifolio and returns the index of the document.\r\nengine.add_doc(document)\r\n\r\n# Adds a list of documents to the Engine portifolio.\r\nengine.extend_docs(documents)\r\n\r\n# We could remove a document using it's index:\r\n# engine.remove_doc(doc_ix=?)\r\n\r\n# To list the documents, we could use, that will return a list of sets with (doc index, document string):\r\n# engine.get_docs()\r\n\r\n# Search a string query on the Engine documents portifolio, by default, it returns the top 1 result.\r\n# To change the number of results, change the parameter 'top_n'\r\nengines.search('How to backup a MySQL database', top_n=2)\r\n```\r\n\r\nThe search method will return the two best matchs for the query:\r\n\r\n```\r\n [(0.39346959912353996,\r\n 5,\r\n 'setting up git to use a subversion svn style workflow moving from subversion svn to git can be a little confusing at first i think the biggest thing i noticed was that git doesnt have a specific workflow you have to pick your own personally i wanted to stick to my subversion like work-flow with a central server which all my machines would pull and push too since it took a while to set up i thought i would throw up a blog post on how to do it'),\r\n (0.38138503569823695,\r\n  2,\r\n  'mysql backups done easily one thing that comes up a lot on sites like stackoverflow and the like is how to backup mysql databases the first answer is usually use mysqldump this is all fine and good till you start to want to dump multiple databases you can do this all in one like using the all databases option however this makes restoring a single database an issue since you have to parse out the parts you want which can be a pain')]\r\n```\r\n\r\nOther available methods are:\r\n\r\n```\r\nengine.save_to_file()\r\n# Dump the Search Engine object to a json file and returns the name of the generated document.\r\n# Default file_name pattern: '\u003ctodays datetime as %d-%m-%Y-%H-%M-%S\u003e-search_engine-\u003cnumber of documents\u003e-docs.json'\r\n\r\nengine.load_from_file(file_name='\u003cFILE NAME\u003e.json')\r\n# Load a Search Engine object from json file.\r\n```\r\n\r\n### Static methods\r\n\r\nAll the static methods available are related to the search process of the engine, and can be used as the example below:\r\n\r\n```\r\nSearchEngine.extract_concordance_dict('\u003cDOCUMENT STRING\u003e')\r\n# Create a concordance dict from the input text.\r\n# A concordance dict counts the number of occurences of a word.\r\n\r\nSearchEngine.magnitude(\u003cCONCORDANCE DICT\u003e)\r\n# Calculates the n-dimensions vector space from a concordance dict.\r\n\r\nSearchEngine.find_relation(query, \u003cCONCORDANCE DICT\u003e)\r\n# Scores the relation between a query and a concordance dict, it is used to search for top matchind documents\r\n# with the query.\r\n```\r\n\r\n## About Creator\r\n\r\nFind me on:\r\n\r\n💡 https://github.com/vicotrbb  \r\n📊 https://www.linkedin.com/in/victorbona/\r\n\r\n## Collaborations\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvicotrbb%2Fpylexitext","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvicotrbb%2Fpylexitext","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvicotrbb%2Fpylexitext/lists"}