{"id":14997620,"url":"https://github.com/jblake1965/elucidoc","last_synced_at":"2026-01-04T03:39:59.168Z","repository":{"id":179852831,"uuid":"658001542","full_name":"jblake1965/eluciDoc","owner":"jblake1965","description":"Screens legal text and extracts sentences containing user input party name-predicate phrases","archived":false,"fork":false,"pushed_at":"2024-05-22T11:29:28.000Z","size":1031,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"developer","last_synced_at":"2024-05-22T12:36:55.196Z","etag":null,"topics":["excel","law","legal-documents","legal-text-analytics","natural-language-processing","python-script","python3","spacy","textacy","word"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jblake1965.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-24T13:09:58.000Z","updated_at":"2024-05-28T12:16:02.490Z","dependencies_parsed_at":"2024-01-03T12:25:43.091Z","dependency_job_id":"19d53c6e-0a8c-4768-ad04-20c3a1213173","html_url":"https://github.com/jblake1965/eluciDoc","commit_stats":null,"previous_names":["jblake1965/elucidoc"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jblake1965%2FeluciDoc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jblake1965%2FeluciDoc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jblake1965%2FeluciDoc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jblake1965%2FeluciDoc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jblake1965","download_url":"https://codeload.github.com/jblake1965/eluciDoc/tar.gz/refs/heads/developer","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244045528,"owners_count":20388950,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["excel","law","legal-documents","legal-text-analytics","natural-language-processing","python-script","python3","spacy","textacy","word"],"created_at":"2024-09-24T17:05:03.716Z","updated_at":"2026-01-04T03:39:59.141Z","avatar_url":"https://github.com/jblake1965.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![eluciDoc_header](https://github.com/jblake1965/eluciDoc/assets/100727736/e7f94b7f-fb1b-4f55-8665-4dc11c6b93af)\n\n[![CodeQL](https://github.com/jblake1965/eluciDoc/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/jblake1965/eluciDoc/actions/workflows/github-code-scanning/codeql) [![GitHub Discussions](https://img.shields.io/github/discussions/jblake1965/eluciDoc?labelColor=blue\u0026color=orange)](https://github.com/jblake1965/eluciDoc/discussions/3) [![PYPI Version](https://img.shields.io/pypi/v/elucidoc?logoColor=blue\u0026labelColor=green)](https://img.shields.io/pypi/v/elucidoc?logoColor=blue\u0026labelColor=green)\n\n# What this is:\nThis CLI Python project, written for the Windows™ environment, filters sentences and clauses containing specific user input\nterms from a single document. This project was originally created as a tool to aid in the review of legal contracts, \nbut can be used with any text. Documents can be in docx, .pdf or .txt file formats.\nThe general principle behind its function is subject-predicate sentence analysis. Searches are based on user-selected parties\nin the document, followed by a user-selected phrase.  It is used in conjunction with Microsoft™ Office 365™\nWord and Excel™ apps.\n# How it works:\nA .docx, .pdf or .txt file and path is entered (drag and drop work in the Windows terminal):\n\n![file_input](https://github.com/jblake1965/eluciDoc/assets/100727736/c08d59a4-a019-4a42-b895-427a1815b474)\n\nThe file is then processed as utf8 text, with MS Word Smart Quotes being converted to straight quotes and non-ASCII and\nnon-breaking spaces removed. The term for the party being searched in the document is entered next:\n\n![enter_party_name](https://github.com/jblake1965/eluciDoc/assets/100727736/bd1e9603-137c-4475-aa1d-d09ed157738a)\n\nand then passed with the processed text to textacy's Keyword in Context (KWIC) function.  The result is saved as an Excel\nfile with the same name in the same location as the searched document, with \"..._[name of the party]_search_result.xlsx\" \nappended. The Excel file automatically opens with a subprocess call, and the results can be converted to a table for\nfurther sorting:\n\n![textacy_rendering](https://github.com/jblake1965/eluciDoc/assets/100727736/a9bfd1a8-8477-4401-8e96-bd83801d5488)\n\nNote: the subprocess call below uses the default Office install location:\n\n```python\nsubprocess.Popen([r'C:\\Program Files\\Microsoft Office\\root\\Office16\\EXCEL.EXE', result_file])\n```\n\nIf the user has Office installed in a different location, then the code must be changed to reflect that directory.\n\nThe document is chunked into sentences (or clauses, depending on the formatting) with the spaCy module.\nThe user is prompted to enter predicate search phrases culled from the Excel search file which phrases are stored in a list.  \nOnce finished entering the predicate search phrases, the script iterates through the list of search phrases looking for\na match in each sentence. Sentences and clauses containing a match are added to a result list. The user has the option of \nhaving the search phrases appear in ALL CAPS in the Word document containing the results, as shown:\n\n![all_caps_rendering](https://github.com/user-attachments/assets/5d07f8a5-ed26-4341-9bf8-0445de315245)\n\n\nThe result list is then saved as a Word file that is opened automatically at the end of the run (as with Excel, \nnote the location of the Word executable and adjust the path if it is not in the standard install location). \n# External Dependencies and Licenses\n\n| Name:        | License:                                                              |\n|--------------|-----------------------------------------------------------------------|\n| docx2python  | [MIT](https://pypi.org/project/docx2python/)                          |\n| openpyxl     | [MIT](https://pypi.org/project/openpyxl/)                             |\n| pandas       | [BSD](https://pypi.org/project/pandas/)                               |\n| pdfminer.six | [MIT/X](https://github.com/pdfminer/pdfminer.six/blob/master/LICENSE) |\n| python-docx  | [MIT](https://github.com/atriumlts/python-docx/blob/master/LICENSE)   |\n| rich         | [MIT](https://pypi.org/project/rich/)                                 |\n| spacy        | [MIT](https://pypi.org/project/spacy/)                                |\n| textacy      | [Apache 2.0](https://pypi.org/project/textacy/)                       |\n\n# Installation\nIt is strongly recommended that this package be installed in a virtual environment.  The package is available at https://pypi.org/project/elucidoc/ \nand can be installed with ```pip install elucidoc``` .\n\n***THE SPACY PIPELINE  `  en_core_web_lg  `  MUST ALSO BE INSTALLED INTO THE VIRTUAL\nENVIRONMENT FOR THE SCRIPT TO WORK***.\n\nThe pipeline can be installed as follows:\n```\npython -m spacy download en_core_web_lg\n```\nYou must also be sure to verify the directory for the Office install is the same as noted above.  If not, the code must be \nchanged to the directory where the Excel and Word apps are located.\n# Running the Script\nThe project is run as a script.  It can be run with a .bat file calling the virtual environment and the executable\nfile per the below example:\n```commandline\n@\"C:\\Users\\..\\venv\\Scripts\\python.exe\" \"C:\\Users\\..\\venv\\lib\\elucidoc\\eluciDoc.py\"\n\n@pause\n```\nAdditionally, the location of the ```elucidoc.py``` executable can be included in the Windows ```PATH``` environment variable.\n# Case Sensitive Searches\nGeneral convention in legal texts is to capitalize defined terms.  For that reason, the user may want to make the search\ncase-sensitive to target the appropriate instances of the term.  For searches where the specific use of the subject term\nis not important but broader capture is, the case-sensitive feature can be turned off.  Once a selection is made, it applies\nfor all subsequent searches until the script is restarted.\n# Possessive Case and Other Punctuation\nTextacy divides the party search term from both following words and punctuation including the possessive case, as shown below:\n\n![textacy_rendering](https://github.com/jblake1965/eluciDoc/assets/100727736/1fd67f92-57bd-402a-b99f-95d5847f49f7)\n\nTo capture an instance of a possessive case of the party being searched, a 's or ' (for the plural possessive) must be\nthe first character in the predicate search phrase, as illustrated by the prompt below:\n\n![enter_predicate_phrase](https://github.com/jblake1965/eluciDoc/assets/100727736/edc9f616-97d7-4bdc-8553-f89292e43332)\n\nThe same principal applies to the comma, colon, semicolon and closed parentheses immediately following the party name.\n\n# Smart Quotes\nMicrosoft Word's default settings utilize smart quotes, which are the\ncurly type fonts. Those are problematic when searching\ndocuments converted to text (rendered as slanted quotes in Utf8), and\nare replaced with straight quotes via the following code:\n\n```python\ntext = re.sub(r'”', '\\\"', text)  # replace double smartquote open quote\ntext = re.sub(r'“', '\\\"', text)  # replace double smartquote close quote\ntext = re.sub(r'’', '\\'', text)  # replace single smartquote close quote\ntext = re.sub(r'‘', '\\'', text)  # replace single smartquote open quote\n```\n\n# PDFs\nDue to the nature of .pdf files and the sometimes-inconsistent results\nthat occur when converting pdf documents to text format, additional\nprocessing is done. Some characters and extra spaces between word boundaries are removed as part of the\ntext processing:\n```python\ntext = re.sub(u'[^\\u0020-\\uD7FF\\u0009\\u000A\\u000D\\uE000-\\uFFFD\\U00010000-\\U0010FFFF]+', '', text)\ntext = re.sub(r'(\\b)(\\s{2,4})(\\b)', r'\\g\u003c1\u003e ', text)\n```\nThe above solution is not a comprehensive fix for pdf issues. The accuracy of the results with searches of .pdf files\nmay be negatively impacted by the quality or formatting of the underlying document, particularly with\nscanned documents.\n\n# Open Files\nIf a consecutive search is run for the same party and the Excel file with the prior search results is still open,\nthe script will notify the user of such and not overwrite the existing Excel file.  With the Word files, the user will \nbe prompted to save the existing file with\nanother name and close it before proceeding with a second search for the same party.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjblake1965%2Felucidoc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjblake1965%2Felucidoc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjblake1965%2Felucidoc/lists"}