{"id":23561859,"url":"https://github.com/parthapray/nlp_pipeline_openai","last_synced_at":"2026-04-10T23:03:29.767Z","repository":{"id":269788284,"uuid":"908463418","full_name":"ParthaPRay/nlp_pipeline_openai","owner":"ParthaPRay","description":"This repo contains nlp pipeline and openai API integration","archived":false,"fork":false,"pushed_at":"2024-12-26T07:01:22.000Z","size":10882,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-17T15:51:52.458Z","etag":null,"topics":["gradio","matplotlib","networkx","nltk","openai","rake-nltk","scikit-learn","seaborn","spacy","textblob","textstat","wordcloud"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ParthaPRay.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-26T06:17:06.000Z","updated_at":"2024-12-26T07:01:25.000Z","dependencies_parsed_at":"2024-12-26T07:25:25.591Z","dependency_job_id":"f95d6e72-713b-4109-b62b-2b05901d3d7d","html_url":"https://github.com/ParthaPRay/nlp_pipeline_openai","commit_stats":null,"previous_names":["parthapray/nlp_pipeline_openai"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ParthaPRay%2Fnlp_pipeline_openai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ParthaPRay%2Fnlp_pipeline_openai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ParthaPRay%2Fnlp_pipeline_openai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ParthaPRay%2Fnlp_pipeline_openai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ParthaPRay","download_url":"https://codeload.github.com/ParthaPRay/nlp_pipeline_openai/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254455632,"owners_count":22074021,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gradio","matplotlib","networkx","nltk","openai","rake-nltk","scikit-learn","seaborn","spacy","textblob","textstat","wordcloud"],"created_at":"2024-12-26T16:13:24.716Z","updated_at":"2026-04-10T23:03:24.715Z","avatar_url":"https://github.com/ParthaPRay.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NLP Pipeline with Graceful Clustering\n\nThis **Natural Language Processing (NLP) Pipeline** along with **gpt-4o-mini** provides a comprehensive solution for analyzing, clustering, and visualizing text data. It integrates advanced machine learning techniques with a user-friendly **Gradio interface**, enabling users to interactively explore results with structured outputs and dynamic visualizations.\n\n---\n\n## Key Features\n\n1. **Text Preprocessing**:\n   - Tokenization, stopword removal, and POS tagging.\n   - Named Entity Recognition (NER) for identifying entities.\n2. **Feature Extraction**:\n   - **TF-IDF Analysis**: Highlights significant terms.\n   - **Keyword Extraction**: Uses RAKE for extracting relevant phrases.\n3. **Analysis**:\n   - **Sentiment Analysis**: Evaluates text polarity and subjectivity.\n   - **Readability Metrics**: Calculates text complexity using multiple readability indices.\n   - **Dependency Parsing**: Identifies linguistic dependencies.\n4. **Clustering**:\n   - Groups documents based on similarity using KMeans clustering.\n5. **Topic Modeling**:\n   - Identifies dominant themes in documents using Latent Dirichlet Allocation (LDA).\n6. **Visualization**:\n   - **Word Cloud**: Displays frequent terms.\n   - **TF-IDF Bar Chart**: Highlights keyword scores.\n   - **Co-occurrence Network**: Visualizes relationships between terms.\n   - **Polarity Heatmap**: Displays sentence-level sentiment variations.\n7. **Interactive Interface**:\n   - Powered by **Gradio**, offering an easy-to-use web-based interface for exploring results.\n\n---\n\n## Requirements\n\n### Dependencies\nThe required Python packages are listed in `requirements.txt`:\n```plaintext\nspacy\nwordcloud\nnetworkx\nnltk\ntextblob\nscikit-learn\nseaborn\nmatplotlib\nrake-nltk\ntextstat\ngradio\nopenai\n```\n\n### Installation\n1. Clone the repository:\n   ```bash\n   git clone https://github.com/ParthaPRay/nlp_pipeline_openai/.git\n   cd nlp_pipeline_openai\n   ```\n\n2. Install the dependencies:\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n3. Download required NLTK and SpaCy resources:\n   ```bash\n   python -c \"import nltk; nltk.download('punkt'); nltk.download('stopwords')\"\n   python -m spacy download en_core_web_sm\n   ```\n\n---\n\n## Workflow\n\n### Workflow Overview\n\n1. **Input Text and Documents**:\n   - Users provide a text string for analysis and optional documents for comparison.\n   \n2. **Preprocessing**:\n   - Tokenize text, remove stopwords, and tag parts of speech.\n   - Extract named entities and clean tokens for further analysis.\n   \n3. **Feature Extraction**:\n   - Compute **TF-IDF** scores for identifying important terms.\n   - Extract keywords using **RAKE**.\n\n4. **Analysis**:\n   - Perform **sentiment analysis** to evaluate polarity and subjectivity.\n   - Assess **readability metrics** using indices like Flesch Reading Ease.\n   - Parse linguistic dependencies to understand relationships in text.\n\n5. **Clustering and Topic Modeling**:\n   - Group similar documents using **KMeans clustering**.\n   - Identify key topics with **LDA (Latent Dirichlet Allocation)**.\n\n6. **Visualization**:\n   - Generate visual outputs like:\n     - **Word Cloud**\n     - **TF-IDF Chart**\n     - **Co-occurrence Network**\n     - **Polarity Heatmap**\n\n7. **Interactive Results**:\n   - Use **Gradio** for an intuitive, web-based exploration of results.\n\n---\n\n## Workflow Diagram\n\n```mermaid\ngraph TD\n    A[Input Text/Documents] --\u003e B[Preprocessing]\n    B --\u003e C[Feature Extraction]\n    C --\u003e D[Sentiment Analysis]\n    C --\u003e E[Topic Modeling]\n    C --\u003e F[Clustering]\n    D --\u003e G[Visualization]\n    E --\u003e G\n    F --\u003e G\n    G --\u003e H[Interactive UI with Gradio]\n```\n\n---\n\n## Code Structure\n\n### Key Functions\n\n#### Text Preprocessing\n```python\ndef dependency_parsing(text):\n    doc = nlp(text)\n    for token in doc:\n        print(f\"{token.text} -\u003e {token.dep_} -\u003e {token.head.text}\")\n```\n\n#### Feature Extraction\n```python\ndef compute_tfidf(documents, top_n=5):\n    vectorizer = TfidfVectorizer(stop_words=\"english\")\n    tfidf_matrix = vectorizer.fit_transform(documents)\n    feature_names = vectorizer.get_feature_names_out()\n    dense = tfidf_matrix.todense()\n    scores = dense[0].tolist()[0]\n    tfidf_scores = [(feature_names[i], scores[i]) for i in range(len(scores))]\n    sorted_scores = sorted(tfidf_scores, key=lambda x: x[1], reverse=True)\n    return sorted_scores[:top_n]\n```\n\n#### Clustering\n```python\ndef cluster_documents(documents, n_clusters=3):\n    vectorizer = TfidfVectorizer(stop_words=\"english\")\n    tfidf_matrix = vectorizer.fit_transform(documents)\n    km = KMeans(n_clusters=n_clusters, random_state=42)\n    km.fit(tfidf_matrix)\n    return km.labels_\n```\n\n#### Visualization\n- **TF-IDF Chart**:\n  ```python\n  def visualize_tfidf_figure(tfidf_scores):\n      fig, ax = plt.subplots()\n      words, scores = zip(*tfidf_scores) if tfidf_scores else ([], [])\n      ax.barh(words, scores)\n      ax.set_xlabel(\"TF-IDF Score\")\n      ax.set_title(\"Top TF-IDF Keywords\")\n      plt.tight_layout()\n      return fig\n  ```\n- **Word Cloud**:\n  ```python\n  def generate_wordcloud_figure(text):\n      wordcloud = WordCloud(width=800, height=400, background_color=\"white\").generate(text)\n      fig, ax = plt.subplots(figsize=(10, 5))\n      ax.imshow(wordcloud, interpolation=\"bilinear\")\n      ax.axis(\"off\")\n      ax.set_title(\"Word Cloud\")\n      plt.tight_layout()\n      return fig\n  ```\n\n---\n\n## Usage\n\n### Running the Application\n1. Start the application:\n   ```bash\n   python app.py\n   ```\n2. Open the Gradio interface at `http://127.0.0.1:7861`.\n\n### Example Input\n- **Text**: `\"Artificial intelligence revolutionizes industries.\"`\n- **Documents**:\n  ```\n  AI is transforming healthcare.\n  Robotics drives automation.\n  Machine learning enables new opportunities.\n  ```\n\n### Example Output\n- **Named Entities**: `[\"Artificial intelligence\", \"industries\"]`\n- **Sentiment Analysis**: `Positive (Polarity: 0.85)`\n- **Clusters**: `[0, 1, 2]`\n- **TF-IDF Keywords**: `[\"artificial\", \"intelligence\", \"revolutionizes\"]`\n- **Readability Scores**:\n  ```json\n  {\n      \"flesch_reading_ease\": 70.2,\n      \"gunning_fog_index\": 8.3,\n      \"smog_index\": 7.2\n  }\n  ```\n\n---\n\n## Gradio Panels\n\n### Inputs\n- **Text**: Multiline input for primary text analysis.\n- **Documents**: Optional multiline input for document clustering and comparison.\n\n### Outputs\n- **JSON Results**: \n  - Named entities, clean tokens, word frequencies, sentiment analysis, etc.\n- **Visualization Panels**:\n  - Word Cloud, Polarity Heatmap, Co-occurrence Network, and TF-IDF Chart.\n\n---\n\n## Customization\n\n- **Adjust Number of Topics**:\n  ```python\n  topic_modeling(documents, n_topics=5)\n  ```\n- **Modify Clusters**:\n  ```python\n  cluster_documents(documents, n_clusters=4)\n  ```\n\n---\n\n## Troubleshooting\n\n| **Issue**                 | **Solution**                                                      |\n|----------------------------|------------------------------------------------------------------|\n| Missing NLTK Data          | Run `nltk.download('punkt')` and `nltk.download('stopwords')`.  |\n| SpaCy Model Missing        | Run `python -m spacy download en_core_web_sm`.                  |\n| Backend Errors             | Uncomment `matplotlib.use('Agg')` for compatibility.           |\n\n---\n\n## Contribution\n\nWe welcome contributions! Fork the repository, make changes, and submit pull requests to enhance features or fix bugs.\n\n---\n\n## License\n\nThis project is licensed under the MIT License. See the `LICENSE` file for details.\n\n---\n\n## Screenshots\n\n### Gradio Interface\n\n![image](https://github.com/user-attachments/assets/ada65ce7-ad2f-49fc-93ce-5ecb90d392e1)\n\n\n### Visualizations\n\n-**Named Entities**\n![image](https://github.com/user-attachments/assets/b235a814-3841-4c43-99b9-b01d9cf7b993)\n\n-**Clean Tokens**\n![image](https://github.com/user-attachments/assets/98593af4-48e7-47b3-9d39-1da9981789ea)\n\n-**Word Frequencies**\n![image](https://github.com/user-attachments/assets/b5d08e53-173b-4122-98d3-893c0f1a13ac)\n\n-**Sentiment ANalysis**\n![image](https://github.com/user-attachments/assets/d2539eb3-5cef-41a5-b180-60fc4b59d097)\n\n-**Top TFIDF keywords**\n![image](https://github.com/user-attachments/assets/08f36633-750c-4a7c-9676-278afe86f6ca)\n\n-**Topics**\n![image](https://github.com/user-attachments/assets/6eb5f354-91c6-4f0a-b8e7-7fdf848d6314)\n\n-**Summary**\n![image](https://github.com/user-attachments/assets/6a837d46-9f46-4f25-9eab-fef462f4faff)\n\n-**RAKE Keywords**\n![image](https://github.com/user-attachments/assets/601c8c4e-296e-4ae5-91dd-27184c946fb4)\n\n-**Document Clusters**\n![image](https://github.com/user-attachments/assets/7f6fb996-e974-4e5b-a539-f356bfb1a65c)\n\n-**POS Tagging Counts**\n![image](https://github.com/user-attachments/assets/05e495a1-48f1-4cf2-a2aa-3e5b2d7462a8)\n\n-**Readability Scores**\n![image](https://github.com/user-attachments/assets/e13f5f61-b175-4529-85f1-81196b6f798d)\n\n\n- **Word Cloud**:\n  ![image](https://github.com/user-attachments/assets/b09e0a8e-906b-4e9f-b365-a47241e43432)\n\n- **Polarity Heatmap**:\n  ![image](https://github.com/user-attachments/assets/cda1c9de-c9d4-4136-a401-10301f18376d)\n\n- **TF-IDF Keywords**:\n![image](https://github.com/user-attachments/assets/d2e3942a-640f-427d-b429-a5e23ca6ecb2)\n\n- **Co-occurence Network**:\n![image](https://github.com/user-attachments/assets/8d49634f-f733-400a-8fac-4349214d1816)\n\n\n## References\n- [SpaCy Documentation](https://spacy.io/)\n- [NLTK Documentation](https://www.nltk.org/)\n- [TextBlob Documentation](https://textblob.readthedocs.io/)\n- [Gradio Documentation](https://gradio.app/)\n\nHappy Analyzing! 🚀\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparthapray%2Fnlp_pipeline_openai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fparthapray%2Fnlp_pipeline_openai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparthapray%2Fnlp_pipeline_openai/lists"}