https://github.com/ranfysvalle02/txtsumm-ai-mdb
scalable, reproducible, and manageable text summarization workflows. This repository provides tools and scripts for storing summarization runs, debugging processes, and maintaining quality control using MongoDB's flexible schema and powerful querying capabilities.
https://github.com/ranfysvalle02/txtsumm-ai-mdb
Last synced: 3 months ago
JSON representation
scalable, reproducible, and manageable text summarization workflows. This repository provides tools and scripts for storing summarization runs, debugging processes, and maintaining quality control using MongoDB's flexible schema and powerful querying capabilities.
- Host: GitHub
- URL: https://github.com/ranfysvalle02/txtsumm-ai-mdb
- Owner: ranfysvalle02
- Created: 2024-12-08T05:24:19.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2024-12-08T06:56:38.000Z (10 months ago)
- Last Synced: 2025-02-05T06:12:03.905Z (8 months ago)
- Language: Python
- Homepage:
- Size: 146 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# txtsumm-ai-mdb
_image credit to: https://ai.plainenglish.io/text-summarization-with-llm-f69b4f2ccb3e_
---
# Reproducible and Manageable Text Summarization with MongoDB
In today's digital age, we're drowning in information. From lengthy articles and research papers to detailed reports and documentation, the sheer volume of text we encounter daily can be overwhelming. Summarizing this information efficiently is not just a convenience—it's a necessity.
**TextSummAI** emerges as a powerful tool to tackle this challenge, offering advanced text summarization capabilities. But as with any complex system, managing and debugging summarization runs can become cumbersome, especially when dealing with large datasets and requiring reproducibility for quality control.
This is where **MongoDB**, a flexible and scalable NoSQL database, comes into play. Integrating MongoDB with TextSummAI not only streamlines the summarization process but also enhances reproducibility, debugging, and quality control.
In this blog post, we'll dive into how MongoDB complements TextSummAI, making it easier to manage summarization runs. We'll explore practical examples and highlight why MongoDB is an excellent fit for this application.
---
## Understanding TextSummAI
**[TextSummAI](https://github.com/ranfysvalle02/txtsumm-ai)** is an open-source Python library designed to simplify and enhance text summarization tasks. It leverages advanced natural language processing (NLP) techniques and models to generate concise summaries from large bodies of text.
**Key Features of TextSummAI:**
- **Multiple Summarization Strategies:**
- **Map-Reduce:** Splits the text into chunks, summarizes each chunk, and then combines the summaries.
- **Critical Vectors:** Selects the most semantically significant and diverse chunks before summarization.
- **Direct Summarization:** Processes the entire text without chunking for shorter texts.
- **Parallel Processing:** Utilizes Python's multiprocessing capabilities to speed up summarization, especially for large texts.
- **Modularity and Extensibility:** Allows users to customize and extend functionalities according to their needs.
While TextSummAI is powerful, managing the data generated during summarization runs—like configurations, logs, intermediate summaries, and errors—can be challenging, especially when aiming for reproducibility and effective debugging.
---
_image credit to: https://www.inc.com/soren-kaplan/how-to-move-forward-when-youre-feeling-overwhelmed-by-data.html_
## The Challenge of Managing Summarization Runs
When working with complex NLP tasks and models, several challenges arise:
- **Reproducibility:** Ensuring that summarization runs can be replicated exactly, which is vital for debugging and validating results.
- **Debugging:** Tracking down issues requires detailed logs and access to intermediate data generated during runs.
- **Quality Control:** Comparing summaries across runs, analyzing performance, and ensuring consistent quality necessitate storing and querying run data effectively.
Traditional methods of storing this data—like plain files or rigid relational databases—may not offer the flexibility or scalability needed. This is where MongoDB comes into the picture.
---
## Why MongoDB is the Perfect Fit
### 1. Flexible Schema
**Challenge:**
Each summarization run generates diverse data, such as:
- Original text
- Chunks and their summaries
- Final summary
- Debug information (errors, logs)
- Prompt metadata
This data doesn't conform to a fixed structure, making traditional relational databases less ideal.
**MongoDB Solution:**
MongoDB's flexible schema allows you to store varied data without the constraints of a predefined table structure. You can evolve your data model as TextSummAI grows and changes.
*Example:*
You can store each run as a document containing all relevant information, like so:
```json
{
"strategy": "critical_vectors",
"parallelization": true,
"original_text": "...",
"chunks": ["Chunk 1 text...", "Chunk 2 text...", "..."],
"chunk_summaries": ["Summary 1...", "Summary 2...", "..."],
"final_summary": "Final summary text...",
"debug_info": {
"map_phase": [...],
"reduce_phase": {...},
"errors": ["Error message if any"]
},
"logs": [
["INFO", "Starting summarization..."],
["DEBUG", "Processed chunk 1..."],
["ERROR", "An error occurred..."]
],
"prompt_metadata": {
"map_prompt": "Map prompt details...",
"reduce_prompt": "Reduce prompt details..."
}
}
```
### 2. Scalability
**Challenge:**
As you process more texts and conduct more runs, the data volume increases. You need a database that scales seamlessly without sacrificing performance.
**MongoDB Solution:**
MongoDB is designed to handle increasing data loads gracefully. Its ability to scale horizontally by adding more servers ensures consistent performance, even as your dataset expands.
*Key Advantage:*
You won't have to worry about performance bottlenecks as your summarization tasks grow.
### 3. Powerful Querying Capabilities
**Challenge:**
For effective debugging and quality control, you need to perform complex queries, such as:
- Finding runs that encountered errors
- Retrieving runs using a specific summarization strategy
- Selecting runs within a certain date range
**MongoDB Solution:**
MongoDB's query language allows you to perform these complex queries with ease.
*Example:*
To find all runs where errors occurred:
```python
error_runs = collection.find({"debug_info.errors": {"$ne": []}})
```
To retrieve runs using the "critical_vectors" strategy:
```python
cv_runs = collection.find({"strategy": "critical_vectors"})
```
### 4. Native JSON Support
**Challenge:**
You need a seamless way to store and retrieve data structures like Python dictionaries, which are used extensively in TextSummAI.
**MongoDB Solution:**
Since MongoDB stores data in BSON (binary JSON), it's naturally suited for applications that handle JSON data. This means Python dictionaries can be stored and retrieved without additional serialization or deserialization overhead.
*Key Advantage:*
Simplifies data handling, reduces code complexity, and minimizes potential errors during data conversion.
### 5. Aggregation Framework
**Challenge:**
To analyze and gain insights from your summarization runs, you need to perform advanced data processing and analytics tasks.
**MongoDB Solution:**
MongoDB's aggregation framework lets you perform complex data aggregation operations directly within the database.
*Examples:*
- **Aggregate statistics on summary lengths:**
```python
pipeline = [
{"$unwind": "$chunk_summaries"},
{"$group": {"_id": None, "avgLength": {"$avg": {"$strLenCP": "$chunk_summaries"}}}}
]
result = collection.aggregate(pipeline)
```
- **Analyze error patterns across runs:**
```python
pipeline = [
{"$match": {"debug_info.errors": {"$ne": []}}},
{"$group": {"_id": "$strategy", "errorCount": {"$sum": 1}}}
]
result = collection.aggregate(pipeline)
```
---
## Bonus MongoDB Benefit: The Challenge of Implementing Advanced Encryption in Databases
When handling sensitive information in text summarization projects—such as confidential financial data, personal health records, or proprietary business information—securing this data is of utmost importance. MongoDB's **Client-Side Field-Level Encryption** and **Queryable Encryption** provide robust, built-in solutions for encrypting sensitive data while maintaining functionality. But replicating these advanced encryption features in other databases presents significant challenges.
### Why Securing Sensitive Data Is Difficult in SQL Databases
SQL databases are powerful for structured data storage and retrieval, but they lack native support for client-side field-level encryption and queryable encryption as provided by MongoDB. Attempting to implement similar functionality in SQL databases can be complex and fraught with potential pitfalls.
#### 1. No Native Client-Side Field-Level Encryption
Most SQL databases do not offer built-in client-side field-level encryption that integrates seamlessly with the database drivers.
**Challenges:**
- **Custom Implementation Required:** Developers must manually implement encryption and decryption logic for sensitive fields within the application code.
- **Increased Complexity and Risk:** Custom encryption is error-prone and may introduce security vulnerabilities if not implemented correctly.
- **Key Management Overhead:** Securely storing and managing encryption keys adds another layer of complexity.
**Example Scenario:**
You're summarizing **patient medical records** containing personal health information (PHI):
- **In SQL:** You need to manually encrypt PHI fields before storing them and decrypt them upon retrieval.
- **Complications:** Every interaction with the encrypted data requires custom logic, increasing the chance of mistakes and security flaws.
#### 2. Difficulty Performing Queries on Encrypted Data
SQL databases generally can't perform meaningful queries on encrypted data.
**Challenges:**
- **Lack of Queryable Encryption:** Encrypted fields can't be used in `WHERE` clauses or `JOIN` operations without first decrypting them.
- **Inefficient Workarounds:** Solutions like decrypting data on the fly or fetching all data and filtering in the application are inefficient and insecure.
- **No Support for Encrypted Indexes:** Without encrypted indexes, query performance on encrypted data is severely degraded.
**Example Scenario:**
You're summarizing **legal documents** and need to retrieve summaries based on specific case numbers, which are sensitive.
- **In SQL:** Since case numbers are encrypted, you can't use them in queries like `SELECT * FROM summaries WHERE case_number = 'ABC123'`.
- **Inefficient Alternatives:** Decrypting all case numbers in memory to find matches is impractical and insecure.
#### 3. Complex Key Management
Managing encryption keys securely is crucial but difficult in SQL databases.
**Challenges:**
- **Key Storage:** There's no built-in, secure key management system in SQL databases.
- **Key Rotation:** Updating encryption keys (key rotation) requires re-encrypting data, complicating maintenance.
- **Access Control:** Ensuring only authorized applications or users can access encryption keys adds complexity.
**Example Scenario:**
You're summarizing **financial reports** that include sensitive numerical data.
- **In SQL:** You need to securely store encryption keys used to encrypt fields like `financial_figures`.
- **Risk:** Improper key management could lead to key leakage, compromising all encrypted data.
### How MongoDB Simplifies Advanced Encryption
MongoDB's encryption features are designed to address these challenges effectively.
#### Client-Side Field-Level Encryption
- **Automatic Encryption/Decryption:** The MongoDB driver handles encryption and decryption transparently.
- **Secure Key Management:** Integration with Key Management Services (KMS) like AWS KMS, Azure Key Vault, or local master keys.
- **Field-Level Control:** Encrypt specific sensitive fields while leaving others in plaintext for querying.
**Benefits:**
- **Reduced Development Effort:** No need to write custom encryption logic.
- **Consistent Security:** Encryption is applied uniformly, reducing the risk of human error.
- **Regulatory Compliance:** Easier to meet standards like GDPR or HIPAA.
**Example Scenario Revisited:**
Summarizing **patient medical records** in MongoDB:
- **Encryption Configured:** Specify fields like `patient_name`, `medical_history`, and `diagnosis` to be encrypted.
- **Seamless Operations:** The application code remains clean, with the driver handling encryption tasks.
#### Queryable Encryption
- **Encrypted Queries:** Perform equality searches on encrypted fields without needing to decrypt them first.
- **Deterministic Encryption:** Enables querying by ensuring the same plaintext value encrypts to the same ciphertext.
- **Secure Indexing:** Create indexes on encrypted fields to maintain query performance.
**Benefits:**
- **Functional Queries:** Retrieve data based on encrypted fields efficiently.
- **Data Security:** Sensitive data remains encrypted during queries, preventing exposure.
- **Performance Maintenance:** Optimized queries ensure the application remains responsive.
**Example Scenario Revisited:**
Retrieving summaries of **legal documents** based on encrypted `case_number`:
- **Queryable Encryption Applied:** Encrypt the `case_number` field deterministically.
- **Efficient Queries:** Use `find` queries to locate documents with a specific encrypted `case_number` without decrypting all data.
- **Secure and Efficient:** Maintain both security and performance.
#### Simplified Key Management
- **Integrated Key Vault:** Store and manage encryption keys securely within MongoDB or through integrated KMS.
- **Automated Key Rotation:** Facilitate regular key rotation without extensive manual intervention.
- **Access Control:** Define which users or applications have access to encryption keys.
**Benefits:**
- **Enhanced Security:** Proper key management reduces the risk of unauthorized access.
- **Operational Efficiency:** Less administrative overhead compared to manual key management in SQL databases.
- **Compliance Support:** Meet regulatory requirements for encryption key management.
### The Difficulty of Replicating These Features in SQL
Implementing similar encryption functionality in SQL databases involves significant hurdles:
- **Custom Middleware Development:** You might need to develop middleware to handle encryption and decryption outside of the database.
- **Performance Trade-offs:** Custom solutions are likely to impact performance negatively.
- **Limited Query Capabilities:** Even with custom encryption, achieving queryable encryption is highly complex and may not be fully feasible.
- **Increased Risk:** Custom implementations increase the risk of security vulnerabilities due to human error.
### Real-World Implications
Attempting to handle advanced encryption in SQL databases can lead to:
- **Delayed Development:** Time spent on building encryption features delays other critical development tasks.
- **Maintenance Burden:** Custom encryption code requires ongoing maintenance and expert knowledge.
- **Security Risks:** Non-standard implementations are more susceptible to flaws that could compromise data security.
- **Reduced Competitiveness:** Inability to efficiently secure data might make your application less attractive in sensitive industries.
Implementing client-side field-level encryption and queryable encryption in SQL databases is a daunting task that can divert valuable resources and introduce security risks. MongoDB simplifies this process by providing robust, built-in encryption features that enable you to:
- **Easily Encrypt Sensitive Data:** With minimal changes to your application code.
- **Perform Secure Queries:** Efficiently query encrypted fields without exposing sensitive information.
- **Manage Encryption Keys Securely:** Utilize integrated key management solutions to protect your encryption keys.
- **Focus on Core Development:** Spend less time on encryption complexities and more on delivering value to your users.
By choosing MongoDB for sensitive text summarization tasks, you gain access to advanced encryption capabilities that are challenging to replicate in SQL databases. MongoDB's solutions not only enhance data security but also streamline development and maintain high performance, making it the superior choice for applications requiring stringent data protection.
---
## Implementing MongoDB with TextSummAI
Let's explore how to integrate MongoDB into TextSummAI to enhance reproducibility, debugging, and quality control.
### Storing Summarization Runs
**Step 1: Initialize MongoDB Connection**
```python
from pymongo import MongoClient
mongodb_uri = "your_mongodb_uri"
client = MongoClient(mongodb_uri)
db = client["text_summ_ai"]
collection = db["summarization_runs"]
```
**Step 2: Modify the Run Class**
Ensure that your `Run` class includes a method to serialize the run data:
```python
class Run:
# ... existing code ...
def to_dict(self):
return {
"strategy": self.strategy,
"parallelization": self.parallelization,
"original_text": self.original_text,
"chunks": self.chunks,
"chunk_summaries": self.chunk_summaries,
"final_summary": self.final_summary,
"debug_info": self.debug_info,
"logs": self.logs,
"prompt_metadata": self.prompt_metadata
}
```
**Step 3: Save Runs to MongoDB**
After each summarization run, store the data:
```python
run = Run(strategy='critical_vectors', parallelization=True)
# ... perform summarization and populate run ...
try:
collection.insert_one(run.to_dict())
print("Run data saved to MongoDB.")
except Exception as e:
print(f"Failed to save run data to MongoDB: {e}")
```
### Example: Debugging and Quality Control
Suppose you notice that some summaries are shorter than expected. Here's how MongoDB can help:
**Step 1: Retrieve Runs with Short Summaries**
```python
short_summaries = collection.find({"final_summary": {"$exists": True}})
for run in short_summaries:
summary_length = len(run["final_summary"])
if summary_length < 100: # Arbitrary length threshold
print(f"Run ID: {run['_id']} has a short summary of length {summary_length}.")
```
**Step 2: Inspect the Specific Run**
```python
run_id = "the_specific_run_id"
run = collection.find_one({"_id": run_id})
# Check the chunks and their summaries
for idx, chunk_summary in enumerate(run["chunk_summaries"]):
print(f"Chunk {idx+1} Summary: {chunk_summary}\n")
# Review debug information and logs
if run["debug_info"]["errors"]:
print("Errors encountered during the run:")
for error in run["debug_info"]["errors"]:
print(error)
print("Logs:")
for log_entry in run["logs"]:
print(f"[{log_entry[0]}] {log_entry[1]}")
```
**Step 3: Analyze Error Patterns**
Use MongoDB's aggregation framework to identify common errors:
```python
pipeline = [
{"$unwind": "$debug_info.errors"},
{"$group": {"_id": "$debug_info.errors", "count": {"$sum": 1}}},
{"$sort": {"count": -1}}
]
error_analysis = collection.aggregate(pipeline)
print("Common Errors:")
for error in error_analysis:
print(f"Error: {error['_id']} occurred {error['count']} times.")
```
---
### The Challenge of Evaluating Summaries That "Look Good"
When working with large texts—like the entire novel *Dracula* by Bram Stoker—summarizing becomes a complex task. At first glance, a generated summary might "look good" because it succinctly condenses the text. However, without detailed tracking of the summarization process, it's difficult to determine:
- **Coverage**: Did the summary include all the critical plot points?
- **Accuracy**: Are there any misinterpretations or factual errors?
- **Balance**: Does the summary overemphasize certain parts while neglecting others?
To evaluate and debug the quality of such a summary effectively, you need access to the individual components of the summarization process:
- **Original Text**: The source material for reference.
- **Chunks**: How the text was divided for processing.
- **Chunk Summaries**: Summaries of each individual chunk.
- **Intermediary Data**: Metadata, configurations, and logs.
Without this granular data, pinpointing issues becomes guesswork. It's like trying to find a typo in a book without knowing which chapter it's in.
#### Case Study: Summarizing *Dracula*
Imagine using TextSummAI to summarize *Dracula*, a novel with over 160,000 words. The summarization process involves:
1. **Splitting the Text**: The novel is divided into manageable chunks.
2. **Summarizing Chunks**: Each chunk is individually summarized.
3. **Combining Summaries**: The chunk summaries are merged into a final summary.
Now, suppose the final summary is only a few paragraphs long. At a glance, it seems acceptable. But without examining the individual chunk summaries and how they contribute to the final summary, you might miss that critical plot points were omitted or misrepresented.
#### The Need for Detailed Tracking
To ensure the summary is truly representative of the original text, you need to:
- **Review Chunk Summaries**: Verify that each chunk's essential information was captured.
- **Check for Missing Content**: Identify if any important sections were accidentally skipped.
- **Analyze Errors**: Look for any errors that occurred during processing.
Maintaining this level of detail requires a robust data storage solution that can handle complex, nested data structures.
---
### Comparing MongoDB and SQL for Tracking Summarization Runs
When it comes to storing the detailed data generated during summarization runs, the choice of database can significantly impact your ability to debug and manage the process. Let's compare using a traditional SQL database with MongoDB.
#### SQL Database: Challenges with Rigid Schemas
**Schema Design Issues**
In a relational SQL database, you would need to define a fixed schema upfront. For tracking summarization runs, you might end up with multiple tables:
- `runs` table to store run metadata.
- `chunks` table to store each chunk, linked to `runs`.
- `chunk_summaries` table to store summaries of each chunk, linked to `chunks`.
- `logs` table to store logs, linked to `runs`.
**Problems:**
- **Complex Joins**: Retrieving the full data for a run requires multiple joins across tables.
- **Rigid Schema**: Adding new fields (e.g., storing additional metadata) requires altering the database schema.
- **Inefficient Debugging**: Navigating through normalized tables hampers quick analysis.
- **Scalability Issues**: Managing large blobs of text (like full chunks or logs) can be inefficient due to size limitations and performance overhead.
**Example SQL Schema**
```sql
-- Runs table
CREATE TABLE runs (
run_id INT PRIMARY KEY,
strategy VARCHAR(50),
parallelization BOOLEAN,
final_summary TEXT,
-- Other metadata fields
);
-- Chunks table
CREATE TABLE chunks (
chunk_id INT PRIMARY KEY,
run_id INT,
chunk_text TEXT,
FOREIGN KEY (run_id) REFERENCES runs(run_id)
);
-- Chunk Summaries table
CREATE TABLE chunk_summaries (
summary_id INT PRIMARY KEY,
chunk_id INT,
summary_text TEXT,
FOREIGN KEY (chunk_id) REFERENCES chunks(chunk_id)
);
-- Logs table
CREATE TABLE logs (
log_id INT PRIMARY KEY,
run_id INT,
level VARCHAR(10),
message TEXT,
timestamp DATETIME,
FOREIGN KEY (run_id) REFERENCES runs(run_id)
);
```
**Drawbacks:**
- **Performance Overhead**: Multiple joins for common queries.
- **Maintenance Burden**: Altering schemas and managing relationships adds complexity.
- **Limited Flexibility**: Difficulty in storing nested or varying data structures.
#### MongoDB: Embracing Flexibility and Nesting
**Flexible Document Structure**
MongoDB stores data in BSON documents, which can have nested structures and do not require a predefined schema. Each summarization run can be stored as a single document containing all related data.
**Benefits:**
- **Single Document Reads**: All data for a run is stored together, making retrieval efficient.
- **Flexible Schema**: Easily add or modify fields without schema migrations.
- **Nested Data**: Support for storing complex, hierarchical data structures.
- **Ease of Debugging**: Access nested components directly within the document.
**Example MongoDB Document Structure**
```json
{
"run_id": ObjectId("..."),
"strategy": "critical_vectors",
"parallelization": true,
"original_text": "...",
"chunks": [
{
"chunk_id": 1,
"chunk_text": "First part of the text...",
"chunk_summary": "Summary of first part..."
},
{
"chunk_id": 2,
"chunk_text": "Second part of the text...",
"chunk_summary": "Summary of second part..."
}
// More chunks...
],
"final_summary": "Final summary text...",
"debug_info": {
"errors": [],
"logs": [
{
"level": "INFO",
"message": "Starting summarization...",
"timestamp": "2023-10-01T12:34:56Z"
},
{
"level": "DEBUG",
"message": "Processed chunk 1",
"timestamp": "2023-10-01T12:35:00Z"
}
// More logs...
]
},
"prompt_metadata": {
"map_prompt": "Details of map prompt...",
"reduce_prompt": "Details of reduce prompt..."
}
}
```
**Advantages:**
- **Efficient Data Retrieval**: Fetch all relevant data for a run with a single query.
- **Simplified Debugging**: Access chunks, summaries, and logs directly within the document.
- **Enhanced Flexibility**: Easily store additional data like error traces or configuration settings without altering schemas.
- **Better Performance with Large Texts**: Optimized for handling large documents, making it suitable for texts like *Dracula*.
---
When summarizing complex and lengthy texts like *Dracula* or other large corpus of text, having detailed tracking of the summarization process is essential for ensuring quality and facilitating debugging. MongoDB's flexible schema, ability to handle nested data, and scalability make it superior to traditional SQL databases for this purpose.
By storing all related data for a summarization run in a single, coherent document, MongoDB allows developers to:
- **Quickly Access Relevant Information**: All data is in one place, eliminating the need for complex queries.
- **Easily Trace Issues**: Detailed logs and chunk-level data make it straightforward to identify where problems occurred.
- **Adapt to Changing Requirements**: The flexible schema of MongoDB accommodates new data fields effortlessly.
- **Scale with Your Needs**: As you process more and larger texts, MongoDB scales horizontally to meet demand.
In contrast, using a SQL database introduces unnecessary complexity and rigidity, hindering efficient debugging and quality control. The challenges of fixed schemas, complex joins, and difficulty handling large text blobs make SQL a less effective choice for managing the intricacies of text summarization processes.
---
## Conclusion
Integrating MongoDB with TextSummAI brings significant benefits:
- **Enhanced Reproducibility:** By storing all relevant data for each run, you can replicate results, compare runs, and ensure consistency.
- **Simplified Debugging:** Detailed logs and error tracking make it easier to identify and resolve issues.
- **Improved Quality Control:** Powerful querying and aggregation capabilities enable you to analyze performance, detect patterns, and make informed improvements.
- **Scalability and Flexibility:** MongoDB's design ensures that as your data grows, you can scale accordingly without reworking your data structures.
By leveraging MongoDB's strengths, you make managing TextSummAI runs more efficient and effective, ultimately leading to better summarization outcomes.
---
## Appendix: Getting Started with TextSummAI and MongoDB
**Step 1: Install MongoDB**
If you haven't already, install MongoDB:
- **For local development:** [Download MongoDB Community Server](https://www.mongodb.com/try/download/community)
- **For cloud deployment:** [Sign up for MongoDB Atlas](https://www.mongodb.com/cloud/atlas)
**Step 2: Install Required Python Packages**
```bash
pip install pymongo
```
**Step 3: Clone the TextSummAI Repository**
```bash
git clone https://github.com/ranfysvalle02/txtsumm-ai-mdb.git
cd txtsumm-ai-mdb
```
**Step 4: Configure MongoDB in Your Script**
Update your Python script to include MongoDB configuration:
```python
import os
from pymongo import MongoClient
mongodb_uri = os.getenv("MONGODB_URI") # Set your MongoDB URI in environment variables
client = MongoClient(mongodb_uri)
db = client["text_summ_ai"]
collection = db["summarization_runs"]
```
**Step 5: Run TextSummAI with MongoDB Integration**
Make sure your summarization runs are saved to MongoDB as demonstrated earlier.
**Step 6: Verify Data Storage**
Use MongoDB Compass or the MongoDB Shell to inspect your database and ensure that runs are being stored correctly.
---
By integrating MongoDB with TextSummAI, you not only enhance the functionality of your summarization tasks but also make your workflows more robust, manageable, and scalable. Whether you're processing a handful of documents or thousands of them, this setup ensures that you're equipped to handle the challenges effectively.
**Happy summarizing with TextSummAI and MongoDB!**