https://github.com/someshdiwan/information-retrieval
Demonstrating techniques for text document processing, including vector space modeling, cosine similarity computation, and other information retrieval methods. Ideal for learning and implementing basic IR concepts.
https://github.com/someshdiwan/information-retrieval
grammar-parser information information-extraction information-retrieval
Last synced: 12 months ago
JSON representation
Demonstrating techniques for text document processing, including vector space modeling, cosine similarity computation, and other information retrieval methods. Ideal for learning and implementing basic IR concepts.
- Host: GitHub
- URL: https://github.com/someshdiwan/information-retrieval
- Owner: Someshdiwan
- Created: 2025-01-07T05:39:37.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-01-12T13:18:06.000Z (over 1 year ago)
- Last Synced: 2025-03-04T15:48:15.230Z (over 1 year ago)
- Topics: grammar-parser, information, information-extraction, information-retrieval
- Language: Jupyter Notebook
- Homepage:
- Size: 119 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Text Document Processing
A collection of scripts and examples demonstrating techniques for text document processing, including vector space modeling, cosine similarity computation, and other information retrieval (IR) methods.
This repository is ideal for learning and implementing basic IR concepts, text classification, web crawling, and document preprocessing.
---
## 🚀 Overview
This repository showcases several fundamental and advanced techniques in **text document processing** and **information retrieval (IR)**, including methods for text classification, vector space modeling, similarity computation, and web crawling.
### Key Techniques:
- **Text Preprocessing**: Text cleaning, stop word removal, stemming, and lemmatization.
- **Vector Space Model (VSM)**: Representing documents as vectors in a high-dimensional space for processing.
- **Cosine Similarity**: Computing the similarity between documents using the cosine similarity measure.
- **Naive Bayes Classifier**: Text classification using the Naive Bayes algorithm (GaussianNB).
- **Web Crawling**: Crawling websites to extract news stories with domain filtering.
---
## 🔧 Features
- **Text Classification**: Naive Bayes classifier for text classification and prediction tasks.
- **Document Preprocessing**: Techniques for cleaning and preparing text documents for analysis.
- **Cosine Similarity**: Implementation of cosine similarity to compare and measure the similarity between documents.
- **Web Crawling**: Scripts for crawling news websites and collecting relevant text content.
- **XML Parsing**: Basic example of parsing and modifying XML documents in Python.
---
## 🌐 Demo
You can try out the various techniques demonstrated in this repository by running the provided Python scripts or Jupyter notebooks. The projects include:
- **Text classification** using Naive Bayes (GaussianNB)
- **Cosine similarity computation** for document comparison
- **Web crawling** to extract news stories from websites
- **XML document processing** for parsing and modification
### Dependencies:
To run the examples, you will need the following libraries:
- Python 3.x
- scikit-learn (for Naive Bayes and vectorizer)
- pandas
- numpy
- requests
- BeautifulSoup (for web scraping)
- nltk (for text preprocessing)
- lxml (for XML parsing)
Install them using pip:
pip install
---
🛠️ Technologies Used
Python 3.x
scikit-learn (for machine learning and vector space modeling)
pandas
numpy
nltk (for natural language processing)
BeautifulSoup (for web scraping)
lxml (for XML parsing)
Jupyter Notebooks (for interactive demos)
## 📂 Project Structure
```plaintext
Text-Document-Processing/
├── notebooks/ # Jupyter notebooks for each technique
├── data/ # Datasets for testing and training models
├── README.md # Project documentation
```
Running the Code
Clone the repository:
git clone [https://github.com/Someshdiwan/Text-Document-Processing](https://github.com/Someshdiwan/Information-Retrieval)
---
```
🌟 Show Your Support
If you like this project, please consider giving it a ⭐ on GitHub!
🤝 Contributing
We welcome contributions to improve the repository! If you have any enhancements, bug fixes, or new project ideas, feel free to fork the repository, make changes, and submit a pull request.