{"id":18077372,"url":"https://github.com/parisneo/safe_store","last_synced_at":"2025-04-12T09:32:20.693Z","repository":{"id":198796172,"uuid":"701557656","full_name":"ParisNeo/safe_store","owner":"ParisNeo","description":"A data indexing library 100% open source with no need to use any closed source embeddings or opaque code.","archived":false,"fork":false,"pushed_at":"2024-07-21T19:50:10.000Z","size":1083,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-11T01:41:34.294Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ParisNeo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-10-06T23:00:47.000Z","updated_at":"2024-10-28T22:36:15.000Z","dependencies_parsed_at":"2024-01-18T09:46:41.297Z","dependency_job_id":"1b06a482-c423-4c8a-ba09-bc7e0da7db7c","html_url":"https://github.com/ParisNeo/safe_store","commit_stats":null,"previous_names":["parisneo/safe_store"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ParisNeo%2Fsafe_store","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ParisNeo%2Fsafe_store/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ParisNeo%2Fsafe_store/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ParisNeo%2Fsafe_store/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ParisNeo","download_url":"https://codeload.github.com/ParisNeo/safe_store/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248546173,"owners_count":21122270,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-31T11:24:24.835Z","updated_at":"2025-04-12T09:32:20.400Z","avatar_url":"https://github.com/ParisNeo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"assets/logo.png\" width=\"200px\"\u003e\n\n# safe_store\n![GitHub Repo](https://img.shields.io/badge/GitHub-Repo-brightgreen.svg)\n![PyPI Version](https://img.shields.io/pypi/v/safe-store.svg)\n![License](https://img.shields.io/github/license/ParisNeo/safe_store.svg)\n![Python Versions](https://img.shields.io/pypi/pyversions/safe-store.svg)\n\n`safe_store` is an open-source Python library that provides essential tools for text data management, vectorization, and document retrieval. It empowers users to work with text documents efficiently and effortlessly.\n\n- [safe\\_store](#safe_store)\n  - [Key Features:](#key-features)\n    - [1. Text Vectorizer](#1-text-vectorizer)\n    - [2. Generic Data Loader](#2-generic-data-loader)\n  - [What Can You Use `safe_store` For?](#what-can-you-use-safe_store-for)\n  - [Text Vectorizer](#text-vectorizer)\n    - [Features](#features)\n    - [Installation](#installation)\n    - [Getting Started](#getting-started)\n      - [Initializing the Text Vectorizer](#initializing-the-text-vectorizer)\n      - [Adding and Indexing Documents](#adding-and-indexing-documents)\n      - [Embedding a Query and Retrieving Similar Documents](#embedding-a-query-and-retrieving-similar-documents)\n  - [Generic Data Loader](#generic-data-loader)\n    - [Features](#features-1)\n    - [Usage](#usage)\n    - [Supported File Types](#supported-file-types)\n  - [Author](#author)\n  - [License](#license)\n\n\n## Key Features:\n\n### 1. Text Vectorizer\n\n- **Versatile Vectorization:** Choose between TF-IDF vectorization, model-based embeddings to convert text documents into numerical representations or use BM25 ranking for text retreival.\n- **Document Similarity:** Find documents similar to a given query text, making it ideal for document retrieval tasks.\n- **Interactive Visualization:** Visualize document embeddings in a scatter plot to gain insights into document relationships.\n- **No Authentication Required:** Use the library without the need for API keys or authentication, making it accessible for everyone.\n- **Commercially Usable:** `safe_store` is 100% open-source and free to use, even for commercial purposes, under the Apache 2.0 License.\n\n### 2. Generic Data Loader\n\n- **Multi-format Support:** Read various file formats, including PDF, DOCX, JSON, HTML, and more.\n- **Simplified Text Extraction:** Convert file content to plain text or data structures with ease.\n- **Efficient and Time-Saving:** Streamline data loading and processing tasks, reducing the need for manual extraction.\n\n## What Can You Use `safe_store` For?\n\n- **Text Document Analysis:** Analyze and understand the content of text documents quickly and efficiently.\n- **Document Retrieval:** Retrieve documents similar to a given query text, facilitating content recommendation and search tasks.\n- **Text Data Preprocessing:** Prepare text data for natural language processing (NLP) tasks, such as sentiment analysis and text classification.\n- **Data Loading:** Streamline the process of reading and extracting content from various file formats.\n\n`safe_store` is designed to be accessible, versatile, and free for all users. It's an ideal choice for developers, data scientists, and researchers who want a user-friendly and open-source solution for working with text data.\n\n---\n\nExplore the world of text data management and analysis with `safe_store` today!\n\n## Text Vectorizer\n\n### Features\n\n- Vectorize and index text documents.\n- Retrieve similar documents based on a query.\n- Supports both TF-IDF vectorization and model-based embeddings.\n- Interactive visualization of document embeddings.\n- No authentication or API keys required.\n\n### Installation\n\nTo install `safe_store`, you can use `pip`:\n\n```bash\npip install safe_store\n```\n\n### Getting Started\n\n#### Initializing the Text Vectorizer\n\n```python\nfrom safe_store import TextVectorizer, VectorizationMethod\nfrom pathlib import Path\n\n# Create an instance of TextVectorizer\nvectorizer = TextVectorizer(\n    vectorization_method=VectorizationMethod.TFIDF_VECTORIZER,\n    database_path=\"database.json\",\n    save_db=False\n)\n```\n\n#### Adding and Indexing Documents\n\n```python\n# Add documents for vectorization\ndocuments = [\"llm\", \"space\", \"submarines\", \"new york\"]\nfor doc in documents:\n    document_name = Path(__file__).parent / f\"{doc}.txt\"\n    with open(document_name, 'r', encoding='utf-8') as file:\n        text = file.read()\n    vectorizer.add_document(document_name, text, chunk_size=100, overlap_size=20, force_vectorize=False, add_as_a_bloc=False)\n\n# Index the documents (perform vectorization)\nvectorizer.index()\n```\n\n#### Embedding a Query and Retrieving Similar Documents\n\n```python\n# Embed a query and retrieve similar documents\nquery_text = \"what is space\"\nsimilar_texts, _, _ = vectorizer.recover_text(query_text, top_k=3)\n\n# Show the interactive document visualization\nvectorizer.show_document(show_interactive_form=True)\n\nprint(\"Similar Documents:\")\nfor i, text in enumerate(similar_texts):\n    print(f\"{i + 1}: {text}\")\n```\nThe `vectorizer.show_document(show_interactive_form=True)` should yield a plot like this where you can read the text by pointing on the dots. Each dot is a chunk of the text. We can clearly see that chunks that come from the same document tend to form a cluster.\n![image](https://github.com/ParisNeo/safe_store/assets/827993/5d9c59f8-656a-423d-ab8a-08ebf77595e4)\n\n---\n\n## Generic Data Loader\n\n### Features\n\n- Read various file formats including PDF, DOCX, JSON, HTML, and more.\n- Convert file content to text or data structures.\n\n### Usage\n\nTo read a file using `GenericDataLoader`, you can use the `read_file` method and provide the file path:\n\n```python\nfrom safe_store import GenericDataLoader\nfrom pathlib import Path\n\nfile_path = Path(\"example.pdf\")\nfile_content = GenericDataLoader.read_file(file_path)\n```\n\n### Supported File Types\n\n- PDF\n- DOCX\n- JSON\n- HTML\n- PPTX\n- TXT\n- RTF\n- MD\n- LOG\n- CPP\n- Java\n- JS\n- Python\n- Ruby\n- Shell Script\n- SQL\n- CSS\n- PHP\n- XML\n- YAML\n- INI\n- INF\n- MAP\n- BAT\n\nFeel free to replace `\"example.pdf\"` with the path to your specific file.\n\n---\n\n## Author\n- ParisNeo\n\n## License\nThis project is licensed under the Apache 2.0 License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparisneo%2Fsafe_store","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fparisneo%2Fsafe_store","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparisneo%2Fsafe_store/lists"}