{"id":50987367,"url":"https://github.com/activeloopai/vector_search_utility","last_synced_at":"2026-06-19T21:03:50.175Z","repository":{"id":342141656,"uuid":"1075279648","full_name":"activeloopai/vector_search_utility","owner":"activeloopai","description":null,"archived":false,"fork":false,"pushed_at":"2025-12-02T10:04:21.000Z","size":77,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-05T04:33:32.960Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/activeloopai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-13T09:31:51.000Z","updated_at":"2025-11-04T17:49:12.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/activeloopai/vector_search_utility","commit_stats":null,"previous_names":["activeloopai/vector_search_utility"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/activeloopai/vector_search_utility","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/activeloopai%2Fvector_search_utility","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/activeloopai%2Fvector_search_utility/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/activeloopai%2Fvector_search_utility/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/activeloopai%2Fvector_search_utility/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/activeloopai","download_url":"https://codeload.github.com/activeloopai/vector_search_utility/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/activeloopai%2Fvector_search_utility/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34547786,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-19T02:00:06.005Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-19T21:03:48.296Z","updated_at":"2026-06-19T21:03:50.168Z","avatar_url":"https://github.com/activeloopai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Vector Search Utility - Parquet to DeepLake Converter\n\nThis utility converts Parquet files to DeepLake format with embedding support, automatic schema parsing, and data validation.\n\n## Features\n\n- **Automatic Schema Detection**: Parses Spark-style schema files to configure column types\n- **Embedding Validation**: Automatically filters out rows with invalid/missing embeddings\n- **Efficient Processing**: Batch processing with memory optimization\n- **Flexible Input**: Support for both folder-based and file-list based ingestion\n- **Index Creation**: Automatic index creation for vector search and inverted text search\n\n## Installation\n\n```bash\npip install deeplake pyarrow numpy tqdm\n```\n\n## Usage\n\n### Option 1: Convert All Files from a Folder\n\nProcess all `.parquet` files in a directory:\n\n```bash\npython main.py \\\n    --schema schema.txt \\\n    --source /path/to/parquet/folder \\\n    --output s3://bucket/dataset \\\n    --org-id=\u003corg_id\u003e --ds-name=\u003cds_name\u003e\n```\n\n```bash\npython main.py \\\n    --schema schema.txt \\\n    --source s3://\u003cbucket\u003e/path \\\n    --output s3://bucket/dataset \\\n    --org-id=\u003corg_id\u003e --ds-name=\u003cds_name\u003e\n```\n\n### Option 2: Convert Specific Files\n\nProcess a specific list of parquet files (space-separated):\n\n```bash\npython main.py \\\n    --schema schema.txt \\\n    --files /path/file1.parquet /path/file2.parquet /path/file3.parquet \\\n    --output s3://bucket/dataset \\\n    --org-id=\u003corg_id\u003e --ds-name=\u003cds_name\u003e\n```\n\n**Important**: Files must be **space-separated**, not comma-separated.\n\n### Full Command with All Options\n\n```bash\npython main.py \\\n    --schema schema.txt \\\n    --source /path/to/parquet/folder \\\n    --output s3://bucket/dataset \\\n    --embedding title_emb \\\n    --embedding-size 768 \\\n    --inverted-column title1 \\\n    --unique-id-column video_id \\\n    --org-id=\u003corg_id\u003e --ds-name=\u003cds_name\u003e\n```\n\n## Command-Line Arguments\n\n| Argument             | Required     | Default     | Description                                      |\n| -------------------- | ------------ | ----------- | ------------------------------------------------ |\n| `--schema`           | Yes          | -           | Path to schema configuration file (Spark format) |\n| `--source`           | Conditional* | -           | Source folder containing parquet files           |\n| `--files`            | Conditional* | -           | List of parquet file paths (space-separated)     |\n| `--output`           | Yes          | -           | Output DeepLake dataset path (local or S3)       |\n| `--embedding`        | No           | `title_emb` | Name of the embedding column                     |\n| `--embedding-size`   | No           | `768`       | Dimension size of embedding vectors              |\n| `--inverted-column`  | No           | `title1`    | String column to index with inverted index       |\n| `--unique-id-column` | No           | `video_id`  | Column to use as unique identifier               |\n\n\\* Either `--source` or `--files` must be provided (not both)\n\n## Schema File Format\n\nThe schema file should follow Spark's `printSchema()` output format:\n\n```\nroot\n |-- ucid: string (nullable = true)\n |-- video_id: string (nullable = true)\n |-- id: long (nullable = true)\n |-- created_at: timestamp (nullable = true)\n |-- title1: string (nullable = true)\n |-- views: long (nullable = true)\n |-- made_for_kids: boolean (nullable = true)\n |-- cluster_index: integer (nullable = true)\n |-- v7_average: float (nullable = true)\n |-- performance_index: float (nullable = true)\n |-- title_emb: array (nullable = true)\n |    |-- element: float (containsNull = true)\n |-- power_keywords: array (nullable = true)\n |    |-- element: struct (containsNull = true)\n```\n\n### Supported Column Types\n\n- **String**: `string`\n- **Integer**: `integer` (32-bit)\n- **Long**: `long` (64-bit)\n- **Float**: `float` (32-bit)\n- **Double**: `double` (64-bit)\n- **Boolean**: `boolean`\n- **Timestamp**: `timestamp`\n- **Array**: `array` (simple arrays like embeddings)\n- **Struct/Dict**: `array` with nested `struct` (complex nested data)\n\n## Data Validation\n\nThe converter automatically:\n\n1. **Validates Embeddings**: Rows with `None`, empty, or invalid embeddings are filtered out\n2. **Maintains Alignment**: All columns remain aligned after filtering\n3. **Logs Filtering**: Prints number of rows filtered per batch\n4. **Skips Empty Batches**: If all rows in a batch have invalid embeddings, the batch is skipped\n\n## Processing Flow\n\n1. **Schema Parsing**: Reads and parses the schema file\n2. **Dataset Creation**: Creates DeepLake dataset with proper column types\n3. **Data Ingestion**:\n   - Processes embedding column first\n   - Identifies invalid rows\n   - Filters all columns to maintain alignment\n   - Appends valid data in batches\n4. **Index Creation**: Enables indexing mode and commits (separate step)\n\n## Examples\n\n### Example 1: Basic Usage with Folder\n\n```bash\npython main.py \\\n    --schema schema.txt \\\n    --source /data/parquet_files \\\n    --output ./local_dataset\n```\n\n### Example 2: S3 Output with Custom Settings\n\n```bash\npython main.py \\\n    --schema schema.txt \\\n    --source /data/parquet_files \\\n    --output s3://my-bucket/my-dataset \\\n    --embedding video_embedding \\\n    --embedding-size 1024 \\\n    --inverted-column description \\\n    --unique-id-column content_id\n```\n\n### Example 3: Specific Files Only\n\n```bash\npython main.py \\\n    --schema schema.txt \\\n    --files \\\n        /data/batch1/part-00001.parquet \\\n        /data/batch1/part-00002.parquet \\\n        /data/batch2/part-00003.parquet \\\n    --output s3://my-bucket/incremental-dataset\n```\n\n## Output\n\nThe converter will display:\n\n1. **Schema Summary**: Column counts by type\n2. **Processing Progress**: File-by-file progress with tqdm\n3. **Filtering Stats**: Number of rows filtered due to invalid embeddings\n4. **Batch Information**: Size of each batch being appended\n5. **Index Creation**: Status of index building\n\nExample output:\n```\nParsed Schema Configuration:\n  String columns: 15\n  Integer columns: 2\n  Long columns: 5\n  Float columns: 3\n  Double columns: 1\n  Boolean columns: 1\n  Timestamp columns: 2\n  Array columns (simple): 1\n  Dict columns (complex): 1\n  Embedding column: title_emb\n  Embedding size: 768\n  Total columns: 31\n\nProcessing 3 parquet files\nConverting files: 100%|████████████| 3/3 [01:23\u003c00:00, 27.8s/it]\n\nIndexing dataset...\nConversion completed successfully!\n```\n\n## Performance Optimization\n\n- **Batch Size**: Default 200,000 rows per batch (configurable in `SchemaConfig`)\n- **Memory Management**: Automatic garbage collection after each batch\n- **Contiguous Arrays**: Ensures numpy arrays are memory-contiguous for optimal performance\n- **Zero-Copy**: Uses PyArrow zero-copy conversion when possible\n\n## Programmatic Usage\n\nYou can also use the converter programmatically:\n\n```python\nfrom parquet_to_deeplake import ParquetToDeepLakeConverter\nfrom schema_parser import parse_schema_file\n\n# Parse schema\nconfig = parse_schema_file(\n    \"schema.txt\",\n    embedding_column=\"title_emb\",\n    embedding_size=768,\n    inverted_column=\"title1\",\n    unique_id_column=\"video_id\"\n)\n\n# Create converter\nconverter = ParquetToDeepLakeConverter(\"s3://bucket/dataset\", config)\n\n# Option 1: Convert folder\nconverter.convert_folder(\"/path/to/folder\")\n\n# Option 2: Convert specific files\nfile_list = [\n    \"/path/file1.parquet\",\n    \"/path/file2.parquet\"\n]\nconverter.convert_files(file_list)\n```\n\n## Troubleshooting\n\n### Issue: \"Either --source or --files must be provided\"\n**Solution**: You must specify either `--source` (for folder) or `--files` (for file list), but not both.\n\n### Issue: Only processing one file when using --files\n**Solution**: Ensure files are **space-separated**, not comma-separated:\n```bash\n# Wrong\n--files file1.parquet,file2.parquet\n\n# Correct\n--files file1.parquet file2.parquet\n```\n\n### Issue: \"Embedding column not found\"\n**Solution**: Ensure the `--embedding` argument matches the column name in your schema file.\n\n### Issue: All rows filtered out\n**Solution**: Check that your embedding column contains valid non-null vectors. Rows with None, empty, or invalid embeddings are automatically filtered.\n\n## Notes\n\n- The converter commits after **every batch** for incremental progress tracking\n- Indexing is enabled in a **separate commit** after all data is ingested\n- Invalid embeddings (None, empty, or wrong type) cause the entire row to be filtered\n- All numeric arrays are converted to contiguous memory layout for performance\n- The unique ID column is indexed with exact matching for fast lookups\n- The inverted column supports full-text search capabilities\n\n## Architecture\n\n```\nmain.py\n  ├─ ParquetToDeepLakeConverter\n  │   ├─ DeepLakeManager (dataset creation/schema)\n  │   └─ ParquetProcessor (data processing)\n  │       ├─ _convert_arrow_array (type conversion + filtering)\n  │       ├─ _transform_batch_data (batch transformation)\n  │       └─ process_file (file iteration)\n  └─ parse_schema_file (schema parsing)\n```\n\n## Data Flow Examples\n\n### 1. Initial Ingest (Create New Dataset)\n\nCreates a new DeepLake dataset from a folder of parquet files:\n\n```bash\npython3 main.py \\\n    --schema=schema.txt \\\n    --source=s3://path/to/short \\\n    --output=s3://path/to/scripts/ds \\\n    --org-id=scripts \\\n    --ds-name=ds \\\n    --num-workers=6\n```\n\n**When to use**: First-time data load into a new dataset.\n\n### 2. Append (Add New Data)\n\nAdds additional data to an existing dataset without modifying existing records:\n\n```bash\npython3 main.py \\\n    --schema=schema.txt \\\n    --output=s3://path/to/scripts/ds \\\n    --org-id=scripts \\\n    --ds-name=ds \\\n    --files \\\n        /path/to/new/part-00001.parquet \\\n        /path/to/new/part-00002.parquet \\\n        /path/to/new/part-00003.parquet\n```\n\n**When to use**: Adding new records to an existing dataset (no duplicates expected). Does not check for existing records.\n\n### 3. Upsert (Update or Insert)\n\nUpdates existing records or inserts new ones based on the unique ID column:\n\n#### Local Files\n\n```bash\npython3 main.py \\\n    --schema=schema.txt \\\n    --output=s3://path/to/scripts/ds \\\n    --org-id=scripts \\\n    --ds-name=ds \\\n    --files \\\n        /path/to/upsert_data/part-6fb66e36-5e67-42bc-89d4-7147c139a20d-1993-1-c000.snappy.parquet \\\n        /path/to/upsert_data/part-6fb66e36-5e67-42bc-89d4-7147c139a20d-2003-1-c000.snappy.parquet \\\n        /path/to/upsert_data/part-6fb66e36-5e67-42bc-89d4-7147c139a20d-2007-1-c000.snappy.parquet \\\n    --upsert\n```\n\n#### S3 Files\n\n```bash\npython3 main.py \\\n    --schema=schema.txt \\\n    --output=s3://path/to/scripts/ds \\\n    --org-id=scripts \\\n    --ds-name=ds \\\n    --files \\\n        s3://data/short/part-00000-6fb66e36-5e67-42bc-89d4-7147c139a20d-1375-1-c000.snappy.parquet \\\n        s3://data/short/part-00004-6fb66e36-5e67-42bc-89d4-7147c139a20d-1399-1-c000.snappy.parquet \\\n    --upsert\n```\n\n**When to use**: Updating existing records or handling incremental updates with potential duplicates.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Factiveloopai%2Fvector_search_utility","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Factiveloopai%2Fvector_search_utility","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Factiveloopai%2Fvector_search_utility/lists"}