{"id":31079565,"url":"https://github.com/kevinsames/spark-fuse","last_synced_at":"2026-05-08T06:13:35.539Z","repository":{"id":313188861,"uuid":"1050379970","full_name":"kevinsames/spark-fuse","owner":"kevinsames","description":"spark-fuse is an open-source toolkit for PySpark — providing utilities, connectors, and tools to fuse your data workflows together.","archived":false,"fork":false,"pushed_at":"2025-09-04T12:52:19.000Z","size":51,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-04T13:12:59.302Z","etag":null,"topics":["data","databricks","fabric","pyspark","python","spark"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/spark-fuse/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kevinsames.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":"GOVERNANCE.md","roadmap":"roadmap.md","authors":"authors.md","dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":"MAINTAINERS","copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-04T10:54:40.000Z","updated_at":"2025-09-04T13:06:34.000Z","dependencies_parsed_at":"2025-09-04T13:13:07.591Z","dependency_job_id":"2ea440d1-40b3-4e46-9339-d9ceccdbf3f5","html_url":"https://github.com/kevinsames/spark-fuse","commit_stats":null,"previous_names":["kevinsames/spark-fuse"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/kevinsames/spark-fuse","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinsames%2Fspark-fuse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinsames%2Fspark-fuse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinsames%2Fspark-fuse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinsames%2Fspark-fuse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kevinsames","download_url":"https://codeload.github.com/kevinsames/spark-fuse/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinsames%2Fspark-fuse/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275402710,"owners_count":25458476,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-16T02:00:10.229Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","databricks","fabric","pyspark","python","spark"],"created_at":"2025-09-16T10:25:26.075Z","updated_at":"2026-05-08T06:13:35.532Z","avatar_url":"https://github.com/kevinsames.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"spark-fuse\n================\n\n![CI](https://github.com/kevinsames/spark-fuse/actions/workflows/ci.yml/badge.svg)\n![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)\n\nspark-fuse is an open-source toolkit for PySpark — providing utilities, data sources, and tools to fuse your data workflows across JSON-centric REST APIs and SPARQL endpoints.\n\nFeatures\n- Data sources for REST APIs (JSON payloads with pagination/retry support), SPARQL services, and Qdrant collections (read/write).\n- SparkSession helpers with sensible defaults and environment detection (databricks/fabric/local heuristics retained for legacy jobs).\n- DataFrame utilities for previews, schema checks, and ready-made date/time dimensions (daily calendar attributes and clock buckets).\n- LLM-powered semantic column normalization and LangChain-backed embedding generation (with optional text splitters) that batch work to limit API calls.\n- Similarity partitioning toolkit with modular embedding preparation, clustering, and representative selection utilities.\n- Change-tracking helpers to write current-only or history-preserving datasets with concise options.\n- Typer-powered CLI: list data sources and preview datasets via the REST/SPARQL helpers.\n\nInstallation\n- Create a virtual environment (recommended)\n  - macOS/Linux:\n    - `python3 -m venv .venv`\n    - `source .venv/bin/activate`\n    - `python -m pip install --upgrade pip`\n  - Windows (PowerShell):\n    - `python -m venv .venv`\n    - `.\\\\.venv\\\\Scripts\\\\Activate.ps1`\n    - `python -m pip install --upgrade pip`\n- From source (dev): `pip install -e \".[dev]\"`\n- From PyPI: `pip install \"spark-fuse\u003e=1.0.2\"`\n\nQuickstart\n1) Create a SparkSession with helpful defaults\n```python\nfrom spark_fuse.spark import create_session\nspark = create_session(app_name=\"spark-fuse-quickstart\")\n```\n\n2) Load paginated REST API responses\n```python\nimport json\nfrom spark_fuse.io import (\n    REST_API_CONFIG_OPTION,\n    REST_API_FORMAT,\n    build_rest_api_config,\n    register_rest_data_source,\n)\n\nregister_rest_data_source(spark)\nconfig = build_rest_api_config(\n    spark,\n    \"https://pokeapi.co/api/v2/pokemon\",\n    source_config={\n        \"request_type\": \"GET\",  # switch to \"POST\" for endpoints that require a body\n        \"records_field\": \"results\",\n        \"pagination\": {\"mode\": \"response\", \"field\": \"next\", \"max_pages\": 2},\n        \"params\": {\"limit\": 20},\n    },\n)\npokemon = (\n    spark.read.format(REST_API_FORMAT)\n    .option(REST_API_CONFIG_OPTION, json.dumps(config))\n    .load()\n)\npokemon.select(\"name\").show(5)\n```\nNeed to hit a POST endpoint? Set `\"request_type\": \"POST\"` and attach your payload with\n`\"request_body\": {...}` (defaults to JSON encoding—add `\"request_body_type\": \"data\"` for form bodies).\nFlip on `\"include_response_payload\": True` to add a `response_payload` column with the raw server JSON.\n\n3) Query a SPARQL endpoint\n```python\nsparql_query = \"\"\"\nPREFIX wd: \u003chttp://www.wikidata.org/entity/\u003e\nPREFIX wdt: \u003chttp://www.wikidata.org/prop/direct/\u003e\n\nSELECT ?pokemon ?pokemonLabel ?pokedexNumber WHERE {\n  ?pokemon wdt:P31 wd:Q3966183 .\n  ?pokemon wdt:P1685 ?pokedexNumber .\n}\nLIMIT 5\n\"\"\"\n\nfrom spark_fuse.io import (\n    SPARQL_CONFIG_OPTION,\n    SPARQL_DATA_SOURCE_NAME,\n    build_sparql_config,\n    register_sparql_data_source,\n)\n\nregister_sparql_data_source(spark)\nsparql_options = build_sparql_config(\n    spark,\n    \"https://query.wikidata.org/sparql\",\n    source_config={\n        \"query\": sparql_query,\n        \"request_type\": \"POST\",\n    \"headers\": {\"User-Agent\": \"spark-fuse-demo/1.0 (contact@example.com)\"},\n    },\n)\nsparql_df = (\n    spark.read.format(SPARQL_DATA_SOURCE_NAME)\n    .option(SPARQL_CONFIG_OPTION, json.dumps(sparql_options))\n    .load()\n)\nif sparql_df.rdd.isEmpty():\n    print(\"Endpoint unavailable — adjust the query or check your network.\")\nelse:\n    sparql_df.show(5, truncate=False)\n```\n\n4) Write to a Qdrant collection\n```python\nfrom spark_fuse.io import (\n    QDRANT_CONFIG_OPTION,\n    QDRANT_FORMAT,\n    build_qdrant_write_config,\n    register_qdrant_data_source,\n)\n\nregister_qdrant_data_source(spark)\nwrite_cfg = build_qdrant_write_config(\n    \"http://localhost:6333\",\n    collection=\"pokemon\",\n    id_field=\"id\",\n    vector_field=\"embedding\",\n    payload_fields=[\"name\", \"type\"],\n)\ndf.write.format(QDRANT_FORMAT).option(QDRANT_CONFIG_OPTION, json.dumps(write_cfg)).save()\n```\n\n5) Generate embeddings with LangChain (optionally split text)\n```python\nfrom langchain_openai import OpenAIEmbeddings\nfrom langchain_text_splitters import RecursiveCharacterTextSplitter\nfrom spark_fuse.utils.llm import with_langchain_embeddings\n\nsplitter = RecursiveCharacterTextSplitter(chunk_size=256, chunk_overlap=32)\nembedded = with_langchain_embeddings(\n    df,\n    input_col=\"text\",\n    embeddings=lambda: OpenAIEmbeddings(model=\"text-embedding-3-small\"),\n    text_splitter=splitter,\n    output_col=\"embedding\",\n    aggregation=\"mean\",\n    batch_size=16,\n)\nembedded.select(\"text\", \"embedding\").show(3, truncate=False)\n```\nPass a factory (`lambda: OpenAIEmbeddings(...)`) when the client cannot be pickled or needs executor-local setup. Provide a LangChain text splitter to chunk long documents before embedding; chunk vectors are combined with the chosen `aggregation` strategy (`mean` or `first`). Install `langchain-core`, `langchain-openai`, and `langchain-text-splitters` to use this helper.\n\n6) Build date/time dimensions with rich attributes\n```python\nfrom spark_fuse.utils.dataframe import create_date_dataframe, create_time_dataframe\n\ndate_dim = create_date_dataframe(spark, \"2024-01-01\", \"2024-01-07\")\ntime_dim = create_time_dataframe(spark, \"00:00:00\", \"23:59:00\", interval_seconds=60)\n\ndate_dim.select(\"date\", \"year\", \"week\", \"day_name\").show()\ntime_dim.select(\"time\", \"hour\", \"minute\").show(5)\n```\nCheck out `notebooks/demos/date_time_dimensions_demo.ipynb` for an interactive walkthrough.\n\n7) Partition embeddings and pick representatives\n```python\nfrom spark_fuse.similarity import (\n    CosineSimilarity,\n    IdentityEmbeddingGenerator,\n    KMeansPartitioner,\n    MaxColumnChoice,\n    SimilarityPipeline,\n)\n\npipeline = SimilarityPipeline(\n    embedding_generator=IdentityEmbeddingGenerator(input_col=\"embedding\"),\n    partitioner=KMeansPartitioner(k=3, seed=7),\n    similarity_metric=CosineSimilarity(embedding_col=\"embedding\"),\n    choice_function=MaxColumnChoice(column=\"score\"),\n)\n\nclustered = pipeline.run(df)\nrepresentatives = pipeline.select_representatives(clustered)\n```\nSee `docs/guides/similarity_partitioning_demo.md` for a walkthrough and `notebooks/demos/similarity_pipeline_demo.ipynb` for an\ninteractive companion.\n\nLLM-Powered Column Mapping\n```python\nfrom spark_fuse.utils.llm import map_column_with_llm\n\nstandard_values = [\"Apple\", \"Banana\", \"Cherry\"]\nmapped_df = map_column_with_llm(\n    df,\n    column=\"fruit\",\n    target_values=standard_values,\n    model=\"o4-mini\",\n    temperature=None,\n)\nmapped_df.select(\"fruit\", \"fruit_mapped\").show()\n```\n\nSet `dry_run=True` to inspect how many rows already match without spending LLM tokens. Configure your OpenAI or Azure OpenAI credentials with the usual environment variables before running live mappings. Some provider models only accept their default sampling configuration—pass `temperature=None` to omit the parameter when needed. The helper is available across spark-fuse 0.2.0 and later, including the 1.0.x series.\n\nCLI Usage\n- `spark-fuse --help`\n- `spark-fuse datasources`\n- `spark-fuse read --format rest --path https://pokeapi.co/api/v2/pokemon --config rest.json --show 5`\n\nCI\n- GitHub Actions runs ruff and pytest for Python 3.9–3.11.\n\nLicense\n- Apache 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkevinsames%2Fspark-fuse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkevinsames%2Fspark-fuse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkevinsames%2Fspark-fuse/lists"}