{"id":34021256,"url":"https://github.com/slingdata-io/sling-python","last_synced_at":"2026-04-07T04:31:46.350Z","repository":{"id":53842023,"uuid":"477690326","full_name":"slingdata-io/sling-python","owner":"slingdata-io","description":"Python wrapper for the Sling CLI tool","archived":false,"fork":false,"pushed_at":"2026-02-07T02:58:33.000Z","size":138,"stargazers_count":64,"open_issues_count":4,"forks_count":12,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-02-22T22:57:55.752Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://docs.slingdata.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/slingdata-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-04-04T12:27:09.000Z","updated_at":"2026-02-21T18:09:51.000Z","dependencies_parsed_at":"2023-01-30T16:49:41.338Z","dependency_job_id":"cd0ab69e-1f7e-4e89-992c-6fc4268f380b","html_url":"https://github.com/slingdata-io/sling-python","commit_stats":{"total_commits":26,"total_committers":1,"mean_commits":26.0,"dds":0.0,"last_synced_commit":"838f924adb0b0c3d699ff3e12159a186caef2144"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/slingdata-io/sling-python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slingdata-io%2Fsling-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slingdata-io%2Fsling-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slingdata-io%2Fsling-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slingdata-io%2Fsling-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/slingdata-io","download_url":"https://codeload.github.com/slingdata-io/sling-python/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slingdata-io%2Fsling-python/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31500397,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-07T03:10:19.677Z","status":"ssl_error","status_checked_at":"2026-04-07T03:10:13.982Z","response_time":105,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-13T15:50:16.925Z","updated_at":"2026-04-07T04:31:46.334Z","avatar_url":"https://github.com/slingdata-io.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\u003cimg src=\"https://github.com/slingdata-io/sling-python/raw/main/logo-with-text.png\" alt=\"logo\" width=\"250\"/\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003eSlings from a data source to a data target.\u003c/p\u003e\n\n## Installation\n\n`pip install sling` or `pip install sling[arrow]` for streaming.\n\nThen you should be able to run `sling --help` from command line.\n\n## Running a Extract-Load Task\n\n### CLI\n\n```shell\nsling run --src-conn MY_PG --src-stream myschema.mytable \\\n  --tgt-conn YOUR_SNOWFLAKE --tgt-object yourschema.yourtable \\\n  --mode full-refresh\n```\n\nOr passing a yaml/json string or file\n\n```shell\ncat '\nsource: MY_POSTGRES\ntarget: MY_SNOWFLAKE\n\n# default config options which apply to all streams\ndefaults:\n  mode: full-refresh\n  object: new_schema.{stream_schema}_{stream_table}\n\nstreams:\n  my_schema.*:\n' \u003e /path/to/replication.yaml\n\nsling run -r /path/to/replication.yaml\n```\n\n### Using the `Replication` class\n\nRun a replication from file:\n\n```python\nimport yaml\nfrom sling import Replication\n\n# From a YAML file\nreplication = Replication(file_path=\"path/to/replication.yaml\")\nreplication.run()\n\n# Or load into object\nwith open('path/to/replication.yaml') as file:\n  config = yaml.load(file, Loader=yaml.FullLoader)\n\nreplication = Replication(**config)\n\nreplication.run()\n```\n\nBuild a replication dynamically:\n\n```python\nfrom sling import Replication, ReplicationStream, Mode\n\n# build sling replication\nstreams = {}\nfor (folder, table_name) in list(folders):\n  streams[folder] = ReplicationStream(\n    mode=Mode.FULL_REFRESH, object=table_name, primary_key='_hash_id')\n\nreplication = Replication(\n  source='aws_s3',\n  target='snowflake',\n  streams=streams,\n  env=dict(SLING_STREAM_URL_COLUMN='true', SLING_LOADED_AT_COLUMN='true'),\n  debug=True,\n)\n\nreplication.run()\n```\n\n### Using the `Sling` Class\n\nFor more direct control and streaming capabilities, you can use the `Sling` class, which mirrors the CLI interface.\n\n#### Basic Usage with `run()` method\n\n```python\nimport os\nfrom sling import Sling, Mode\n\n# Set postgres \u0026 snowflake connection\n# see https://docs.slingdata.io/connections/database-connections\nos.environ[\"POSTGRES\"] = 'postgres://...'\nos.environ[\"SNOWFLAKE\"] = 'snowflake://...'\n\n# Database to database transfer\nSling(\n    src_conn=\"postgres\",\n    src_stream=\"public.users\",\n    tgt_conn=\"snowflake\",\n    tgt_object=\"public.users_copy\",\n    mode=Mode.FULL_REFRESH\n).run()\n\n# Database to file\nSling(\n    src_conn=\"postgres\", \n    src_stream=\"select * from users where active = true\",\n    tgt_object=\"file:///tmp/active_users.csv\"\n).run()\n\n# File to database\nSling(\n    src_stream=\"file:///path/to/data.csv\",\n    tgt_conn=\"snowflake\",\n    tgt_object=\"public.imported_data\"\n).run()\n```\n\n\n#### Input Streaming - Python Data to Target\n\n\u003e **💡 Tip:** Install `pip install sling[arrow]` for better streaming performance and improved data type handling.\n\n\u003e **📊 DataFrame Support:** The `input` parameter accepts lists of dictionaries, pandas DataFrames, or polars DataFrames. DataFrame support preserves data types when using Arrow format.\n\n\u003e **⚠️ Note:** Be careful with large numbers of `Sling` invocations using `input` or `stream()` methods when working with external systems (databases, file systems). Each call re-opens the connection since it invokes the underlying sling binary. For better performance and connection reuse, consider using the `Replication` class instead, which maintains open connections across multiple operations.\n\n```python\nimport os\nfrom sling import Sling, Format\n\n# Set postgres connection\n# see https://docs.slingdata.io/connections/database-connections\nos.environ[\"POSTGRES\"] = 'postgres://...'\n\n# Stream Python data to CSV file\ndata = [\n    {\"id\": 1, \"name\": \"John\", \"age\": 30},\n    {\"id\": 2, \"name\": \"Jane\", \"age\": 25},\n    {\"id\": 3, \"name\": \"Bob\", \"age\": 35}\n]\n\nSling(\n    input=data,\n    tgt_object=\"file:///tmp/output.csv\"\n).run()\n\n# Stream Python data to database\nSling(\n    input=data,\n    tgt_conn=\"postgres\",\n    tgt_object=\"public.users\"\n).run()\n\n# Stream Python data to JSON Lines file\nSling(\n    input=data,\n    tgt_object=\"file:///tmp/output.jsonl\",\n    tgt_options={\"format\": Format.JSONLINES}\n).run()\n\n# Stream from generator (memory efficient for large datasets)\ndef data_generator():\n    for i in range(10000):\n        yield {\"id\": i, \"value\": f\"item_{i}\", \"timestamp\": \"2023-01-01\"}\n\nSling(input=data_generator(), tgt_object=\"file:///tmp/large_dataset.csv\").run()\n\n# Stream pandas DataFrame to database\nimport pandas as pd\n\ndf = pd.DataFrame({\n    \"id\": [1, 2, 3, 4],\n    \"name\": [\"Alice\", \"Bob\", \"Charlie\", \"Diana\"],\n    \"age\": [25, 30, 35, 28],\n    \"salary\": [50000, 60000, 70000, 55000]\n})\n\nSling(\n    input=df,\n    tgt_conn=\"postgres\",\n    tgt_object=\"public.employees\"\n).run()\n\n# Stream polars DataFrame to CSV file\nimport polars as pl\n\ndf = pl.DataFrame({\n    \"product_id\": [101, 102, 103],\n    \"product_name\": [\"Laptop\", \"Mouse\", \"Keyboard\"],\n    \"price\": [999.99, 25.50, 75.00],\n    \"in_stock\": [True, False, True]\n})\n\nSling(\n    input=df,\n    tgt_object=\"file:///tmp/products.csv\"\n).run()\n\n# DataFrame with column selection\nSling(\n    input=df,\n    select=[\"product_name\", \"price\"],  # Only export specific columns\n    tgt_object=\"file:///tmp/product_prices.csv\"\n).run()\n```\n\n#### Output Streaming with `stream()`\n\n```python\nimport os\nfrom sling import Sling\n\n# Set postgres connection\n# see https://docs.slingdata.io/connections/database-connections\nos.environ[\"POSTGRES\"] = 'postgres://...'\n\n# Stream data from database\nsling = Sling(\n    src_conn=\"postgres\",\n    src_stream=\"public.users\",\n    limit=1000\n)\n\nfor record in sling.stream():\n    print(f\"User: {record['name']}, Age: {record['age']}\")\n\n# Stream data from file\nsling = Sling(\n    src_stream=\"file:///path/to/data.csv\"\n)\n\n# Process records one by one (memory efficient)\nfor record in sling.stream():\n    # Process each record\n    processed_data = transform_record(record)\n    # Could save to another system, send to API, etc.\n\n# Stream with parameters\nsling = Sling(\n    src_conn=\"postgres\",\n    src_stream=\"public.orders\",\n    select=[\"order_id\", \"customer_name\", \"total\"],\n    where=\"total \u003e 100\",\n    limit=500\n)\n\nrecords = list(sling.stream())\nprint(f\"Found {len(records)} high-value orders\")\n```\n\n#### High-Performance Streaming with `stream_arrow()`\n\n\u003e **🚀 Performance:** The `stream_arrow()` method provides the highest performance streaming with full data type preservation by using Apache Arrow's columnar format. Requires `pip install sling[arrow]`.\n\n\u003e **📊 Type Safety:** Unlike `stream()` which may convert data types during CSV serialization, `stream_arrow()` preserves exact data types including integers, floats, timestamps, and more.\n\n```python\nimport os\nfrom sling import Sling\n\n# Set postgres connection  \n# see https://docs.slingdata.io/connections/database-connections\nos.environ[\"POSTGRES\"] = 'postgres://...'\n\n# Basic Arrow streaming from database\nsling = Sling(src_conn=\"postgres\", src_stream=\"public.users\", limit=1000)\n\n# Get Arrow RecordBatchStreamReader for maximum performance\nreader = sling.stream_arrow()\n\n# Convert to Arrow Table for analysis\ntable = reader.read_all()\nprint(f\"Received {table.num_rows} rows with {table.num_columns} columns\")\nprint(f\"Column names: {table.column_names}\")\nprint(f\"Schema: {table.schema}\")\n\n# Convert to pandas DataFrame with preserved types\nif table.num_rows \u003e 0:\n    df = table.to_pandas()\n    print(df.dtypes)  # Shows preserved data types\n\n# Stream Arrow file with type preservation\nsling = Sling(\n    src_stream=\"file:///path/to/data.arrow\",\n    src_options={\"format\": \"arrow\"}\n)\n\nreader = sling.stream_arrow()\ntable = reader.read_all()\n\n# Access columnar data directly (very efficient)\nfor column_name in table.column_names:\n    column = table.column(column_name)\n    print(f\"{column_name}: {column.type}\")\n\n# Process Arrow batches for large datasets (memory efficient)\nsling = Sling(\n    src_conn=\"postgres\", \n    src_stream=\"select * from large_table\"\n)\n\nreader = sling.stream_arrow()\nfor batch in reader:\n    # Process each batch separately to manage memory\n    print(f\"Processing batch with {batch.num_rows} rows\")\n    # Convert batch to pandas if needed\n    batch_df = batch.to_pandas()\n    # Process batch_df...\n\n# Round-trip with Arrow format preservation\nimport pandas as pd\n\n# Write DataFrame to Arrow file with type preservation\ndf = pd.DataFrame({\n    \"id\": [1, 2, 3],\n    \"amount\": [100.50, 250.75, 75.25],\n    \"timestamp\": pd.to_datetime([\"2023-01-01\", \"2023-01-02\", \"2023-01-03\"]),\n    \"active\": [True, False, True]\n})\n\nSling(\n    input=df,\n    tgt_object=\"file:///tmp/data.arrow\",\n    tgt_options={\"format\": \"arrow\"}\n).run()\n\n# Read back with full type preservation\nsling = Sling(\n    src_stream=\"file:///tmp/data.arrow\",\n    src_options={\"format\": \"arrow\"}\n)\n\nreader = sling.stream_arrow()\nrestored_table = reader.read_all()\nrestored_df = restored_table.to_pandas()\n\n# Types are exactly preserved (no string conversion)\nprint(restored_df.dtypes)\nassert restored_df['active'].dtype == 'bool'\nassert 'datetime64' in str(restored_df['timestamp'].dtype)\n```\n\n**Notes:**\n- `stream_arrow()` requires PyArrow: `pip install sling[arrow]`\n- Cannot be used with a target object (use `run()` instead)\n- Provides the best performance for large datasets\n- Preserves exact data types including timestamps, decimals, and booleans\n- Ideal for analytics workloads and data science applications\n\n#### Round-trip Examples\n\n```python\nimport os\nfrom sling import Sling\n\n# Set postgres connection\n# see https://docs.slingdata.io/connections/database-connections\nos.environ[\"POSTGRES\"] = 'postgres://...'\n\n# Python → File → Python\noriginal_data = [\n    {\"id\": 1, \"name\": \"Alice\", \"score\": 95.5},\n    {\"id\": 2, \"name\": \"Bob\", \"score\": 87.2}\n]\n\n# Step 1: Python data to file\nsling_write = Sling(\n    input=original_data,\n    tgt_object=\"file:///tmp/scores.csv\"\n)\nsling_write.run()\n\n# Step 2: File back to Python\nsling_read = Sling(\n    src_stream=\"file:///tmp/scores.csv\"\n)\nloaded_data = list(sling_read.stream())\n\n# Python → Database → Python (with transformations)\nsling_to_db = Sling(\n    input=original_data,\n    tgt_conn=\"postgres\",\n    tgt_object=\"public.temp_scores\"\n)\nsling_to_db.run()\n\nsling_from_db = Sling(\n    src_conn=\"postgres\", \n    src_stream=\"select *, score * 1.1 as boosted_score from public.temp_scores\",\n)\ntransformed_data = list(sling_from_db.stream())\n\n# DataFrame → Database → DataFrame (with pandas/polars)\nimport pandas as pd\n\n# Start with pandas DataFrame\ndf = pd.DataFrame({\n    \"user_id\": [1, 2, 3],\n    \"purchase_amount\": [100.50, 250.75, 75.25],\n    \"category\": [\"electronics\", \"clothing\", \"books\"]\n})\n\n# Write DataFrame to database\nSling(\n    input=df,\n    tgt_conn=\"postgres\",\n    tgt_object=\"public.purchases\"\n).run()\n\n# Read back with SQL transformations as pandas DataFrame\nsling_query = Sling(\n    src_conn=\"postgres\",\n    src_stream=\"\"\"\n        SELECT category, \n               COUNT(*) as purchase_count,\n               AVG(purchase_amount) as avg_amount\n        FROM public.purchases \n        GROUP BY category\n    \"\"\"\n)\nsummary_data = list(sling_query.stream())\nsummary_df = pd.DataFrame(summary_data)\nprint(summary_df)\n```\n\n\n### Using the `Pipeline` class\n\nRun a [Pipeline](https://docs.slingdata.io/concepts/pipeline):\n\n```python\nfrom sling import Pipeline\nfrom sling.hooks import StepLog, StepCopy, StepReplication, StepHTTP, StepCommand\n\n# From a YAML file\npipeline = Pipeline(file_path=\"path/to/pipeline.yaml\")\npipeline.run()\n\n# Or using Hook objects for type safety\npipeline = Pipeline(\n    steps=[\n        StepLog(message=\"Hello world\"),\n        StepCopy(from_=\"sftp//path/to/file\", to=\"aws_s3/path/to/file\"),\n        StepReplication(path=\"path/to/replication.yaml\"),\n        StepHTTP(url=\"https://trigger.webhook.com\"),\n        StepCommand(command=[\"ls\", \"-l\"], print_output=True)\n    ],\n    env={\"MY_VAR\": \"value\"}\n)\npipeline.run()\n\n# Or programmatically using dictionaries\npipeline = Pipeline(\n    steps=[\n        {\"type\": \"log\", \"message\": \"Hello world\"},\n        {\"type\": \"copy\", \"from\": \"sftp//path/to/file\", \"to\": \"aws_s3/path/to/file\"},\n        {\"type\": \"replication\", \"path\": \"path/to/replication.yaml\"},\n        {\"type\": \"http\", \"url\": \"https://trigger.webhook.com\"},\n        {\"type\": \"command\", \"command\": [\"ls\", \"-l\"], \"print\": True}\n    ],\n    env={\"MY_VAR\": \"value\"}\n)\npipeline.run()\n```\n\n\n### Building API Specs with `ApiSpec`\n\nBuild [API Spec](https://docs.slingdata.io/concepts/api-specs) YAML files programmatically with type checking and validation. API specs define how Sling extracts data from REST APIs.\n\n```python\nfrom sling.api_spec import (\n    ApiSpec, Endpoint, Request, Pagination, Response, Records,\n    Processor, Rule, Iterate, Call, DynamicEndpoint,\n    AuthType, HTTPMethod, RuleAction, AggregationType, BackoffType, ResponseFormat,\n)\n\nspec = ApiSpec(\n    name=\"My API\",\n    description=\"Extract data from My API\",\n    queues=[\"user_ids\"],\n\n    defaults=Endpoint(\n        state={\"base_url\": \"https://api.example.com/v1\", \"limit\": 100},\n        request=Request(\n            headers={\n                \"Authorization\": 'Bearer {require(secrets.api_key, \"api_key required\")}',\n                \"Accept\": \"application/json\",\n            },\n            rate=5,\n            concurrency=3,\n        ),\n        response=Response(\n            records=Records(jmespath=\"data[]\", primary_key=[\"id\"]),\n            rules=[\n                Rule(\n                    action=RuleAction.RETRY,\n                    condition=\"response.status == 429\",\n                    max_attempts=5,\n                    backoff=BackoffType.EXPONENTIAL,\n                    backoff_base=2,\n                ),\n            ],\n        ),\n        pagination=Pagination(\n            next_state={\"offset\": \"{state.offset + state.limit}\"},\n            stop_condition=\"length(response.records) \u003c state.limit\",\n        ),\n    ),\n\n    endpoints={\n        \"users\": Endpoint(\n            description=\"List all users\",\n            state={\"offset\": 0},\n            request=Request(\n                url=\"{state.base_url}/users\",\n                parameters={\"limit\": \"{state.limit}\", \"offset\": \"{state.offset}\"},\n            ),\n            response=Response(\n                processors=[\n                    Processor(expression=\"record.id\", output=\"queue.user_ids\"),\n                ],\n            ),\n        ),\n\n        \"user_orders\": Endpoint(\n            description=\"Get orders for each user\",\n            iterate=Iterate(over=\"queue.user_ids\", into=\"state.user_id\", concurrency=5),\n            request=Request(url=\"{state.base_url}/users/{state.user_id}/orders\"),\n            response=Response(\n                processors=[\n                    Processor(expression=\"state.user_id\", output=\"record.user_id\"),\n                ],\n            ),\n        ),\n\n        \"metrics\": Endpoint(\n            description=\"Daily metrics (incremental)\",\n            state={\n                \"offset\": 0,\n                \"since\": '{coalesce(sync.last_date, date_format(date_add(now(), -30, \"day\"), \"%Y-%m-%d\"))}',\n            },\n            sync=[\"last_date\"],\n            request=Request(\n                url=\"{state.base_url}/metrics\",\n                parameters={\"since\": \"{state.since}\"},\n            ),\n            response=Response(\n                records=Records(primary_key=[\"id\"], update_key=\"date\"),\n                processors=[\n                    Processor(\n                        expression=\"record.date\",\n                        output=\"state.last_date\",\n                        aggregation=AggregationType.MAXIMUM,\n                    ),\n                ],\n            ),\n        ),\n    },\n)\n\n# Validate\nerrors = spec.validate()\nassert errors == [], errors\n\n# Write to file\nspec.to_yaml_file(\"my_api.yaml\")\n\n# Or get as string\nprint(spec.to_yaml())\nprint(spec.to_json())\n```\n\nParse an existing spec:\n\n```python\nfrom sling.api_spec import ApiSpec, Endpoint, Request, Response, Records\n\nspec = ApiSpec.parse_file(\"path/to/spec.yaml\")\nprint(spec.name)\nprint(list(spec.endpoints.keys()))\n\n# Modify and re-export\nspec.endpoints[\"new_endpoint\"] = Endpoint(\n    request=Request(url=\"{state.base_url}/new\"),\n    response=Response(records=Records(primary_key=[\"id\"])),\n)\nspec.to_yaml_file(\"updated_spec.yaml\")\n```\n\nUse `+rules`/`+processors` modifiers to append to defaults without replacing them:\n\n```python\nfrom sling.api_spec import Endpoint, Request, Response, Rule, RuleAction\n\nendpoint = Endpoint(\n    request=Request(url=\"{state.base_url}/fragile\"),\n    response=Response(\n        # append_rules serializes as \"rules+\" in YAML, keeping default rules intact\n        append_rules=[Rule(action=RuleAction.SKIP, condition=\"response.status == 404\")],\n    ),\n)\n```\n\n## Testing\n\n```bash\npytest sling/tests/tests.py -v\npytest sling/tests/test_sling_class.py -v\n```\n\n## MCP\n\nTo Login:\n\n```\nmcp-publisher login dns --domain slingdata.io --private-key $(openssl pkey -in mcp-key.pem  -noout -text | grep -A3 \"priv:\" | tail -n +2 | tr -d ' :\\n')`\n```\n\nTo Publish:\n\n```bash\n# to publish, adjust the version first in server.json\nmcp-publisher publish\n\n# check\ncurl \"https://registry.modelcontextprotocol.io/v0/servers?search=io.slingdata/sling-cli\"\n```\n\n\nmcp-name: io.slingdata/sling-cli\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fslingdata-io%2Fsling-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fslingdata-io%2Fsling-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fslingdata-io%2Fsling-python/lists"}