{"id":28975068,"url":"https://github.com/octoenergy/tentaclio","last_synced_at":"2025-06-24T12:08:18.119Z","repository":{"id":34741704,"uuid":"165826816","full_name":"octoenergy/tentaclio","owner":"octoenergy","description":"Single repository regrouping IO connectors used in the data world.","archived":false,"fork":false,"pushed_at":"2024-05-08T14:49:09.000Z","size":910,"stargazers_count":22,"open_issues_count":12,"forks_count":3,"subscribers_count":97,"default_branch":"master","last_synced_at":"2025-05-19T11:42:59.129Z","etag":null,"topics":["data","data-science","database-connections","protocols","python3"],"latest_commit_sha":null,"homepage":"https://tentaclio.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/octoenergy.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.txt","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-15T09:55:12.000Z","updated_at":"2025-01-21T22:17:04.000Z","dependencies_parsed_at":"2023-02-16T21:15:54.817Z","dependency_job_id":"e53fb1de-859c-45c2-b409-67cc39108173","html_url":"https://github.com/octoenergy/tentaclio","commit_stats":{"total_commits":405,"total_committers":16,"mean_commits":25.3125,"dds":"0.49135802469135803","last_synced_commit":"25381741d692f8cc21325c1e34a851426e9400ae"},"previous_names":[],"tags_count":38,"template":false,"template_full_name":null,"purl":"pkg:github/octoenergy/tentaclio","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/octoenergy%2Ftentaclio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/octoenergy%2Ftentaclio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/octoenergy%2Ftentaclio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/octoenergy%2Ftentaclio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/octoenergy","download_url":"https://codeload.github.com/octoenergy/tentaclio/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/octoenergy%2Ftentaclio/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261669021,"owners_count":23192362,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","data-science","database-connections","protocols","python3"],"created_at":"2025-06-24T12:08:15.933Z","updated_at":"2025-06-24T12:08:18.087Z","avatar_url":"https://github.com/octoenergy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tentaclio\n\n[![CircleCI status](https://circleci.com/gh/octoenergy/tentaclio/tree/master.png?circle-token=df7aad11367f1ace5bce253b18efb6b21eaa65bc)](https://circleci.com/gh/octoenergy/tentaclio/tree/master)\n[![Documentation Status](https://readthedocs.org/projects/tentaclio/badge/?version=latest)](https://tentaclio.readthedocs.io/en/latest/?badge=latest)\n\nPython library that simplifies:\n* Handling streams from different protocols such as `file:`, `ftp:`, `sftp:`, `s3:`, ...\n* Opening database connections.\n* Managing the credentials in distributed systems.\n\nMain considerations in the design:\n* Easy to use: all streams are open via `tentaclio.open`, all database connections through `tentaclio.db`.\n* URLs are the basic resource locator and db connection string.\n* Automagic authentication for protected resources.\n* Extensible: you can add your own handlers for other schemes.\n* Pandas interaction.\n\n# Quick Examples.\n\n## Read and write streams.\n```python\nimport tentaclio\ncontents = \"👋 🐙\"\n\nwith tentaclio.open(\"ftp://localhost:2021/upload/file.txt\", mode=\"w\") as writer:\n    writer.write(contents)\n\n# Using boto3 authentication under the hood.\nbucket = \"s3://my-bucket/octopus/hello.txt\"\nwith tentaclio.open(bucket) as reader:\n    print(reader.read())\n```\n\n## Copy streams\n```python\nimport tentaclio\n\ntentaclio.copy(\"/home/constantine/data.csv\", \"sftp://constantine:tentacl3@sftp.octoenergy.com/uploads/data.csv\")\n```\n## Delete resources\n```python\nimport tentaclio\n\ntentaclio.remove(\"s3://my-bucket/octopus/the-9th-tentacle.txt\")\n```\n## List resources\n```python\nimport tentaclio\n\nfor entry in tentaclio.listdir(\"s3:://mybucket/path/to/dir\"):\n    print(\"Entry\", entry)\n```\n\n## Authenticated resources.\n```python\nimport os\n\nimport tentaclio\n\nprint(\"env ftp credentials\", os.getenv(\"OCTOIO__CONN__OCTOENERGY_FTP\"))\n# This prints `sftp://constantine:tentacl3@sftp.octoenergy.com/`\n\n# Credentials get automatically injected.\n\nwith tentaclio.open(\"sftp://sftp.octoenergy.com/uploads/data.csv\") as reader:\n    print(reader.read())\n```\n\n## Database connections.\n```python\nimport os\n\nimport tentaclio\n\nprint(\"env TENTACLIO__CONN__DB\", os.getenv(\"TENTACLIO__CONN__DB\"))\n\n# This prints `postgresql://octopus:tentacle@localhost:5444/example`\n\n# hostname is a wildcard, the credentials get injected.\nwith tentaclio.db(\"postgresql://hostname/example\") as pg:\n    results = pg.query(\"select * from my_table\")\n```\n\n## Pandas interaction.\n```python\nimport pandas as pd  # 🐼🐼\nimport tentaclio  # 🐙\n\ndf = pd.DataFrame([[1, 2, 3], [10, 20, 30]], columns=[\"col_1\", \"col_2\", \"col_3\"])\n\nbucket = \"s3://my-bucket/data/pandas.csv\"\n\nwith tentaclio.open(bucket, mode=\"w\") as writer:  # supports more pandas readers\n    df.to_csv(writer, index=False)\n\nwith tentaclio.open(bucket) as reader:\n    new_df = pd.read_csv(reader)\n\n# another example: using pandas.DataFrame.to_sql() with tentaclio to upload\nwith tentaclio.db(\n        connection_info,\n        connect_args={'options': '-csearch_path=schema_name'}\n    ) as client:\n    df.to_sql(\n        name='observations', # table name\n        con=client.conn,\n    )\n```\n\n# Installation\n\nYou can get tentaclio using pip\n\n```sh\npip install tentaclio\n```\nor pipenv\n```sh\npipenv install tentaclio\n```\n\n## Developing.\n\nClone this repo and install [pipenv](https://pipenv.readthedocs.io/en/latest/):\n\nIn the `Makefile` you'll find some useful targets for linting, testing, etc. i.e.:\n```sh\nmake test\n```\n\n\n## How to use\nThis is how to use `tentaclio` for your daily data ingestion and storing needs.\n\n### Streams\nIn order to open streams to load or store data the universal function is:\n\n```python\nimport tentaclio\n\nwith tentaclio.open(\"/path/to/my/file\") as reader:\n    contents = reader.read()\n\nwith tentaclio.open(\"s3://bucket/file\", mode='w') as writer:\n    writer.write(contents)\n\n```\nAllowed modes are `r`, `w`, `rb`, and `wb`. You can use `t` instead of `b` to indicate text streams, but that's the default.\n\nIn order to keep tentaclio as light as possible, it only includes `file`, `ftp`, `sftp`, `http` and `https` schemes by default.\nHowever, many more are easily available by installing extra packages:\n\nDefault:\n* `/local/file`\n* `file:///local/file`\n* `ftp://path/to/file`\n* `sftp://path/to/file`\n* `http://host.com/path/to/resource`\n* `https://host.com/path/to/resource`\n\n[tentaclio-s3](https://github.com/octoenergy/tentaclio-s3)\n* `s3://bucket/file`\n\n[tentaclio-gs](https://github.com/octoenergy/tentaclio-gs)\n* `gs://bucket/file`\n* `gsc://bucket/file`\n\n[tentaclio-gdrive](https://github.com/octoenergy/tentaclio-gdrive)\n* `gdrive:/My Drive/file`\n* `googledrive:/My Drive/file`\n\n[tentaclio-postgres](https://github.com/octoenergy/tentaclio-postgres)\n* `postgresql://host/database::table` will allow you to write from a csv format into a database with the same column names (note that the table goes after `::` :warning:).\n\n\nYou can add the credentials for any of the urls in order to access protected resources.\n\n\nYou can use these readers and writers with pandas functions like:\n\n```python\nimport pandas as pd\nimport tentaclio\n\nwith tentaclio.open(\"/path/to/my/file\") as reader:\n    df = pd.read_csv(reader)\n\n[...]\n\nwith tentaclio.open(\"s3::/path/to/my/file\", mode='w') as writer:\n    df.to_parquet(writer)\n```\n`Readers`, `Writers` and their closeable versions can be used anywhere expecting a file-like object; pandas or pickle are examples of such functions.\n\n##### Notes on writing files for Spark, Presto, and similar downstream systems\n\nThe default behaviour for the `open` context manager in python is to create an empty file when opening\nit in writable mode. This can be annoying if the process that creates the data within the `with` clause\nyields empty dataframes and nothing gets written. This will make Spark and Presto panic.\n\nTo avoid this we can make the stream _empty safe_ so the empty buffer won't be flushed if no writes have been performed so no empty file will be created.\n\n\n```\nwith tio.make_empty_safe(tio.open(\"s3://bucket/file.parquet\", mode=\"wb\")) as writer:\n    if not df.empty:\n        df.to_parquet(writer)\n```\n\n### File system like operations to resources\n#### Listing resources\nSome URL schemes allow listing resources in a pythonnic way:\n```python\nimport tentaclio\n\nfor entry in tentaclio.listdir(\"s3:://mybucket/path/to/dir\"):\n    print(\"Entry\", entry)\n```\n\nWhereas `listdir` might be convinient we also offer `scandir`, which returns a list of [DirEntry](https://github.com/octoenergy/tentaclio/blob/ddbc28615de4b99106b956556db74a20e4761afe/src/tentaclio/fs/scanner.py#L13)s, and, `walk`. All functions follow as closely as possible their standard library definitions.\n\n\n### Database access\n\nIn order to open db connections you can use `tentaclio.db` and have instant access to postgres, sqlite, athena and mssql.\n\n```python\nimport tentaclio\n\n[...]\n\nquery = \"select 1\";\nwith tentaclio.db(POSTGRES_TEST_URL) as client:\n    result =client.query(query)\n[...]\n```\n\nThe supported db schemes are:\n\nDefault:\n* `sqlite://`\n* `mssql://`\n* + Any other scheme supported by sqlalchemy.\n\n[tentaclio-postgres](https://github.com/octoenergy/tentaclio-postgres)\n* `postgresql://`\n\n[tentaclio-athena](https://github.com/octoenergy/tentaclio-athena)\n* `awsathena+rest://`\n\n[tentaclio-databricks](https://github.com/octoenergy/tentaclio-databricks)\n* `databricks+thrift://`\n\n[tentaclio-snowflake](https://github.com/octoenergy/tentaclio-snowflake)\n* `snowflake://`\n\n\n#### Extras for databases\nFor postgres you can set the variable `TENTACLIO__PG_APPLICATION_NAME` and the value will be injected\nwhen connecting to the database.\n\n### Automatic credentials injection\n\n1. Configure credentials by using environmental variables prefixed with `TENTACLIO__CONN__`  (i.e.  `TENTACLIO__CONN__DATA_FTP=sfpt://real_user:132ldsf@ftp.octoenergy.com`).\n\n2. Open a stream:\n```python\nwith tentaclio.open(\"sftp://ftp.octoenergy.com/file.csv\") as reader:\n    reader.read()\n```\nThe credentials get injected into the url.\n\n3. Open a db client:\n```python\nimport tentaclio\n\nwith tentaclio.db(\"postgresql://hostname/my_data_base\") as client:\n    client.query(\"select 1\")\n```\nNote that `hostname` in the url to be authenticated is a wildcard that will match any hostname. So `authenticate(\"http://hostname/file.txt\")` will be injected to `http://user:pass@octo.co/file.txt` if the credential for `http://user:pass@octo.co/` exists.\n\nDifferent components of the URL are set differently:\n- Scheme and path will be set from the URL, and null if missing.\n- Username, password and hostname will be set from the stored credentials.\n- Port will be set from the stored credentials if it exists, otherwise from the URL.\n- Query will be set from the URL if it exists, otherwise from the stored credentials (so it can be\n  overriden)\n\n#### Credentials file\n\nYou can also set a credentials file that looks like:\n```\nsecrets:\n    db_1: postgresql://user1:pass1@myhost.com/database_1\n    db_2: mssql://user2:pass2@otherhost.com/database_2?driver=ODBC+Driver+17+for+SQL+Server\n    ftp_server: ftp://fuser:fpass@ftp.myhost.com\n```\nAnd make it accessible to tentaclio by setting the environmental variable `TENTACLIO__SECRETS_FILE`. The actual name of each url is for traceability and has no effect in the functionality.\n\n(Note that you may need to add `?driver={driver from /usr/local/etc/odbcinst.ini}` for mssql database connection strings; see above example)\n\nAlternatively you can run `curl https://raw.githubusercontent.com/octoenergy/tentaclio/master/extras/init_tentaclio.sh` to create a secrets file in `~/.tentaclio.yml` and\nautomatically configure your environment.\n\nEnvironment variables can be included in the credentials file by using `${ENV_VARIABLE}` as it follows:\n```\nsecrets:\n    db: postgresql://${DB_USER}:${DB_PASS}@myhost.com/database\n```\nTentaclio will search `DB_USER` and `DB_PASS` in the environment and will interpolate their values with the secrets file content.\n\n\n## Quick note on protocols structural subtyping.\n\nIn order to abstract concrete dependencies from the implementation of data related functions (or in any part of the system really) we use typed [protocols](https://mypy.readthedocs.io/en/latest/protocols.html#simple-user-defined-protocols). This allows a more flexible dependency injection than using subclassing or [more complex approches](http://code.activestate.com/recipes/413268/). This idea is heavily inspired by how this exact thing is done in [go](https://www.youtube.com/watch?v=ifBUfIb7kdo). Learn more about this principle in our [tech blog](https://tech.octopus.energy/news/2019/03/21/python-interfaces-a-la-go.html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foctoenergy%2Ftentaclio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foctoenergy%2Ftentaclio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foctoenergy%2Ftentaclio/lists"}