{"id":16200201,"url":"https://github.com/mobarski/st_s3kv_connection","last_synced_at":"2026-04-29T00:34:41.927Z","repository":{"id":185716702,"uuid":"673990452","full_name":"mobarski/st_s3kv_connection","owner":"mobarski","description":"Streamlit S3 connection acting as a key-value store.","archived":false,"fork":false,"pushed_at":"2023-08-03T20:22:51.000Z","size":207,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-07T17:49:54.179Z","etag":null,"topics":["cloudpickle","kev-value","python","s3","streamlit"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mobarski.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-02T22:27:59.000Z","updated_at":"2023-08-03T19:17:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"90bfa3bb-d376-4f87-b5f9-ecdc14618070","html_url":"https://github.com/mobarski/st_s3kv_connection","commit_stats":null,"previous_names":["mobarski/st_s3kv_connection"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mobarski/st_s3kv_connection","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mobarski%2Fst_s3kv_connection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mobarski%2Fst_s3kv_connection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mobarski%2Fst_s3kv_connection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mobarski%2Fst_s3kv_connection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mobarski","download_url":"https://codeload.github.com/mobarski/st_s3kv_connection/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mobarski%2Fst_s3kv_connection/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32405901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-28T19:38:08.556Z","status":"ssl_error","status_checked_at":"2026-04-28T19:37:55.688Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudpickle","kev-value","python","s3","streamlit"],"created_at":"2024-10-10T09:29:31.609Z","updated_at":"2026-04-29T00:34:41.911Z","avatar_url":"https://github.com/mobarski.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Streamlit S3KV Connection\n\nConnect from Streamlit to S3 and use it as an encrypted key-value database.\n\n\n\nFeatures:\n\n- key-value (dict) interface → easy to use\n- cloudpickle serialization → serialize anything\n- data at rest encryption (AES) → better data security\n- compression →faster transfers, lower storage costs\n- ability to work without S3 → easy testing\n- caching →less transfers\n\n\n\n**Note**\n\nThe S3KV connection communicates with S3 in the cloud, which can  introduce latency compared to a local key-value store. This makes it  less ideal for use cases requiring millisecond response times.\n\nHowever, the S3KV connection excels at securely storing large  documents like JSON, CSV, PDFs, images, etc. The cloud storage and  compression provide very cost-effective storage for large amounts of  data. The encryption keeps sensitive documents secure. So while not the  fastest, the S3KV connection offers inexpensive, encrypted document  storage with a simple key-value interface. The latency trade-off is  often worth it for document-heavy use cases.\n\n\n\n## Installation\n\n`pip install git+https://github.com/mobarski/st_s3kv_connection`\n\n\n\n## Quick demonstration\n\n```python\nimport streamlit as st\nfrom st_s3kv_connection import S3KV\n\ns3 = st.experimental_connection('my_kv', type=S3KV)\ndb = s3.collection('my-collection')\n\ndb['my-key'] = '1234567890' * 10_000_000\n\nst.write(len(db['my-key']))\nst.write(db['my-key'][:100])\n```\n\n**No S3? No problem!**\n\n```python\ns3 = st.experimental_connection('my_kv', type=S3KV, mode='local', path='path/to/local/data/directory')\n```\n\n```python\ns3 = st.experimental_connection('my_kv', type=S3KV, mode='dict')\n```\n\n\n\n## Main methods\n\n\n\n#### collection()\n\n`connection.collection(name, **kwargs) -\u003e dictionary-like object`\n\nGet dictionary-like object that can be used to interact with the collection.\n\n\n\n## Configuration\n\nThe connection configuration can be:\n\n- passed via collection kwargs\n- passed via connection kwargs\n- passed through environmental variables\n- stored in Streamlit's [secrets.toml](https://docs.streamlit.io/library/advanced-features/secrets-management) file (~/.streamlit/secrets.toml on Linux)\n\nYou can find more information about managing connections in [this section](https://docs.streamlit.io/library/advanced-features/connecting-to-data#global-secrets-managing-multiple-apps-and-multiple-data-stores) of Streamlit documentation **and some examples below**.\n\n\n\n##### most importatnt parameters\n\n- `mode` - connection mode\n  - `s3` - store data in S3 bucket\n\n  - `local` - store data in local directory\n\n  - `dict` - store data in RAM\n\n- `password` - password that will be used to generate data encryption key (default: '')\n\n- `salt` - cryptographic salt that will be used to generate data encryption key (default: '')\n\n- `ttl` - Streamlit's cache time-to-live option (default: None → no caching), more info [here](docs.streamlit.io/library/advanced-features/caching#controlling-cache-size-and-duration)\n\n\n\n##### mode = 's3'\n\n- `access_key` - access key for the account\n- `secret_key ` - secret key for the account\n- `endpoint` - endpoint URL (required for providers other than AWS)\n- `addressing` - [addressing style](https://boto3.amazonaws.com/v1/documentation/api/1.9.42/guide/s3.html#changing-the-addressing-style) [`path`|`auto`|`virtual`] (default: '' → 'auto')\n- `region` - region to use\n- `bucket` - bucket name\n- `prefix` - prefix to use for all the keys (default: '')\n\n\n\n##### mode = 'local'\n\n- `path` - directory where the data will be stored (default: '' → current working directory)\n\n\n\n##### mode = 'dict'\n\n- `data` - dictionary-like object that will be used to store the data (default: None → new dictionary will be created)\n\n  \n\n## Usage examples\n\n\n\n##### simple_app.py\n\n```python\nimport streamlit as st\nimport numpy as np\nfrom st_s3kv_connection import S3KV\n\ns3 = st.experimental_connection('my_kv', type=S3KV)\ndb = s3.collection('examples', 'password-to-this-collection')\n\ndb['key1'] = \"🎈\"\ndb['key2'] = {'name':'Boaty McBoatface', 'launched':2017}\ndb['key3'] = np.array([42, 2501, 1337])\ndb['key4'] = lambda x: f'Hello {x}!'\n\nst.write(db['key1'])\nst.write(db['key4'](db['key2']['name']))\n\ndel db['key3']\nst.write(db.keys())\n```\n\n\n\n##### demo_app.py\n\nYou can find live demo of this app [here](https://s3kv-demo.streamlit.app)\n\n```python\nimport streamlit as st\nfrom st_s3kv_connection import S3KV\nimport requests\n\nst.title('S3KV Demo')\nss = st.session_state\n\ns3 = st.experimental_connection('my_s3', type=S3KV)\ndb = s3.collection('demo_s3kv')\n\nKEYS = list(range(1,100)) # 1-99\nURL = 'https://picsum.photos/300/300'\n\nc1,c2,c3 = st.columns(3)\n\nc1.header('Fetch Data')\nc2.header('Store Data')\nc3.header('Load Data')\n\nkey1 = c2.selectbox('Select Key', KEYS, key='key1', disabled='image1' not in ss)\nkey2 = c3.selectbox('Select Key', KEYS, key='key2')\n\nc1,c2,c3 = st.columns(3)\n\nwith c1:\n    if st.button('fetch random picture'):\n        resp = requests.get(URL)\n        ss['image1'] = resp.content\n        ss['url1'] = resp.url.split('?')[0]\n    if 'image1' in ss:\n        st.image(ss['image1'], use_column_width=True)\n\nwith c2:\n    if c2.button('save in DB', disabled='image1' not in ss):\n        db[str(key1)] = ss['image1']\n\nwith c3:\n    if c3.button('load from DB'):\n        img = db[str(key2)]\n        ss['image2'] = img\n    if 'image2' in ss and ss['image2']:\n        st.image(ss['image2'], use_column_width=True)\n```\n\n\n\n## Configuration examples\n\n\n\n##### secrets.toml\n\n```toml\n[connections.my_kv]\naccess_key = \"XXXXXXXXXXXXXXXXXXXX\"\nsecret_key = \"ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ\"\nendpoint   = \"https://sfo3.digitaloceanspaces.com\" # NOT REQUIRED ON AWS\nregion = \"sfo3\"\nbucket = \"my-bucket-name\"\nprefix = \"something/version-1\"  # NOT REQUIRED\npassword = \"my-secret-password\" # NOT REQUIRED\nsalt = \"crypto-salt\" # NOT REQUIRED\n\n[connections.my_kv2]\nmode = \"local\"\npath = \"/mnt/data/my-kv2-data\"\n```\n\n\n\n##### environmental variables\n\n```bash\nmy_kv_access_key = \"XXXXXXXXXXXXXXXXXXXX\"\nmy_kv_secret_key = \"ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ\"\nmy_kv_endpoint   = \"https://sfo3.digitaloceanspaces.com\"\nmy_kv_region = \"sfo3\"\nmy_kv_bucket = \"my-bucket-name\"\nmy_kv_prefix = \"something/version-1\"\nmy_kv_password = \"my-secret-password\"\nmy_kv_salt = \"crypto-salt\"\n\nmy_kv2_mode = \"local\"\nmy_kv2_path = \"/mnt/data/my-kv2-data\"\n```\n\n\n\n##### connection kwargs\n\n```python\n# override my_kv connection mode to test it offline / without S3\ns3 = st.experimental_connection('my_kv', type=S3KV, mode=\"dict\")\n\n# persistent local storage \ns3 = st.experimental_connection(None, type=S3KV, mode=\"local\", path=\"data\", password=\"xxx\")\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmobarski%2Fst_s3kv_connection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmobarski%2Fst_s3kv_connection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmobarski%2Fst_s3kv_connection/lists"}