{"id":17667479,"url":"https://github.com/josepsampe/storage-functions","last_synced_at":"2025-07-09T05:39:54.789Z","repository":{"id":104619794,"uuid":"88100635","full_name":"JosepSampe/storage-functions","owner":"JosepSampe","description":"Lightweight Serverless Functions Framework for OpenStack Swift","archived":false,"fork":false,"pushed_at":"2020-07-10T14:56:30.000Z","size":43094,"stargazers_count":11,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-08T04:01:58.809Z","etag":null,"topics":["functions","middleware","openstack-swift","serverless-framework"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JosepSampe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-12T22:12:09.000Z","updated_at":"2023-02-13T03:48:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"5c61d727-8dbc-4bf3-837f-43a51289d14b","html_url":"https://github.com/JosepSampe/storage-functions","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/JosepSampe/storage-functions","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosepSampe%2Fstorage-functions","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosepSampe%2Fstorage-functions/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosepSampe%2Fstorage-functions/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosepSampe%2Fstorage-functions/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JosepSampe","download_url":"https://codeload.github.com/JosepSampe/storage-functions/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosepSampe%2Fstorage-functions/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264402445,"owners_count":23602594,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["functions","middleware","openstack-swift","serverless-framework"],"created_at":"2024-10-23T22:07:30.797Z","updated_at":"2025-07-09T05:39:54.768Z","avatar_url":"https://github.com/JosepSampe.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e \u003cimg src=\"docs/img/zion.png\" width=\"250\"\u003e\u003c/p\u003e\n\nZion is a Serverless Computing Framework for OpenStack Swift that allows to run storage functions directly on the data. It aims to solve the scalability and resource contention problems of active storage, while benefiting from data locality to reduce latency by placing computations close to the data. Our model is data-driven and not event-driven, because ZIon computations are located in the data pipeline, and intercept the data flows that arrive and return from the object store.\n\n\n## Architecture\n![Architecture](docs/img/architecture.png?raw=true \"Architecture\")\n\n### Interception Software and Metadata Service\nWe built a new Swift interception middleware for Zion to accomplish two primary tasks: 1. The management of function code deployment and libraries, including the triggers that cause the functions to be run; and 2. Redirection of requests and responses through the computation layer when they need to be processed by any function\n\n### Computation layer\n\nIn Zion, the computation layer is composed by a pool of compute nodes. They are located between the proxies and the storage nodes. In a compute node, each function is run inside a separate Docker container, what is called a *worker*. Zion’s runtime is Java-based, and consequently, every function is run in a Java Virtual Machine (JVM). At a very high level, a worker can be viewed as a container running a specific function. Every new invocation to the function is handled by a new thread inside the JVM of a worker. This means that a single worker can handle more than one request at a time. By default, each worker uses 1 exclusive CPU, then a service called *Zion Service Manager* is responsible to monitor the function workers and scale them up and down depending of the CPU usage.\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"docs/img/compute_node.png\"\u003e\u003c/p\u003e\n\n## Installation\n\n### All-In-One Machine\nFor testing purposes, it is possible to install an All-In-One (AiO) machine with all the Zion components and requirements.\nWe prepared a script for automating this task. The requirements of the machine are a clean installation of **Ubuntu Server 20.04**, **2CPU Cores**, at least **2GB** of RAM, and a **fixed IP address**. It is preferable to upgrade the system to the latest versions of the packages with `apt update \u0026\u0026 apt dist-upgrade` before starting the installation, and set the server name as `controller` in the `/etc/hostname` file. Then, download the `aio_u20_ussuri.sh` script and run it as sudo:\n\n```bash\ncurl -fsSL https://git.io/JJq4t | sudo bash /dev/stdin install aio\n```\n\nThe script first installs Keystone, Swift and Horizon (Ussuri release), then it proceeds to install the Zion framework package. Note that the script uses weak passwords for the installed services. If you want more secure services, please change them at the top of the script.\n\nBy default, the script has low verbosity. To see the full installation log, run the following command in another terminal:\n\n```bash\ntail -f /tmp/zion_aio_installation.log\n```\n\nThe script takes long to complete (~10 minutes) (it depends of the network connection). Once completed, you can access to the dashboard by typing the following URL in the web browser: `http://\u003cnode-ip\u003e/horizon`.\n\nIf you already ran the installation script, you can update the Zion framework from this repository by the following command:\n\n```bash\ncurl -fsSL https://git.io/JJq4t | sudo bash /dev/stdin update aio\n```\n\n## Verify\n### Test Swift\nTo verify the correct operation of the Swift installation, follow these steps:\n\n1- Load credentials:\n```bash\nsource zion-openrc\n```\n\n2- Create data bucket:\n```bash\nswift post data\n```\n\n3- Create new .json file and upload it to data bucket:\n```bash\necho test \u003e test.json\nswift upload data test.json\n```\n\n4- Test if you can download the .json file:\n```bash\nswift download data test.json\nor\ncurl -H \"X-Auth-Token:$TOKEN\" $STORAGE_URL/data/test.json\n```\n\n\n### Test Zion\nNavigate into [Function Samples](Function%20Samples/java), and compile and deploy the *NoopDataIterator* storage function by using the [deployment script](Utils/deploy_function.py).\n\n1- Assign the No-operation storage function to the .json file upon GET requests:\n```bash\ncurl -H \"X-Auth-Token:$TOKEN\" $STORAGE_URL/data/test.json -X POST -H \"X-Function-onGet:noop.tar.gz\"\n```\n\n2- Download the .json file that will put into execution the storage function:\n```bash\nswift download data test.json\nor\ncurl -H \"X-Auth-Token:$TOKEN\" $STORAGE_URL/data/test.json\n```\n\n3- Delete the No-operation storage function:\n```bash\ncurl -H \"X-Auth-Token:$TOKEN\" $STORAGE_URL/data/test.json -X POST -H \"X-Function-onGet-Delete:noop.tar.gz\"\n```\n\nNavigate into [Function Samples](Function%20Samples/java), and compile and deploy the *Counter* storage function by using the [deployment script](Utils/deploy_function.py).\n\n4- Assign the Counter storage function to the .json file upon GET requests:\n```bash\ncurl -H \"X-Auth-Token:$TOKEN\" $STORAGE_URL/data/test.json -X POST -H \"X-Function-onGet:access-counter.tar.gz\"\n```\n\n5- Download the .json file that will put into execution the storage function:\n```bash\nswift download data test.json\nor\ncurl -H \"X-Auth-Token:$TOKEN\" $STORAGE_URL/data/test.json\n```\n\n6- The Counter storage function adds into the object metadata an access counter and the last access timestamp. Verify the correct execution of the storage function by running the following command:\n```bash\nswift stat data test.json\n```\n\n## See also\n\n\"[Data-driven Serverless Functions for Object Storage](https://dl.acm.org/doi/abs/10.1145/3135974.3135980)\" by Josep Sampé, Marc Sánchez-Artigas, Pedro García-López and Gerard París. Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference (Middleware 17), December 2017, Las Vegas, Nevada.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjosepsampe%2Fstorage-functions","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjosepsampe%2Fstorage-functions","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjosepsampe%2Fstorage-functions/lists"}