{"id":18470443,"url":"https://github.com/ds2-lab/infinistore","last_synced_at":"2026-04-01T21:50:20.672Z","repository":{"id":90303493,"uuid":"606916605","full_name":"ds2-lab/infinistore","owner":"ds2-lab","description":"InfiniStore: an elastic serverless cloud storage system (VLDB'23)","archived":false,"fork":false,"pushed_at":"2023-05-05T21:53:24.000Z","size":133939,"stargazers_count":24,"open_issues_count":0,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2026-03-27T13:44:11.650Z","etag":null,"topics":["faas","serverless","storage"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ds2-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-26T23:11:20.000Z","updated_at":"2025-08-21T15:19:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"3933a4d0-d51d-4d34-ba81-7e575fe36f50","html_url":"https://github.com/ds2-lab/infinistore","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/ds2-lab/infinistore","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2Finfinistore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2Finfinistore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2Finfinistore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2Finfinistore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ds2-lab","download_url":"https://codeload.github.com/ds2-lab/infinistore/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2Finfinistore/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31046896,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-27T09:35:52.079Z","status":"ssl_error","status_checked_at":"2026-03-27T09:35:20.916Z","response_time":164,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["faas","serverless","storage"],"created_at":"2024-11-06T10:13:59.521Z","updated_at":"2026-04-01T21:50:20.651Z","avatar_url":"https://github.com/ds2-lab.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# InfiniStore\n\n![Logo](https://github.com/ds2-lab/ds2-lab.github.io/blob/master/docs/images/infinistore_logo.png)\n\n**InfiniStore** is an elastic, cost-effective, and high-performance object storage built atop ephemeral cloud funtions. Built on top of the [InfiniCache](https://ds2-lab.github.io/infinicache/) codebase, InfiniStore offers automatic elasticity, durability, strong consistency, and high performance.\n\nThe preprint of our VLDB'23 paper can be viewed at: [InfiniStore: Elastic Serverless Cloud Storage](https://arxiv.org/abs/2209.01496).\n\n## Prepare\n\n- ### EC2 Proxy\n\n  Amazon EC2 AMI: ubuntu-18.04\n\n  Golang version: 1.18\n\n  Be sure the port **6378 - 6379** is avaiable on the proxy\n\n  We recommend that EC2 proxy and Lambda functions are under the same VPC network, and deploy InfiniStore proxy on EC2 instances with high bandwidth (`c5n` family maybe a good choice).\n\n- ### Golang install\n\n  Jump to [install_go.md](https://github.com/ds2-lab/infinistore/blob/master/install_go.md)\n\n- ### Package install\n\n  Install basic package\n  ```shell\n  sudo apt-get update\n  sudo apt-get -y upgrade\n  sudo apt install awscli\n  sudo apt install zip\n  ```\n\n  Clone this repo\n  ```shell\n  git clone https://github.com/ds2-lab/infinistore.git\n  ```\n\n  Run `aws configure` to setup your AWS credential.\n  ```shell\n  aws configure\n  ```\n\n- ### Lambda Runtime\n\n  #### Lambda Role setup\n\n  Go to AWS IAM console and create a role for the Lambda memory node (Lambda function).\n\n  AWS IAM console -\u003e Roles -\u003e Create Role -\u003e Lambda -\u003e\n\n  **`AWSLambdaFullAccess, `**\n\n  **`AWSLambdaVPCAccessExecutionRole, `**\n\n  **`AWSLambdaENIManagementAccess`**\n\n  #### Enable Lambda Internet Access under VPC\n\n  \n\n- ### S3\n\n  Create the S3 bucket to store the zip file of the Lambda code and data output from Lambda functions. Remember the name of this bucket for the configuration in next step.\n\n- ### Configuration\n\n  #### Lambda Function Configuration and Creation\n\n  Edit the aws settings and the VPC configuration in `deploy/config.yml`. If you do not want to run InfiniStore under VPC, skip `aws-vpc-subnets` and `aws-security-groups` settings. A valid configuration will like:\n  ```yml\n  region: \"us-east-1\"\n  aws-iam-role: \"arn:aws:iam::1234567890:role/jzhang33\"\n  aws-vpc-subnets: \"subnet-12345abcdef,subnet-67890abcdef\"\n  aws-security-groups: \"sg-abcdef0987654321\"\n  s3-bucket-cos: \"infinistore.cos\"    # S3 bucket for InfiniStore COS layer.\n  s3-bucket-data: \"infinistore.data\"  # Optional. S3 bucket for collecting data required for reproducibility experiments.\n  ```\n\n  Edit `deploy/create_function.sh` and `deploy/update_function.sh`\n  ```shell\n  DEPLOY_PREFIX=\"your lambda function prefix\"\n  DEPLOY_CLUSTER=1000   # The number of Lambda deployments used for window rotation.\n  DEPLOY_MEM=1536       # The memory of Lambda deployments.\n  S3=\"your bucket name\" # S3 bucket for uploading the binary of AWS Lambda functions.\n  ```\n\n  Run script to create and deploy lambda functions (Also, if you do not want to run InfiniStore under VPC, you need to remove the `--no-vpc` flag on executing `deploy/create_function.sh`).\n\n  ```shell\n  go get\n  deploy/create_function.sh --no-vpc 600\n  ```\n\n  If lambda functions are deployed in VPC, create NAT gateway to give Lambdas access to the proxy. You may use AWS console or `create_nat.sh`. Besure to change the settings as described in the script before executing. Plese [refer to this article](https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/).\n\n  ```shell\n  deploy/create_nat.sh\n  ```\n\n  #### Proxy Configuration\n\n  Edit `proxy/config/config.go`, change the aws region, cluster size, and prefix of the Lambda functions.\n  ```go\n  const AWSRegion = \"us-east-1\"\n  const NumLambdaClusters = 1000\n  const LambdaPrefix = \"Your Lambda Function Prefix\"\n  const ServerPublicIp = \"\"  // Leave it empty if using VPC.\n  ```\n\n## Execution\n\n- Proxy server\n\n  Run `make start` for quick starting the proxy server. The proxy will show a console dashboard to track Lambda invocation. Logs by default is in file `log` in the same folder. `debug` option is available for more detail logs.\n\n  ```bash\n  make start [\"PARAMS=--debug\"]\n  ```\n\n  To stop proxy server, press `ctrl+c` or `q` if dashboard is displayed, otherwise, run `make stop`. If `make stop` were not working, you could use `pgrep proxy`, `pgrep go`, or check `/tmp/infinistore.pid` to find the pid and kill the proxy.\n\n- Client library\n\n  The toy demo for Client Library\n\n  ```bash\n  go run client/example/main.go\n  ```\n\n  The result should be\n\n  ```bash\n  ~$ go run client/example/main.go\n  2023/02/26 23:07:48 EcRedis Set foo 15 160250633\n  2023/02/26 23:07:48 EcRedis Got foo 15 28614169 ( 28574763 34758 )\n  GET foo:Hello infinity!(28.698376ms)\n  ```\n\n- Stand-alone local simulation\n\n  On Mac, InfiniStore can be run in local simulation mode, which will initiate local processes as cloud functions.\n\n  Enable local function execution by editing `lambda/config.go`:\n  ```go\n  S3_BACKUP_BUCKET = \"your S3 bucket%s\"  // Leave %s at the end your S3 bucket.\n  DRY_RUN = true\n  ```\n\n  Run `make start-local` to start a stand-alone local proxy server, which will invoke functions locally to simulation Lambda execution.\n\n  Run `make test` to put/get a toy object.\n\n\n\n## Related repos\n\nBenchmark tool and workload replayer [infinibench](https://github.com/ds2-lab/infinibench).\n\nRESP (REdis Serialization Protocol) library [redeo](https://github.com/mason-leap-lab/redeo).   \n\n\n\n## To cite InfiniStore\n\n```\n@inproceedings {vldb23-infinistore,\n  author       = {Jingyuan Zhang and Ao Wang and Xiaolong Ma and Benjamin Carver and Nicholas John Newman and Ali Anwar and Lukas Rupprecht and Dimitrios Skourtis and Vasily Tarasov and Feng Yan and Yue Cheng},\n  title        = {InfiniStore: Elastic Serverless Cloud Storage},\n  journal      = {Proc. {VLDB} Endow.},\n  volume       = {16},\n  number       = {7},\n  pages        = {1629--1642},\n  year         = {2023},\n  url          = {https://www.vldb.org/pvldb/vol16/p1629-zhang.pdf},\n}\n\n```\n\n\n## Contributing\n\nPlease feel free to hack on InfiniStore! We're happy to accept contributions.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fds2-lab%2Finfinistore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fds2-lab%2Finfinistore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fds2-lab%2Finfinistore/lists"}