{"id":33179064,"url":"https://github.com/stanford-mast/pocket","last_synced_at":"2025-11-20T21:03:21.930Z","repository":{"id":36437662,"uuid":"138868146","full_name":"stanford-mast/pocket","owner":"stanford-mast","description":"Elastic ephemeral storage","archived":false,"fork":false,"pushed_at":"2022-04-12T21:57:25.000Z","size":29442,"stargazers_count":113,"open_issues_count":5,"forks_count":27,"subscribers_count":13,"default_branch":"master","last_synced_at":"2023-11-07T18:54:41.467Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stanford-mast.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-06-27T10:44:32.000Z","updated_at":"2023-10-25T19:18:53.000Z","dependencies_parsed_at":"2022-08-08T15:00:06.017Z","dependency_job_id":null,"html_url":"https://github.com/stanford-mast/pocket","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"purl":"pkg:github/stanford-mast/pocket","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stanford-mast%2Fpocket","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stanford-mast%2Fpocket/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stanford-mast%2Fpocket/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stanford-mast%2Fpocket/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stanford-mast","download_url":"https://codeload.github.com/stanford-mast/pocket/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stanford-mast%2Fpocket/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285511775,"owners_count":27184237,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-20T02:00:05.334Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-11-16T03:00:36.822Z","updated_at":"2025-11-20T21:03:21.924Z","avatar_url":"https://github.com/stanford-mast.png","language":"Java","funding_links":[],"categories":["大数据"],"sub_categories":[],"readme":"\u003cimg src=\"https://s3-us-west-2.amazonaws.com/pocket-logo/pocket-logo.jpg\" width=\"250\"\u003e\n\nPocket is a storage system designed for ephemeral data sharing. Pocket provides fast, distributed and elastic storage for data with low durability requirements.\nThe key properties of Pocket are:\n\n* high throughput and low latency for a wide range of object sizes\n* automatic resource scaling and rightsizing\n* intelligent data placement across multiple storage tiers\n\t\nPocket offers a *serverless* abstraction to storage, meaning users do not manually configure or manage storage servers. A key use-case for Pocket is ephemeral data sharing in serverless analytics, when serverless tasks need to exchange intermediate data between execution stages. \n\n## Building and Deploying Pocket from Source\n\nTo build Pocket from source execute the following steps:\n```\ncd client \n./build.sh \t\t\t# builds libcppcrail.so and libpocket.so\ncd ../pocket-core\nmvn -DskipTests install \t# builds crail-1.0-bin.tar.gz\n```\n\nFor deployment, copy the shared libraries libcppcrail.so and libpocket.so together with the python scripts pocket.py, controller.py and pocket_metadata_cmds.py to one directory, and un-tar crail-1.0-bin.tar.gz into a different directory. pocket.py is the client interface applications use to store and share ephemeral data between function invocations. controller.py is the Pocket controller responsible for managing storage resources in an elastic manner. crail-1.0-bin contains the server side of the Pocket storage platform. \n\n## Running Pocket\n\nFollow the instructions in the [deploy README](https://github.com/stanford-mast/pocket/blob/master/deploy/README.md) to set up and run a Pocket deployment on Amazon Web Services. \n\nFollow the instructions in the [microbenchmark README](https://github.com/stanford-mast/pocket/tree/master/microbenchmark) to run a simple example test on AWS Lambda that measures Pocket PUT, GET and metadata lookup latency.\n\n\n## Pocket Design \n\nPocket divides responsibility across a control plane, metadata plane and data plane, which can each be scaled indepdently across nodes. Pocket has a logically centralized controller which decides how to distribute data for incoming application jobs. The meatdata plane routes client requests and keeps track of data stored across nodes in the data plane.\n\nPocket is designed for sub-second response times. I/O operations are kept deliberatly simple and the storage tier implementation leverages low-latency, high-throughput user-level network and storage processing stacks. The controller monitors per-second cluster resource utilization across multiple dimensions (compute, network bandwidth and storage capacity) and scales cluster resource allocations to match application requirements. The controller allocates resource with the primary goal of providing high I/O performance to jobs. The secondary goals is to minimizing the resource cost. \n\nPocket uses multiple storage media (DRAM, Flash and HDD) to balance performance and cost, storing an application's data on the lowest cost storage media that still satisfies the application's I/O demands.\n\n\n## Implementation\n\nWe leverage several open source projects to implement Pocket. Pocket's metadata and data planes are based on [Apache Crail](http://crail.io). The NVMe Flash and block  storage tiers are based on [ReFlex](https://github.com/stanford-mast/reflex). We deploy Pocket metadata and data nodes in [Docker](https://www.docker.com/) containers. Pocket's controller uses the [Kubernetes](https://kubernetes.io) container orchestration system to elastically scale the cluster based on dynamic application demands.\n\n\n## Pocket repository structure\n\n* **deploy**: deployment scripts and instructions to run Pocket with Kubernetes on AWS EC2\n* **controller**: control plane logic for automatic resource scaling \n* **client**: python client library for Pocket, based on the Crail C++ client\n* **dockerfiles**: build new Pocket container images if you modify Pocket source code\n* **microbenchmark**: a simple lambda latency test for Pocket\n* **pocket-core**: source code for Pocket metadata and data plane implementation, based on Apache Crail and ReFlex\n\n## Reference\n\nPlease refer to the Pocket OSDI'18 paper: \n\nAna Klimovic, Yawen Wang, Patrick Stuedi, Animesh Trivedi, Jonas Pfefferle, and Christos\nKozyrakis. [Pocket: Elastic Ephemeral Storage for Serverless\nAnalytics.](https://www.usenix.org/system/files/osdi18-klimovic.pdf) In *Proceedings of the\nUSENIX Symposium on Operating Systems Design and Implementation* (OSDI), 2018.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstanford-mast%2Fpocket","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstanford-mast%2Fpocket","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstanford-mast%2Fpocket/lists"}