{"id":13906358,"url":"https://github.com/twitter-archive/gizzard","last_synced_at":"2025-07-18T04:31:07.171Z","repository":{"id":861913,"uuid":"597840","full_name":"twitter-archive/gizzard","owner":"twitter-archive","description":"[Archived] A flexible sharding framework for creating eventually-consistent distributed datastores","archived":true,"fork":false,"pushed_at":"2017-03-16T23:21:54.000Z","size":2736,"stargazers_count":2254,"open_issues_count":11,"forks_count":211,"subscribers_count":242,"default_branch":"master","last_synced_at":"2024-08-07T23:47:50.254Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/twitter-archive.png","metadata":{"files":{"readme":"README.markdown","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2010-04-06T21:13:26.000Z","updated_at":"2024-07-19T19:40:49.000Z","dependencies_parsed_at":"2022-08-06T10:01:11.997Z","dependency_job_id":null,"html_url":"https://github.com/twitter-archive/gizzard","commit_stats":null,"previous_names":[],"tags_count":45,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twitter-archive%2Fgizzard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twitter-archive%2Fgizzard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twitter-archive%2Fgizzard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twitter-archive%2Fgizzard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/twitter-archive","download_url":"https://codeload.github.com/twitter-archive/gizzard/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226344618,"owners_count":17610176,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-06T23:01:34.114Z","updated_at":"2024-11-25T14:31:22.378Z","avatar_url":"https://github.com/twitter-archive.png","language":"Scala","funding_links":[],"categories":["Scala","数据库中间件"],"sub_categories":[],"readme":"# STATUS\n\nTwitter is no longer maintaining this project or responding to issues or PRs.\n\n## Gizzard: a library for creating distributed datastores\n\nCheck out [Using gizzard](http://github.com/twitter/gizzard/blob/master/doc/using.md)\nfor details on requirements, how to build gizzard, and a demo app.\n\nAlso check out [the gizzard mailing list](http://groups.google.com/group/gizzard).\n\n### An introduction to sharding\n\nMany modern web sites need fast access to an amount of information so large\nthat it cannot be efficiently stored on a single computer. A good way to deal\nwith this problem is to “shard” that information; that is, store it across\nmultiple computers instead of on just one.\n\nSharding strategies often involve two techniques: partitioning and\nreplication. With *partitioning*, the data is divided into small chunks and\nstored across many computers. Each of these chunks is small enough that the\ncomputer that stores it can efficiently manipulate and query the data. With\nthe other technique of *replication*, multiple copies of the data are stored\nacross several machines. Since each copy runs on its own machine and can\nrespond to queries, the system can efficiently respond to tons of queries for\nthe same data by adding more copies. Replication also makes the system\nresilient to failure because if any one copy is broken or corrupt, the system\ncan use another copy for the same task.\n\nThe problem is: sharding is difficult. Determining smart partitioning schemes\nfor particular kinds of data requires a lot of thought. And even more\ndifficult is ensuring that all of the copies of the data are *consistent*\ndespite unreliable communication and occasional computer failures. Recently, a\nlot of open-source distributed databases have emerged to help solve this\nproblem. Unfortunately, as of the time of writing, most of the available\nopen-source projects are either too immature or too limited to deal with the\nvariety of problems that exist on the web. These new databases are hugely\npromising but for now it is sometimes more practical to build a custom\nsolution.\n\n### What is a sharding framework?\n\nTwitter has built several custom distributed data-stores. Many of these\nsolutions have a lot in common, prompting us to extract the commonalities so\nthat they would be more easily maintainable and reusable. Thus, we have\nextracted Gizzard, a Scala framework that makes it easy to create custom\nfault-tolerant, distributed databases.\n\nGizzard is a *framework* in that it offers a basic template for solving a\ncertain class of problem. This template is not perfect for everyone’s needs\nbut is useful for a wide variety of data storage problems. At a high level,\nGizzard is a middleware networking service that manages partitioning data\nacross arbitrary backend datastores (e.g., SQL databases, Lucene, etc.). The\npartitioning rules are stored in a forwarding table that maps key ranges to\npartitions. Each partition manages its own replication through a declarative\nreplication tree. Gizzard supports “migrations” (for example, elastically\nadding machines to the cluster) and gracefully handles failures. The system is\nmade *eventually consistent* by requiring that all write-operations are\nidempotent *and* commutative and as operations fail (because of, e.g., a network\npartition) they are retried at a later time.\n\nA very simple sample use of Gizzard is [Rowz](http://github.com/twitter/Rowz),\na distributed key-value store. To get up-and-running with Gizzard quickly,\nclone Rowz and start customizing!\n\nBut first, let’s examine how Gizzard works in more detail.\n\n### How does it work?\n\n#### Gizzard is middleware\n\n![diagram](http://github.com/twitter/gizzard/raw/master/doc/middleware.png?raw=true)\n\nGizzard operates as a *middleware* networking service. It sits “in the middle”\nbetween clients (typically, web front-ends like PHP and Ruby on Rails\napplications) and the many partitions and replicas of data. Sitting in the\nmiddle, all data querying and manipulation flow through Gizzard. Gizzard\ninstances are stateless so run as many gizzards as are necessary to sustain\nthroughput or manage TCP connection limits. Gizzard, in part because it is\nruns on the JVM, is quite efficient. One of Twitter’s Gizzard applications\n(FlockDB, our distributed graph database) can serve 10,000 queries per second\nper commodity machine. But your mileage may vary.\n\n#### Gizzard supports any datastorage backend\n\nGizzard is designed to replicate data across any network-available data\nstorage service. This could be a relational database, Lucene, Redis, or\nanything you can imagine. As a general rule, Gizzard requires that all write\noperations be idempotent *and* commutative (see the section on Fault Tolerance\nand Migrations), so this places some constraints on how you may use the\nback-end store. In particular, Gizzard does not guarantee that write\noperations are applied in order. It is therefore imperative that the system is\ndesigned to reach a consistent state regardless of the order in which writes\nare applied.\n\n#### Gizzard handles partitioning through a forwarding table\n\nGizzard handles partitioning (i.e., dividing exclusive ranges of data across\nmany hosts) by mappings *ranges* of data to particular shards. These mappings\nare stored in a forwarding table that specifies lower-bound of a numerical\nrange and what shard that data in that range belongs to.\n\n![forwarding table](http://github.com/twitter/gizzard/raw/master/doc/forwarding_table.png)\n\nTo be precise, you provide Gizzard a hash function that, given a key for your\ndata (and this key can be application specific), produces a number that\nbelongs to one of the ranges in the forwarding table. These functions are\nprogrammable so you can optimize for locality or balance depending on your\nneeds.\n\nThis range-based approach differs from the \"consistent hashing\" technique used\nin many other distributed data-stores. This allows for heterogeneously sized\npartitions so that you easily manage *hotspots*, segments of data that are\nextremely popular. In fact, Gizzard does allows you to implement completely\ncustom forwarding strategies like consistent hashing, but this isn't the\nrecommended approach. For some more detail on partitioning schemes, [read\nwikipedia](http://en.wikipedia.org/wiki/Partition_(database)):\n\n#### Gizzard handles replication through a replication tree\n\nEach shard referenced in the forwarding table can be either a physical shard\nor a logical shard. A physical shard is a reference to a particular data\nstorage back-end, such as a SQL database. In contrast, A *logical shard* is\njust a tree of other shards, where each *branch* in the tree represents some\nlogical transformation on the data, and each *node* is a data-storage\nback-end. These logical transformations at the branches are usually rules\nabout how to propagate read and write operations to the children of that\nbranch. For example, here is a two-level replication tree. Note that this\nrepresents just ONE partition (as referenced in the forwarding table):\n\n![Alt text](http://github.com/twitter/gizzard/raw/master/doc/replication_tree.png)\n\nThe “Replicate” branches in the figure are simple strategies to repeat write\noperations to all children and to balance reads across the children according\nto health and a weighting function. You can create custom branching/logical\nshards for your particular data storage needs, such as to add additional\ntransaction/coordination primitives or quorum strategies. But Gizzard ships\nwith a few standard strategies of broad utility such as Replicating,\nWrite-Only, Read-Only, and Blocked (allowing neither reads nor writes). The\nutility of some of the more obscure shard types is discussed in the section on\n`Migrations`.\n\nThe exact nature of the replication topologies can vary per partition. This\nmeans you can have a higher replication level for a “hotter” partition and a\nlower replication level for a “cooler” one. This makes the system highly\nconfigurable. For instance, you can specify that the back-ends mirror one\nanother in a primary-secondary-tertiary-etc. configuration for simplicity.\nAlternatively, for better fault tolerance (but higher complexity) you can\n“stripe” partitions across machines so that no machine is a mirror of any\nother.\n\n#### Gizzard is fault-tolerant\n\nFault-tolerance is one of the biggest concerns of distributed systems. Because\nsuch systems involve many computers, there is some likelihood that one (or\nmany) are malfunctioning at any moment. Gizzard is designed to avoid any\nsingle points of failure. If a certain replica in a partition has crashed,\nGizzard routes requests to the remaining healthy replicas, bearing in mind the\nweighting function. If all replicas of in a partition are unavailable, Gizzard\nwill be unable to serve read requests to that shard, but all other shards will\nbe unaffected. Writes to an unavailable shard are buffered until the shard\nagain becomes available.\n\nIn fact, if any number of replicas in a shard are unavailable, Gizzard will\ntry to write to all healthy replicas as quickly as possible and buffer the\nwrites to the unavailable shard, to try again later when the unhealthy shard\nreturns to life. The basic strategy is that all writes are materialized to a\ndurable, transactional journal. Writes are then performed asynchronously (but\nwith manageably low latency) to all replicas in a shard. If a shard is\nunavailable, the write operation goes into an error queue and is retried\nlater.\n\nIn order to achieve “eventual consistency”, this “retry later” strategy\nrequires that your write operations are idempotent *and* commutative. This is\nbecause a retry later strategy can apply operations out-of-order (as, for\ninstance, when newer jobs are applied before older failed jobs are retried).\nIn most cases this is an easy requirement. A demonstration of commutative,\nidempotent writes is given in the Gizzard demo app,\n[Rowz](http://github.com/twitter/Rowz).\n\n#### Winged migrations\n\nIt’s sometimes convenient to copy or move data from shards from one computer\nto another. You might do this to balance load across more or fewer machines,\nor to deal with hardware failures. It’s interesting to explain some aspect of\nhow migrations work just to illustrate some of the more obscure logical shard\ntypes. When migrating from `Datastore A` to `Datastore A'`, a Replicating\nshard is set up between them but a WriteOnly shard is placed in front of\n`Datastore A'`. Then data is copied from the old shard to the new shard. The\nWriteOnly shard ensures that while the new Shard is bootstrapping, no data is\nread from it (because it has an incomplete picture of the corpus).\n\n![Alt text](http://github.com/twitter/gizzard/raw/master/doc/migration.png)\n\nBecause writes will happen out of order (new writes occur before older ones\nand some writes may happen twice), all writes must be idempotent and\ncommutative to ensure data consistency.\n\n#### How does Gizzard handle write conflicts?\n\nWrite conflicts are when two manipulations to the same record try to change\nthe record in differing ways. Because Gizzard does not guarantee that\noperations will apply in order, it is important to think about write conflicts\nwhen modeling your data. As described elsewhere, write operations must be both\nidempotent and commutative in order to avoid conflicts. This is actually an\n*easy* requirement in many cases, way easier than trying to guarantee ordered\ndelivery of messages with bounded latency and high availability. As mentioned\nabove, Rowz illustrates a technique of using time-stamps to only apply\noperations that are \"newer\". More documentation on this will be forthcoming.\n\n### Contributors\n\n* Robey Pointer\n* Nick Kallen\n* Ed Ceaser\n* John Kalucki\n* Matt Freels\n* Kyle Maxwell\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwitter-archive%2Fgizzard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftwitter-archive%2Fgizzard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwitter-archive%2Fgizzard/lists"}