{"id":21346895,"url":"https://github.com/queer/singyeong","last_synced_at":"2025-07-12T17:31:25.044Z","repository":{"id":35904253,"uuid":"149969065","full_name":"queer/singyeong","owner":"queer","description":"신경 - Cloud-native messaging/pubsub with powerful routing","archived":false,"fork":false,"pushed_at":"2023-02-27T12:01:32.000Z","size":3838,"stargazers_count":75,"open_issues_count":13,"forks_count":4,"subscribers_count":2,"default_branch":"mistress","last_synced_at":"2023-03-01T17:17:05.657Z","etag":null,"topics":["broker","elixir","hacktoberfest","heckin-cool-stuff-right-here","http-proxying","message-broker","message-bus","message-queue","metadata","metadata-query","microservice","microservices","pubsub","service-mesh","websockets"],"latest_commit_sha":null,"homepage":"https://discord.gg/aJSRXdd","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/queer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-09-23T09:12:59.000Z","updated_at":"2023-02-20T17:15:28.000Z","dependencies_parsed_at":"2023-02-14T16:35:13.643Z","dependency_job_id":null,"html_url":"https://github.com/queer/singyeong","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queer%2Fsingyeong","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queer%2Fsingyeong/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queer%2Fsingyeong/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queer%2Fsingyeong/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/queer","download_url":"https://codeload.github.com/queer/singyeong/tar.gz/refs/heads/mistress","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225829064,"owners_count":17530666,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["broker","elixir","hacktoberfest","heckin-cool-stuff-right-here","http-proxying","message-broker","message-bus","message-queue","metadata","metadata-query","microservice","microservices","pubsub","service-mesh","websockets"],"created_at":"2024-11-22T02:11:58.447Z","updated_at":"2024-11-22T02:11:58.963Z","avatar_url":"https://github.com/queer.png","language":"Elixir","funding_links":["https://patreon.com/amyware"],"categories":[],"sub_categories":[],"readme":"# 신경\n\n![Build status](https://github.com/queer/singyeong/workflows/Publish%20Docker/badge.svg) [![codecov](https://codecov.io/gh/queer/singyeong/branch/master/graph/badge.svg)](https://codecov.io/gh/queer/singyeong) [![Docker Hub](https://img.shields.io/badge/Docker%20Hub-queer%2Fsingyeong-%23007ec6)](https://hub.docker.com/r/queer/singyeong/tags) [![Dependabot Status](https://api.dependabot.com/badges/status?host=github\u0026repo=queer/singyeong)](https://dependabot.com)\n\n\u003e ## 신경\n\u003e [/ɕʰinɡjʌ̹ŋ/ • sin-gyeong](https://en.wiktionary.org/wiki/%EC%8B%A0%EA%B2%BD#Pronunciation) \u003csup\u003e\u003csup\u003e(try it with [IPA Reader](http://ipa-reader.xyz))\u003c/sup\u003e\u003c/sup\u003e\n\u003e 1. nerve\n\u003e\n\u003e ## Nerve\n\u003e /nərv/ • *noun*\n\u003e 1. (in the body) a whitish fiber or bundle of fibers that transmits impulses of sensation to the brain or spinal cord, and impulses from these to the muscles and organs.\n\u003e \n\u003e \u003csmall style=\"color:grey;\"\u003e\"the optic nerve\"\u003c/small\u003e\n\n신경 is the nerve-center of your microservices, providing a message bus + message\nqueue with powerful routing, automatic load-balancing + failover, powerful HTTP\nrequest proxying + routing, and more. 신경 aims to be simple to get started with\nwhile still providing the features for a variety of use-cases.\n\n신경 is a part of the [amyware Discord server](https://discord.gg/aJSRXdd).\n\nIf you like what I make, consider supporting me on Patreon:\n\n[\u003cimg src=\"https://i.imgur.com/YFjoCd1.png\" width=\"162\" height=\"38\" /\u003e](https://patreon.com/amyware)\n\n### Clients:\n\nLanguage   | Author                                             | Link                                              | Maintained?\n-----------|----------------------------------------------------|---------------------------------------------------|-------------\nElixir     | [@queer](https://queer.gg)                         | https://github.com/queer/singyeong-client-elixir  | yes\n.NET       | [@FiniteReality](https://github.com/FiniteReality) | https://github.com/finitereality/singyeong.net    | yes\nPython     | [@PanKlipcio](https://github.com/PanKlipcio)       | https://github.com/StartITBot/singyeong.py        | yes\nTypescript | [@cyyynthia](https://github.com/cyyynthia)         | https://github.com/borkenware/singyeongjs         | looks like yes (used in [squirrelchat](https://github.com/squirrelchat/squirrel)?)\n\n#### Unmaintained clients\n\nLanguage   | Author                                             | Link                                              | Notes\n-----------|----------------------------------------------------|---------------------------------------------------|------------------------------------\nPython     | [@PendragonLore](https://github.com/PendragonLore) | https://github.com/PendragonLore/shinkei          | Owner said it's unmaintained\nJava       | [@queer](https://queer.gg)                         | https://github.com/queer/singyeong-java-client    | I don't write much Java anymore S:\nTypescript | [@alula](https://github.com/alula)                 | https://github.com/KyokoBot/node-singyeong-client | Repo gone\n\n### Credit\n\n신경 was inspired by [sekitsui](https://gitlab.com/sekitsui), a project by the\n[Ayana](https://ayana.io) developers. 신경 was developed due to sekitsui\nseemingly having halted development (no release as far as I'm aware, no repo\nactivity in the last 1-2 years).\n\n## WARNING\n\n신경 is **ALPHA-QUALITY** software. The core functionality works, but there's\nno guarantee that it won't break, eat your cat, ... Use at your own risk!\n\n## Configuration\n\nConfiguration is done via environment variables, or via a custom configuration\nfile. See [`config.exs`](https://github.com/queer/singyeong/blob/master/config/config.exs)\nfor more information about config options.\n\n### Custom config files\n\nSometimes, it's necessary to include custom configuration files - such as for\nsomething that the environment variables don't cover. In such a case, you can\nadd a `custom.exs` file to `config/` that includes the custom configuration you\nwant / need.\n\nExample: Prove that custom config works:\n```Elixir\nuse Mix.Config\n\nIO.puts \"Loading some cool custom config :blobcatcooljazz:\"\n```\n\nExample: Always have debug-level logging, even in prod mode:\n```Elixir\nuse Mix.Config\n\nconfig :logger, level: :debug\n```\n\n## Plugins\n\nPlugins belong in a directory named `plugins` at the root directory. See the\n[plugin API](https://github.com/queer/singyeong_plugin) and the\n[example plugin](https://github.com/queer/singyeong-test-plugin) for more info.\n\n## Clustering\n\n신경 is capable of discovering cluster members automatically, using libcluster\nand the gossip strategy by default.\n\n## What exactly is it?\n\n신경 is a metadata-oriented message bus + message queue + HTTP proxy. Clients\nconnect over a [websocket](https://github.com/queer/singyeong/blob/master/PROTOCOL.md),\nand can send messages, queue messages, and send HTTP requests that can be\nrouted to clients based on client metadata.\n\n### Metadata-oriented?\n\n신경 clients are identified by three factors:\n\n1. Application id.\n2. Client id.\n3. Client metadata.\n\nWhen sending messages or HTTP requests over 신경, you do not choose a target\nservice instance directly, nor does 신경 choose for you. Rather, you specify a\ntarget application and a metadata query. 신경 will then run this query on all\nclients under the given application, and choose one that matches to receive the\nmessage or request.\n\nFor example, suppose you wanted to let users who had opted-in to a beta program\nuse beta features, but not all users. You could express this as a 신경 query,\nand say something like \"send this message to some service in the `backend`\napplication where `version_number \u003e= 2.0.0`.\"\n\nOf course, something like that is easy, but 신경 lets you do all sorts of\nthings easily. For example, suppose you had a cluster of websocket gateways\nthat users connected to and received events over. Instead of having to know\nwhich gateway a user is connected to, you could trivially express this as a\n신경 query - \"send this message to a `gateway` node that has `123` in its\n`connected_users` metadata.\" Importantly, **sending messages like this is done\nin exactly the same way as sending any other message.** 신경 tries to make it\nvery easy to express potentially-complicated routing with the same syntax as a\nsimple \"send to any one service in this application group.\"\n\n### Do I need to know exact client IDs to send messages?\n\nNo. You should not try to route to a specific 신경 client by id; instead you\nshould be expressing a metadata query that will send to the client you want.\nGenerally speaking, clients should be capable of running statelessly, or you\nshould use metadata to route messages effectively.\n\n### Do I need sidecar containers if I'm running in Kubernetes?\n\nNope.\n\n### Does it support clustering / multi-master / ...?\n\n신경 has masterless clustering support.\n\n## Why should I use this?\n\n- No need for Kubernetes or something similar - anything that can speak\n  websockets is a valid 신경 client.\n- No configuration. 신경 is meant to be \"drop in and get started\" - a few\n  options exist for things like authentication, but beyond that, no\n  configuration should be needed (at least to start out).\n- Fully dynamic. 신경 is meant to work well with clients randomly appearing and\n  disappearing (ex. browser clients when using 신경 as a websocket gateway).\n- No sidecars.\n- Choose where messages / requests are routed at runtime; no need to bake exact\n  targets into your application.\n- Service discovery without DNS.\n- Service discovery integrated into HTTP proxying / message sending.\n\n## Why should I NOT use this?\n\n- Query performance might be unacceptable.\n- Websockets might not be acceptable.\n- Development is still fairly early-stage; the alpha-ish quality of it may be\n  nonviable.\n\n## Why make this?\n\nMetadata-based routing is useful for all sorts of things:\n\n- Storing documentation about what messages your services send/recv, what REST\n  endpoints they expose, ... and querying on it to route to a service that\n  accepts a specific format of a specific message.\n- Sending a message to a subset of clients without doing a full pubsub and\n  relying on clients to drop them properly.\n- [Discord](https://discord.com) bot message-passing between services. No more\n  pubsub or whatever, just \"send this message to the service where\n  `123 in guild_ids`.\"\n- Websocket gateway. \"Send this message to the client where `user_id = 123`.\"\n- [Container scheduling](https://github.com/queer/pig).\n- [Monitoring host stats](https://github.com/queer/agma).\n- Routing messages to an audit-logging service and a handler service at the\n  same time.\n- Message queues that can only dispatch messages when a client is capable (ex.\n  \"dispatch this message from the queue to a client where `latency \u003c 10`\")\n- Anything you can think of!\n\nIn general, 신경 can get messages to the right place with some very complicated\nconditions very easily.\n\n### Why Elixir? Why not Go, Rust, Java, ...?\n\nI like Elixir :thumbsup: Elixir is well-suited to the use-case of\nmessage-queuing, and imo is a lot friendlier to write scalable messaging code\nin with little effort relative to any of the above-mentioned languages.\n\n### Why using Phoenix? Why not just use Cowboy directly?\n\nPhoenix's socket abstraction is really really useful. Also I didn't want to\nhave to build eg. HTTP routing from scratch; Phoenix does a great job of it\nalready so no need to reinvent the wheel. While it is possible to use Plug or a\nsimilar library on top of Cowboy or another HTTP server, I just liked the\nconvenience of getting it all out-of-the-box with Phoenix and being able to\nfocus on writing my application-level code instead of setting up a ton of weird\nplumbing.\n\n## How do I write my own client for it?\n\nCheck out [PROTOCOL.md](https://github.com/queer/singyeong/blob/master/PROTOCOL.md).\n\n## How does it work internally?\n\nThe code is intended to be pretty easy to read. The general direction that data\nflows is something like:\n\n```\nclient -\u003e decoder -\u003e gateway -\u003e dispatch         -\u003e process and dispatch response\n                             -\u003e identifier       -\u003e allow or reject connection\n                             -\u003e metadata updater -\u003e apply or reject metadata updates\n```\n\nIn terms of module structure, the way things go is something like:\n\n```\n|-\u003e \u003cphx/cowboy code\u003e\n|           |\n|           V\n|   SingyeongWeb.Transport.Raw\n|           |\n|           V\n|   Singyeong.Gateway\n|           |\n|           |------------------------------------|-----------------------------------------|\n|           V                                    V                                         V\n|   Singyeong.Gateway.Handler.Identify   Singyeong.Gateway.Handler.DispatchEvent   Singyeong.Gateway.Handler.Heartbeat\n|           |                                    |                                         |\n|           V                                    V                                         V\n|   Singyeong.Gateway                    Singyeong.Gateway.Dispatch                Singyeong.Store ---------|\n|           |                                    |                                                          |\n|           V                     |--------------|-----------------|-----------------------------|          |\n|-- SingyeongWeb.Transport.Raw    |              V                 |                             |          |\n            ^                     V      (METADATA_UPDATE)         V                             V          |\n            |        Singyeong.Gateway   Singyeong.Store   Singyeong.MessageDispatcher   Singyeong.Queue    |\n            |----------------|                                     |                             |          |\n            |                                                      |                             |          |\n            |------------------------------------------------------|-----------------------------|----------|\n```\n\nAdditionally, I aim to keep the server fairly small, ideally \u003c5k LoC, but\nabsolutely \u003c10kLoC no matter what.. At the time of writing, the server is\n~3.600 LoC:\n\n```\ngit:(master) X | -\u003e  tokei lib/\n===============================================================================\n Language            Files        Lines         Code     Comments       Blanks\n===============================================================================\n Elixir                 48         4558         3621          267          670\n===============================================================================\n Total                  48         4558         3621          267          670\n===============================================================================\ngit:(master) X | -\u003e\n```\n\n### How does querying / messaging work?\n\nWhen you send a message, the server inspects its target (ie. its metadata\nquery) and queries its own internal metadata store to find clients that can be\nrouted to that match said query. In the case of a multi-node setup, each node\nis only aware of its own clients, and thus metadata queries are a parallel RPC\nacross the cluster.\n\nA very basic query looks like this:\n\n```javascript\n{\n  \"application\": \"api\",\n  \"ops\": [],\n}\n```\n\nThat is, \"route this message to any client that's a member of the `api`\napplication.\" More-complex queries may do things like specifying constraints\nbased on latency:\n\n```javascript\n{\n  \"application\": \"api\",\n  \"ops\": [\n    {\n      \"path\": \"/latency/http\",\n      \"op\": \"$lte\",\n      \"to\": {\"value\": 100},\n    }\n  ],\n}\n```\n\nThat is, \"route this message to any client that's a member of the `api`\napplication, where the value at that application's `/latency/http` key is less\nthan or equal to `100`.\"\n\nAs metadata is arbitrary JSON -- the only restriction being that the top-level\nJSON is an object -- you can query effectively-infinitely-nested structures.\nYou can query paths with a syntax inspired by\n[JSON Patch](http://jsonpatch.com/).\n\nFor more about how the query language works, see\n[the relevant docs](https://github.com/queer/singyeong/blob/master/PROTOCOL.md#%EC%8B%A0%EA%B2%BD-node-queries)\n\n#### Delivery guarantees\n\nMessages are delivered at-least-zero times. That is, speaking broadly, your\nmessages will be delivered exactly once, but there are cases (network issues\netc.) that can lead to either no delivery, or more than one delivery, of the\nsame message.\n\n## What about test coverage and the like?\n\nYou can run tests with `mix test`. Note that the plugin tests **WILL FAIL**\nunless you set up the [test plugin](https://github.com/queer/singyeong-test-plugin)\nin `priv/test/plugin/`. If you don't want to deal with the plugin-related code\nwhen running tests (tho you REALLY should care...), you can skip those tests by\nrunning `mix test --exclude plugin`.\n\nNote that the HTTP proxying tests use an echo server I wrote (`echo.amy.gg`),\nrather than using a locally-hosted one. If you don't want to run these tests,\nset the `DISABLE_PROXY_TESTS` env var.\n\n## Security\n\nNote that **there is no ratelimit on authentication attempts**. This means that\na malicious client can constantly open connections and attempt passwords until\nit gets the correct one. It is **highly** recommended that you use a very long,\nprobably-very-complicated password in order to help protect against this sort\nof attack.\n\n## What is that name?\n\n신경 means nerve, and since the nervous system is how the entire body\ncommunicates, it seemed like a fitting name for a messaging system. I\nconsidered naming this something like 등뼈 (deungppyeo, \"spine\"/\"backbone\") or\n회로망 (hoelomang, \"network\") or even 별자리 (byeoljali, \"constellation), but I\nfigured that 신경 would be easier for people who don't know Korean to\npronounce, as well as being easier to find from GitHub search.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqueer%2Fsingyeong","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqueer%2Fsingyeong","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqueer%2Fsingyeong/lists"}