{"id":18562472,"url":"https://github.com/replit/rfbproxy","last_synced_at":"2025-04-10T03:31:55.063Z","repository":{"id":39586735,"uuid":"341072697","full_name":"replit/rfbproxy","owner":"replit","description":"An RFB proxy that enables WebSockets and audio.","archived":false,"fork":false,"pushed_at":"2025-02-18T14:56:20.000Z","size":245,"stargazers_count":22,"open_issues_count":2,"forks_count":8,"subscribers_count":41,"default_branch":"main","last_synced_at":"2025-03-24T15:52:18.415Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Nix","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/replit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-22T03:50:18.000Z","updated_at":"2025-02-26T02:59:27.000Z","dependencies_parsed_at":"2024-09-13T06:18:10.107Z","dependency_job_id":"8c9b0d86-8c0e-4eae-8489-4ad0d1c133bc","html_url":"https://github.com/replit/rfbproxy","commit_stats":null,"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replit%2Frfbproxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replit%2Frfbproxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replit%2Frfbproxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replit%2Frfbproxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/replit","download_url":"https://codeload.github.com/replit/rfbproxy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248151079,"owners_count":21056020,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T22:09:52.137Z","updated_at":"2025-04-10T03:31:50.054Z","avatar_url":"https://github.com/replit.png","language":"Nix","readme":"# rfbproxy\n\nAn RFB proxy that enables WebSockets and audio.\n\nThis crate proxies a TCP Remote Framebuffer server connection and exposes a\nWebSocket endpoint, translating the connection between them. It can optionally\nenable audio using the [Replit Audio RFB extension](#replit-audio-rfb-extension) if the\n`--enable-audio` flag is passed or the `VNC_ENABLE_EXPERIMENTAL_AUDIO`\nenvironment variable is set to a non-empty value.\n\nNote that since this project only supports RFB-over-WebSockets\nthe only VNC client that supports that particular combination and\nthe only one that will work with it is [noVNC](https://novnc.com).\n\n# Running\n\nSince this is a proxy, you'll need to have an RFB server running\nalready. [TigerVNC](https://tigervnc.org/) is a good option:\n\n```shell\nXvnc --SecurityTypes={None,VNCAuth} --rfbport=5901 --localhost :1\n```\n\nNow `rfbproxy` can run:\n\n```shell\ncargo run -- [--enable-audio] [--address=0.0.0.0:5900] [--rfb-server=127.0.0.1:5901]\n```\n\n# Replit Audio RFB extension\n\nThis uses a proposed extension to the [RFB\nprotocol](https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst) in\norder to negotiate and transmit encoded audio. This is the main difference from\nthe pre-existing [QEMU Audio\nmessages](https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#qemu-audio-client-message).\n\n## Encodings\n\nThis registers the following pseudo-encodings:\n\n| Number     | Name                          |\n|----------- | ------------------------------|\n| 0x52706C41 | Replit Audio Pseudo-encoding |\n\nA client that supports this encoding is indicating that it is able to receive\nan encoded audio data stream. If a server wishes to send encoded audio data, it\nwill send a pseudo-rectangle with the following contents:\n\n| No. of bytes           | Type        | Description        |\n|------------------------|-------------|--------------------|\n| 2                      | `U16`       | _version_          |\n| 2                      | `U16`       | _number-of-codecs_ |\n| 2 * _number-of-codecs_ | `U16` array | _codecs_           |\n\nThe supported codecs are as follow:\n\n| Codec | Description                 |\n|-------|-----------------------------|\n| 0     | Opus codec, WebM container  |\n| 1     | MP3 codec, MPEG-1 container |\n\nAfter receiving this notification, clients may optionally use the [Replit Audio\nClient Message](#replit-audio-client-client-to-server-messages).\n\n## Client to Server Messages\n\nThis registers the following message types:\n\n| Number | Name                        |\n|--------|-----------------------------|\n| 245    | Replit Audio Client Message |\n\nThis message may only be sent if the client has previously received a\n_FrameBufferUpdate_ that confirms support for the intended message-type. Every\n`Replit Audio Client Message` begins with a standard header\n\n| No. of bytes | Type  | [Value] | Description       |\n|--------------|-------|---------|-------------------|\n| 1            | `U8`  | 245     | _message-type_    |\n| 1            | `U8`  |         | _submessage-type_ |\n| 2            | `U16` |         | _payload-length_  |\n\nThis header is then followed by arbitrary data of length _payload-length_, and\nwhose format is determined by the _submessage-type_. Possible values for\n_submessage-type_ and their associated minimum versions are\n\n| Submessage Type | Minimum version | Description                                                                       |\n|-----------------|-----------------|-----------------------------------------------------------------------------------|\n| 0               | 0               | [Start Encoder](#replit-audio-client-start-encoder-message)                       |\n| 1               | 0               | [Frame Request](#replit-audio-client-frame-request-message)                       |\n| 2               | 0               | [Start Continuous Updates](#replit-audio-client-start-continuous-updates-message) |\n\n### Replit Audio Client Start Encoder Message\n\nThis submessage allows the client to request the server to start audio capture\nwith the provided configuration\n\n| No. of bytes | Type  | [Value] | Description       |\n|--------------|-------|---------|-------------------|\n| 1            | `U8`  | 245     | _message-type_    |\n| 1            | `U8`  | 0       | _submessage-type_ |\n| 2            | `U16` | 6       | _payload-length_  |\n| 1            | `U8`  |         | _enabled_         |\n| 1            | `U8`  |         | _channels_        |\n| 2            | `U16` |         | _codec_           |\n| 2            | `U16` |         | _kbytes_per_sec_  |\n\nAfter invoking this operation, the client will receive a [Replit Audio Server\nStart Encoder Message](#replit-audio-server-start-encoder-message) with the\nresult of the operation.\n\nValid values for the _enabled_ field are 0, which disables/stops the audio\nencoder, and 1, which starts the audio encoder. Valid values for the _channels_\nfield are 1 (Mono audio) and 2 (Stereo audio). Valid values for the _codec_\nfield are the ones sent by the server in the [Replit Audio\nPseudo-encoding](#encodings) pseudo-rect. Valid values for the _kbytes_per_sec_\nfield are codec-dependent. The Opus codec achieves good performance with 32,\nwhereas the MP3 codec might require 128 for a comparable experience.\n\n### Replit Audio Client Frame Request Message\n\nThis submessage allows the client to request the server for a single audio\nframe. The length of an audio frame is codec-dependent, but is typically\nbetween 5 and 40 milliseconds. Each frame is encoded with the parameters chosen\nby the [Start Encoder](#replit-audio-client-start-encoder-message) message.\nThe client MUST send a [Start\nEncoder](#replit-audio-client-start-encoder-message) message and have received\nacknowledgement from the server that the chosen parameters are valid prior to\nsending this message.\n\n| No. of bytes | Type  | [Value] | Description       |\n|--------------|-------|---------|-------------------|\n| 1            | `U8`  | 245     | _message-type_    |\n| 1            | `U8`  | 1       | _submessage-type_ |\n| 2            | `U16` | 0       | _payload-length_  |\n\nAfter invoking this operation, the client will receive a [Replit Audio Server\nFrame Response Message](#replit-audio-server-frame-response-message) with the\nencoded audio frame in the corresponding container format.\n\n### Replit Audio Client Start Continuous Updates Message\n\nThis submessage allows the client to request the server send audio frames\ncontinuously, which saves bandwidth and reduces audio latency incurred by the\nTCP stack by half compared to requesting frames individually. The length of an\naudio frame is codec-dependent, but is typically between 5 and 40 milliseconds.\nEach frame is encoded with the parameters chosen by the [Start\nEncoder](#replit-audio-client-start-encoder-message) message.  The client MUST\nsend a [Start Encoder](#replit-audio-client-start-encoder-message) message and\nhave received acknowledgement from the server that the chosen parameters are\nvalid prior to sending this message.\n\n| No. of bytes | Type  | [Value] | Description       |\n|--------------|-------|---------|-------------------|\n| 1            | `U8`  | 245     | _message-type_    |\n| 1            | `U8`  | 1       | _submessage-type_ |\n| 2            | `U16` | 0       | _payload-length_  |\n\nAfter invoking this operation, the client will receive a [Replit Audio Server\nStart Continuous Updates\nMessage](#replit-audio-server-start-continuous-updates-message) with the result\nof the operation. If the operation was successful, that message will be\nfollowed by [Replit Audio Server Frame Response\nMessage](#replit-audio-server-frame-response-message) messages with and encoded\naudio frame in the corresponding container format.\n\nOnce audio frames start being continuously sent, this can be stopped by sending\na [Start Encoder](#replit-audio-client-start-encoder-message) message with the\n_enabled_ field set to `0`. Due to inherent race conditions in the protocol,\nafter disabling the encoder, the client may still receive further [Replit Audio\nServer Frame Response Message](#replit-audio-server-frame-response-message)\nmessages, but once the server acknowledges the receipt of the [Start\nEncoder](#replit-audio-client-start-encoder-message) message, no further audio\nframes will be sent.\n\n## Server to Client Messages\n\nThis registers the following message types:\n\n| Number | Name                        |\n|--------|-----------------------------|\n| 245    | Replit Audio Server Message |\n\nThis message may only be sent if the client has previously sent a [Replit Audio\nClient Message](#replit-audio-client-message) that confirms support for the\nintended message-type. Every `Replit Audio Server Message` begins with a\nstandard header\n\n| No. of bytes | Type  | [Value] | Description       |\n|--------------|-------|---------|-------------------|\n| 1            | `U8`  | 245     | _message-type_    |\n| 1            | `U8`  |         | _submessage-type_ |\n| 2            | `U16` |         | _payload-length_  |\n\nThis header is then followed by arbitrary data of length _payload-length_, and\nwhose format is determined by the _submessage-type_. Possible values for\n_submessage-type_ and their associated minimum versions are\n\n| Submessage Type | Minimum version | Description                                                                       |\n|-----------------|-----------------|-----------------------------------------------------------------------------------|\n| 0               | 0               | [Start Encoder](#replit-audio-server-start-encoder-message)                       |\n| 1               | 0               | [Frame Request](#replit-audio-server-frame-request-message)                       |\n| 2               | 0               | [Start Continuous Updates](#replit-audio-server-start-continuous-updates-message) |\n\n### Replit Audio Server Start Encoder Message\n\nThis submessage is a response to the [Replit Audio Client Start Encoder\nMessage](#replit-audio-client-start-encoder-message), and acknowledges the\nreceipt and/or support for the requested configuration\n\n| No. of bytes | Type  | [Value] | Description       |\n|--------------|-------|---------|-------------------|\n| 1            | `U8`  | 245     | _message-type_    |\n| 1            | `U8`  | 0       | _submessage-type_ |\n| 2            | `U16` | 1       | _payload-length_  |\n| 1            | `U8`  |         | _enabled_         |\n\nIf the parameters in the [Replit Audio Client Start Encoder\nMessage](#replit-audio-client-start-encoder-message) were valid and the server\nwas able to successfully start an audio capture session, the value of _enabled_\nwill be 1. Otherwise it will be 0.\n\nAfter receiveing this message with _enabled_ set to 1, the client can send\nother [Replit Audio Client Message](#client-to-server-messages) messages.\n\n### Replit Audio Server Frame Request Message\n\nThis submessage contains audio data for a single audio frame wrapped in the\ncontainer format associated with it. The length of an audio frame is\ncodec-dependent, but is typically between 5 and 40 milliseconds. The frame is\nencoded with the parameters chosen by the [Start\nEncoder](#replit-audio-client-start-encoder-message) message. This is a\nresponse to either the [Replit Audio Client Frame Request\nMessage](#replit-audio-client-frame-request-message) or the [Replit Audio\nClient Start Continuous Updates\nMessage](#replit-audio-client-start-continuous-updates-message).\n\n| No. of bytes   | Type       | [Value]           | Description       |\n|----------------|------------|-------------------|-------------------|\n| 1              | `U8`       | 245               | _message-type_    |\n| 1              | `U8`       | 1                 | _submessage-type_ |\n| 2              | `U16`      | 4 + _data-length_ | _payload-length_  |\n| 4              | `U32`      |                   | _timestamp_       |\n| _data-length_  | `U8` array |                   | _data_            |\n\nThe most significant bit of _timestamp_ denotes whether the audio frame\ncontains a start-of-stream header or is otherwise a keyframe, which enables\nclients to use this information for seeking purposes. Servers SHOULD send\nkeyframes every few seconds / minutes to allow clients to re-synchronize with\nthe stream. The 31 least significant bits of _timestamp_ contain the number of\nmilliseconds from the first audio frame that was captured in the session since\nthe [Start Encoder](#replit-audio-client-start-encoder-message) message was\nacknowledged by the server. _data_ SHOULD be a self-contained audio frame, and\nall the audio frames should be concatenable into a valid audio stream.\nFurthermore, dropping of a non-keyframe SHOULD not cause the client to\nde-synchronize, and SHOULD be recoverable by inserting silence for the duration\nof the dropped frame.\n\n### Replit Audio Server Start Continuous Updates Message\n\nThis submessage is a response to the [Replit Audio Client Start Continuous\nUpdates Message](#replit-audio-client-start-continuous-updates-message), and\nacknowledges the receipt of it and signals the client that the server will send\n[Replit Audio Server Frame Request\nMessage](#replit-audio-server-frame-request-message) messages continuously.\n\n| No. of bytes | Type  | [Value] | Description       |\n|--------------|-------|---------|-------------------|\n| 1            | `U8`  | 245     | _message-type_    |\n| 1            | `U8`  | 0       | _submessage-type_ |\n| 2            | `U16` | 1       | _payload-length_  |\n| 1            | `U8`  |         | _enabled_         |\n\n_enabled_ will be set to 1 when the stream of [Replit Audio Server Frame\nRequest Message](#replit-audio-server-frame-request-message) messages will\nstart. _enabled_ will be set to 0 if client had not sent a [Start\nEncoder](#replit-audio-client-start-encoder-message) message beforehand, or if\nthere was any other problem starting the stream. If there is an error at any\nfuture point, or if the client sent a [Start\nEncoder](#replit-audio-client-start-encoder-message) with the _enabled_ field\nset to 0, the server will send an additional [Replit Audio Server Start\nContinuous Updates\nMessage](#replit-audio-server-start-continuous-updates-message) with _enabled_\nset to 0 after sending the last audio frame.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freplit%2Frfbproxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freplit%2Frfbproxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freplit%2Frfbproxy/lists"}