{"id":13563904,"url":"https://github.com/knqyf263/go-plugin","last_synced_at":"2025-05-15T06:06:18.181Z","repository":{"id":58202299,"uuid":"526497091","full_name":"knqyf263/go-plugin","owner":"knqyf263","description":"Go Plugin System over WebAssembly","archived":false,"fork":false,"pushed_at":"2025-03-12T12:26:37.000Z","size":3309,"stargazers_count":643,"open_issues_count":14,"forks_count":33,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-04-14T10:42:36.115Z","etag":null,"topics":["go","golang","plugin","plugins","protobuf3","protocol-buffers"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/knqyf263.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-08-19T06:58:57.000Z","updated_at":"2025-04-12T15:25:34.000Z","dependencies_parsed_at":"2024-01-14T03:48:00.240Z","dependency_job_id":"1ed3c38f-bb41-4ebf-bf6f-89846a8cd10d","html_url":"https://github.com/knqyf263/go-plugin","commit_stats":{"total_commits":82,"total_committers":8,"mean_commits":10.25,"dds":"0.36585365853658536","last_synced_commit":"114c6257e441f37a3b5c970a5f34d420729b9a9a"},"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knqyf263%2Fgo-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knqyf263%2Fgo-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knqyf263%2Fgo-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knqyf263%2Fgo-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/knqyf263","download_url":"https://codeload.github.com/knqyf263/go-plugin/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254283339,"owners_count":22045140,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","plugin","plugins","protobuf3","protocol-buffers"],"created_at":"2024-08-01T13:01:24.429Z","updated_at":"2025-05-15T06:06:13.163Z","avatar_url":"https://github.com/knqyf263.png","language":"Go","readme":"# Go Plugin System over WebAssembly\n\n\u003cimg src=\"imgs/icon.png\" width=\"200\"\u003e\n\n`go-plugin` is a Go (golang) plugin system over WebAssembly (abbreviated Wasm).\nAs a plugin is compiled to Wasm, it can be size-efficient, memory-safe, sandboxed and portable.\nThe plugin system auto-generates Go SDK for plugins from [Protocol Buffers][protobuf] files.\nWhile it is powered by Wasm, plugin authors/users don't have to be aware of the Wasm specification since the raw Wasm APIs are capsulated by the SDK.\n\nIt uses the same definition as gRPC, but `go-plugin` communicates with plugins in memory, not over RPC.\n\nIt is inspired by [hashicorp/go-plugin][hashicorp-go-plugin].\n\n## Features\nThe Go plugin system supports a number of features:\n\n**Auto-generated Go interfaces:** The plugin system generates Go code for hosts and plugins from Protocol Buffers files like gRPC.\nIt is easy to learn how to use `go-plugin` for protobuf/gRPC users.\n\n**Plugins are Go interface implementations:** \nRaw Wasm APIs are hidden so that user can write and consume plugins naturally.\nTo a plugin author: you just implement an interface as if it were going to run in the same process.\nFor a plugin user: you just use and call functions on an interface as if it were in the same process.\nThis plugin system handles the communication in between.\n\n**Safe:** Wasm describes a memory-safe, sandboxed execution environment.\nPlugins cannot access filesystem and network unless hosts allow those operations.\nEven 3rd-party plugins can be executed safely.\nPlugins can't crash the host process as it is sandboxed.\n\n**Portable:** Wasm is designed as a portable compilation target for programming languages.\nPlugins compiled to Wasm can be used anywhere.\nA plugin author doesn't have to distribute multi-arch binaries.\n\n**Efficient:**\nThe Wasm stack machine is designed to be encoded in a size- and load-time-efficient binary format.\n\n**Bidirectional communication:**\nWasm allows embedding host functions.\nAs Wasm restricts some capabilities such as network access for security, plugins can call host functions that explicitly embedded by a host to extend functionalities.\n\n**Stdout/Stderr Syncing:**\nPlugins can use stdout/stderr as usual and the output will get mirrored back to the host process.\nThe host process can control what io.Writer is attached to stdout/stderr of plugins.\n\n**Protocol Versioning:**\nA very basic \"protocol version\" is supported that can be incremented to invalidate any previous plugins.\nThis is useful when interface signatures are changing, protocol level changes are necessary, etc.\nWhen a protocol version is incompatible, a human friendly error message is shown to the end user.\n\n## Architecture\n`go-plugin` generates Go SDK for a host and plugins.\nThe plugin system works by loading the Wasm file and communicating over exporting/exported methods.\n\nThis architecture has a number of benefits:\n\n- Plugins can't crash your host process: a panic in a plugin is handled by the Wasm runtime and doesn't panic the plugin user.\n- Plugins are very easy to write: just write a Go application and `GOOS=wasip1 GOARCH=wasm go build`.\n- Plugins are very easy to distribute: just compile the Go source code to the Wasm binary once and distribute it.\n- Plugins are very easy to install: just put the Wasm binary in a location where the host will find it.\n- Plugins can be secure: the plugin is executed in a sandbox and doesn't have access to the local filesystem and network by default.\n\n## Installation\n\nDownload a binary [here][releases] and put it in `$PATH`.\n\n## Requirements\n- Go 1.24+\n\n## Support Policy\n`go-plugin` is based on [Wazero][wazero] runtime and has Support Policy which follows [same rules](https://github.com/tetratelabs/wazero/?tab=readme-ov-file#go):\n\u003e wazero follows the same version policy as Go's [Release Policy](https://go.dev/doc/devel/release): two versions. wazero will ensure these versions work and bugs are valid if there's an issue with a current Go version.\n\nFor example, if current version of Go is `go1.25`, `go-plugin` is ensured to work with Go versions:\n- `go1.24`\n- `go1.25`\n\n## Usage\nTo use the plugin system, you must take the following steps.\nThese are high-level steps that must be done.\nExamples are available in the `examples/` directory.\n\n1. Choose the interface(s) you want to expose for plugins.\n    - Define [Messages][protobuf-message] and [Services][protobuf-service] in a proto file.\n1. Generate SDK for a host and plugin by `go-plugin`.\n1. Implement the Go interface defined in the plugin SDK.\n1. Compile your plugin to Wasm.\n1. Load the plugin and call the defined methods.\n\nThe development flow is as below.\n\n\u003cimg src=\"imgs/architecture.png\" width=\"600\"\u003e\n\n## Tutorial\nLet's create a hello-world plugin.\n\n### Prerequisite\nInstall the following tools:\n\n- `knqyf263/go-plugin` (See `Installation`)\n- [protoc][protoc]\n- [Go][go-installation]\n\n### Choose the interface you want to expose for plugins\nCreate `greeting.proto`.\n\n```protobuf\nsyntax = \"proto3\";\npackage greeting;\n\noption go_package = \"github.com/knqyf263/go-plugin/examples/helloworld/greeting\";\n\n// The greeting service definition.\n// go:plugin type=plugin version=1\nservice Greeter {\n  // Sends a greeting\n  rpc SayHello(GreetRequest) returns (GreetReply) {}\n}\n\n// The request message containing the user's name.\nmessage GreetRequest {\n  string name = 1;\n}\n\n// The reply message containing the greetings\nmessage GreetReply {\n  string message = 1;\n}\n```\n\nMost of the definitions are simply as per [the Protocol Buffers specification][protobuf-spec].\nThe only difference is the line starting with `// go:plugin`.\nIt defines parameters for `go-plugin`.\n`type=plugin` means the service defines the plugin interface.\n\n### Generate SDK\nRun the following command.\n\n```shell\n$ protoc --go-plugin_out=. --go-plugin_opt=paths=source_relative greeting.proto\n```\n\nThen, you will find 4 files generated in the same directory, `greet.pb.go`, `greet_host.pb.go`, `greet_plugin.pb.go` and `greet_vtproto.pb.go`.\n\n#### Optional: Skip the go protobuf generation\n\nIf the plugin should be used together with other generators like\n[`protoc-gen-go`](https://pkg.go.dev/github.com/golang/protobuf/protoc-gen-go),\nthen it may be helpful to skip the standard protobuf generation. This can be\ndone by setting the option `disable_pb_gen=true`, for example:\n\n```shell\n$ protoc \\\n    --go_opt=paths=source_relative --go_out=. \\\n    --go-plugin_opt=paths=source_relative,disable_pb_gen=true --go-plugin_out=. \\\n    greeting.proto\n```\n\n### Implement a plugin\nThe `Greeter` interface is generated as below in the previous step. \n\n```go\ntype Greeter interface {\n\tSayHello(context.Context, GreetRequest) (GreetReply, error)\n}\n```\n\nA plugin author needs to implement `Greeter` and registers the struct via `RegisterGreeter`.\nIn this tutorial, we use `plugin.go` as a file name, but it doesn't matter.\n\n```go\n//go:build wasip1\n\npackage main\n\nimport (\n\t\"context\"\n\n\t\"github.com/path/to/your/greeting\"\n)\n\n// main is required for Go to compile to Wasm.\nfunc main() {}\n\nfunc init() {\n\tgreeting.RegisterGreeter(MyPlugin{})\n}\n\ntype MyPlugin struct{}\n\nfunc (m MyPlugin) SayHello(ctx context.Context, request greeting.GreetRequest) (greeting.GreetReply, error) {\n\treturn greeting.GreetReply{\n\t\tMessage: \"Hello, \" + request.GetName(),\n\t}, nil\n}\n```\n\nThen, compile it to Wasm by Go.\n\n```shell\n$ GOOS=wasip1 GOARCH=wasm go build -o plugin.wasm -buildmode=c-shared plugin.go\n```\n\n### Implement a host\nLoad the plugin binary and call `SayHello`.\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"log\"\n\n\t\"github.com/path/to/your/greeting\"\n)\n\nfunc main() {\n\tctx := context.Background()\n\t\n\t// Initialize a plugin loader\n\tp, err := greeting.NewGreeterPlugin(ctx)\n\tif err != nil {...}\n\tdefer p.Close(ctx)\n\n\t// Load a plugin\n\tplugin, err := p.Load(ctx, \"path/to/plugin.wasm\")\n\tif err != nil {...}\n\n\t// Call SayHello\n\treply, err := plugin.SayHello(ctx, greeting.GreetRequest{Name: \"go-plugin\"})\n\tif err != nil {...}\n\t\n\t// Display the reply\n\tfmt.Println(reply.GetMessage())\n}\n```\n\n### Run\n\n```shell\n$ go run main.go\nHello, go-plugin\n```\n\nThat's it! It is easy and intuitive.\nYou can see the `hello-world` example [here][hello-world].\n\n## References\n### Host functions\nWasm has limited capability as it is secure by design, but those can't be achieved with Wasm itself.\nTo expand the capability, many compilers implement system calls using WebAssembly System Interface ([WASI][wasi]).\nBut it is still draft ([wasi_snapshot_preview1][wasi_snapshot_preview1]) and some functions are not implemented yet in [wazero][wazero] that `go-plugin` uses for Wasm runtime.\nFor example, `sock_recv` and `sock_send` are not supported for now.\nIt means plugins don't have network access.\n\nHost functions can be used for this purpose.\nA host function is a function expressed outside WebAssembly but passed to a plugin as an import.\nYou can define functions in your host and pass them to plugins so that plugins can call the functions.\nEven though Wasm itself doesn't have network access, you can embed such function to plugins.\n\nYou can define a service for host functions in a proto file.\nNote that `// go:plugin type=host` is necessary so that `go-plugin` recognizes the service is for host functions.\nThe service name is `HostFunctions` in this example, but it doesn't matter.\n\n```protobuf\n// go:plugin type=host\nservice HostFunctions {\n  // Sends a HTTP GET request\n  rpc HttpGet(HttpGetRequest) returns (HttpGetResponse) {}\n}\n```\n\n**NOTE:** the service for host functions must be defined in the same file where other plugin services are defined.\n\nLet's say `Greeter` is defined in the same file as `HostFunctions`.\nThen, `Load()` will be able to take `HostFunctions` as an argument as mentioned later.\n\n```protobuf\n// go:plugin type=plugin version=1\nservice Greeter {\n  rpc SayHello(GreetRequest) returns (GreetReply) {}\n}\n```\n\n`go-plugin` generates the corresponding Go interface as below.\n\n```go\n// go:plugin type=host\ntype HostFunctions interface {\n    HttpGet(context.Context, HttpGetRequest) (HttpGetResponse, error)\n}\n```\n\nImplement the interface.\n\n```go\n// myHostFunctions implements HostFunctions\ntype myHostFunctions struct{}\n\n// HttpGet is embedded into the plugin and can be called by the plugin.\nfunc (myHostFunctions) HttpGet(ctx context.Context, request greeting.HttpGetRequest) (greeting.HttpGetResponse, error) {\n    ...\n}\n```\n\nAnd pass it when loading a plugin.\nAs described above, `Load()` takes the `HostFunctions` interface.\n\n```go\ngreetingPlugin, err := p.Load(ctx, \"plugin/plugin.wasm\", myHostFunctions{})\n```\n\nNow, plugins can call `HttpGet()`.\nYou can see an example [here][host-functions-example].\n\n### Define an interface version\nYou can define an interface version in the `// go:plugin` line.\n\n```protobuf\n// go:plugin type=plugin version=2\nservice Greeter {\n  // Sends a greeting\n  rpc Greet(GreetRequest) returns (GreetReply) {}\n}\n```\n\nThis is useful when interface signatures are changing. \nWhen an interface version is incompatible, a human friendly error message is shown to the end user like the following.\n\n```shell\nAPI version mismatch, host: 2, plugin: 1\n```\n\n## Tips\n### File access\nRefer to [this example][wasi-example].\n\n### Logging\n`fmt.Printf` can be used in plugins if you attach `os.Stdout` as below. See [the example][wasi-example] for more details.\n\n```Go\nmc := wazero.NewModuleConfig().\n    WithStdout(os.Stdout). // Attach stdout so that the plugin can write outputs to stdout\n    WithStderr(os.Stderr). // Attach stderr so that the plugin can write errors to stderr\n    WithFS(f)              // Loaded plugins can access only files that the host allows.\np, err := cat.NewFileCatPlugin(ctx, cat.WazeroModuleConfig(mc))\n```\n\nIf you need structured and leveled logging, you can define host functions so that plugins can call those logging functions.\n\n```protobuf\n// The host functions embedded into the plugin\n// go:plugin type=host\nservice LoggingFunctions {\n  // Debug log\n  rpc Debug(LogMessage) returns (google.protobuf.Empty) {}\n  // Info log\n  rpc Info(LogMessage) returns (google.protobuf.Empty) {}\n  // Warn log\n  rpc Info(LogMessage) returns (google.protobuf.Empty) {}\n  // Error log\n  rpc Error(LogMessage) returns (google.protobuf.Empty) {}\n}\n```\n\n### Plugin distribution\nA plugin author can use OCI registries such as GitHub Container registry (GHCR) to distribute plugins.\n\nPush:\n\n```shell\n$ oras push ghcr.io/knqyf263/my-plugin:latest plugin.wasm:application/vnd.module.wasm.content.layer.v1+wasm\n```\n\nPull:\n\n```shell\n$ oras pull ghcr.io/knqyf263/my-plugin:latest\n```\n\n## Under the hood\n`go-plugin` uses [wazero][wazero] for Wasm runtime.\nAlso, it customizes [protobuf-go][protobuf-go] and [vtprotobuf][vtprotobuf] for generating Go code from proto files.\n\n## Q\u0026A\n### Why not hashicorp/go-plugin?\nLaunching a plugin as a subprocess is not secure.\nIn addition, plugin authors need to distribute multi-arch binaries.\n\n### Why not [the official `plugin` package][plugin]?\nIt is not schema-driven like Protocol Buffers and can easily break signature.\n\n### Why not using [protobuf-go][protobuf-go] directly?\n\n`go-plugin` used to rely on TinyGo, which [doesn't support Protocol Buffers](https://github.com/tinygo-org/tinygo/issues/2667) natively.\n`go-plugin` generates Go code differently from [protobuf-go] so that TinyGo could compile it.\nNow that Go supports WASI (wasip1), I haven't fully verified if Protocol Buffers works properly when Go code is compiled to Wasm.\nThis is an area that needs further testing and validation.\n\n### Why replacing known types with custom ones?\nYou might be aware that your generated code imports [github.com/knqyf263/go-plugin/types/known][go-plugin-known], not [github.com/protocolbuffers/protobuf-go/types/known][protobuf-go-known] when you import types from `google/protobuf/xxx.proto` (a.k.a well-known types) in your proto file.\nAs described above, `TinyGo` cannot compile `github.com/protocolbuffers/protobuf-go/types/known` since those types use reflection.\n`go-plugin` provides well-known types compatible with TinyGo and use them.\nWith the release of Go 1.24, which [has improved Wasm support](https://go.dev/blog/wasmexport), these workarounds might no longer be necessary. However, I haven't fully verified this yet.\n\n### Why using `// go:plugin` for parameters rather than [protobuf extensions][protobuf-extensions]?\nAn extension must be registered in [Protobuf Global Extension Registry][protobuf-registry] to issue a unique extension number.\nEven after that, users needs to download a proto file for the extension.\nIt is inconvenient for users and the use case in `go-plugin` is simple enough, so I decided to use comments.\n\n### Why not supporting Go for plugins?\nGo doesn't support [WASI][wasi].\nYou can see other reasons [here][wazero-go].\nWe might be able to add support for Go as an experimental feature.\n\n### What about other languages?\n`go-plugin` currently supports Go plugins only, but theoretically, any language that can be compiled into Wasm can be supported.\nWelcome your contribution :)\n\n## TODO\n\n- Specification\n  - [x] Packages\n  - [x] Messages\n    - [x] Nested Types\n  - [x] Fields\n    - [x] Singular Message Fields  \n      - [x] double  \n      - [x] float  \n      - [x] int32  \n      - [x] int64  \n      - [x] uint32  \n      - [x] uint64  \n      - [x] sint32  \n      - [x] sint64  \n      - [x] fixed32  \n      - [x] fixed64  \n      - [x] sfixed32  \n      - [x] sfixed64  \n      - [x] bool  \n      - [x] string  \n      - [x] bytes  \n    - [x] Repeated Fields  \n    - [x] Map Fields  \n    - Oneof Fields (not planned)\n  - [x] Enumerations\n  - Extensions (not planned)\n  - [x] Services\n- [ ] Well-known types\n  - [x] Any (Some functions/methods are not yet implemented)\n  - [ ] Api\n  - [x] BoolValue\n  - [x] BytesValue\n  - [x] DoubleValue\n  - [x] Duration\n  - [x] Empty\n  - [x] Enum\n  - [x] EnumValue\n  - [x] Field\n  - [x] Field_Cardinality\n  - [x] Field_Kind\n  - [ ] FieldMask\n  - [x] FloatValue\n  - [x] Int32Value\n  - [x] Int64Value\n  - [x] ListValue\n  - [ ] Method\n  - [ ] Mixin\n  - [x] NullValue\n  - [x] Option\n  - [x] SourceContext\n  - [x] StringValue\n  - [x] Struct\n  - [x] Syntax\n  - [x] Timestamp\n  - [x] Type\n  - [x] UInt32Value\n  - [x] UInt64Value\n  - [x] Value\n- [x] Generate codes\n  - [x] Structs without reflection\n  - [x] Marshaling/Unmarshaling\n  - [x] Host code calling plugins\n  - [x] Plugin code called by host\n  - [x] Interface version\n  - [x] Host functions\n\n\n[protobuf]: https://developers.google.com/protocol-buffers/docs/overview\n[protobuf-message]: https://developers.google.com/protocol-buffers/docs/proto3#simple\n[protobuf-service]: https://developers.google.com/protocol-buffers/docs/proto3#services\n[protobuf-spec]: https://developers.google.com/protocol-buffers/docs/proto3\n[protobuf-extensions]: https://developers.google.com/protocol-buffers/docs/proto#extensions\n[protobuf-registry]: https://github.com/protocolbuffers/protobuf/blob/main/docs/options.md\n\n[wazero]: https://github.com/tetratelabs/wazero\n[hashicorp-go-plugin]: https://github.com/hashicorp/go-plugin\n[protoc]: https://grpc.io/docs/protoc-installation/\n[vtprotobuf]: https://github.com/planetscale/vtprotobuf\n[plugin]: https://pkg.go.dev/plugin\n[wazero-go]: https://wazero.io/languages/go/\n\n[protobuf-go]: https://github.com/protocolbuffers/protobuf-go\n[protobuf-go-known]: https://github.com/protocolbuffers/protobuf-go/tree/master/types/known\n\n[go-installation]: https://go.dev/doc/install\n\n[hello-world]: https://github.com/knqyf263/go-plugin/tree/1ebeeca373affc319802989c0fe6304f014861c4/examples/helloworld\n[go-plugin-known]: https://github.com/knqyf263/go-plugin/tree/1ebeeca373affc319802989c0fe6304f014861c4/types/known\n\n[wasi]: https://github.com/WebAssembly/WASI\n[wasi_snapshot_preview1]: https://github.com/WebAssembly/WASI/blob/snapshot-01/phases/snapshot/docs.md\n\n[wasi-example]: https://github.com/knqyf263/go-plugin/tree/main/examples/wasi\n[host-functions-example]: https://github.com/knqyf263/go-plugin/tree/main/examples/host-functions\n\n[releases]: https://github.com/knqyf263/go-plugin/releases\n\n[semver]: https://semver.org/\n","funding_links":[],"categories":["Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fknqyf263%2Fgo-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fknqyf263%2Fgo-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fknqyf263%2Fgo-plugin/lists"}