{"id":21740151,"url":"https://github.com/bserdar/watermelon","last_synced_at":"2025-07-15T06:48:00.688Z","repository":{"id":67456345,"uuid":"219843063","full_name":"bserdar/watermelon","owner":"bserdar","description":"Infrastructure as real code","archived":false,"fork":false,"pushed_at":"2020-06-19T15:38:46.000Z","size":168,"stargazers_count":14,"open_issues_count":1,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-13T03:46:12.192Z","etag":null,"topics":["configuration-management","infrastructure-as-code"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bserdar.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-05T20:20:53.000Z","updated_at":"2024-11-24T10:23:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"fb79b118-2e1e-4801-bbca-fae8d49642fb","html_url":"https://github.com/bserdar/watermelon","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bserdar/watermelon","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bserdar%2Fwatermelon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bserdar%2Fwatermelon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bserdar%2Fwatermelon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bserdar%2Fwatermelon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bserdar","download_url":"https://codeload.github.com/bserdar/watermelon/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bserdar%2Fwatermelon/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265414852,"owners_count":23761084,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["configuration-management","infrastructure-as-code"],"created_at":"2024-11-26T06:12:25.133Z","updated_at":"2025-07-15T06:48:00.641Z","avatar_url":"https://github.com/bserdar.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Watermelon \u0026nbsp; \n[![GoDoc](https://godoc.org/github.com/bserdar/watermelon?status.svg)](https://godoc.org/github.com/bserdar/watermelon)\n[![Go Report](https://goreportcard.com/badge/github.com/bserdar/watermelon)](https://goreportcard.com/report/github.com/bserdar/watermelon)\n\n\n\nWatermelon is a tool for building infrastructure as code. It lets you\nuse a regular programming language (as opposed to some specialized\nscripting language) to connect to remote machines, run commands, copy\nfiles to manage configuration remotely. Watermelon abstracts the\ndetails of accessing those remote machines. \n\nWatermelon is written in Go, and currently supports infrastructure\ncode written in Go. When you write infrastructure code (a *module*),\nyou import the watermelon client package that provides the runtime for\nyour code. Using the client, you can write code that looks like this:\n\n```\nfunc OpenPorts(session *client.Session) {\n  session.ForAllSelected(client.Has(\"smtp\"), func(host client.Host) error {\n    for _, p := range PORTS {\n      host.Commandf(\"firewall-cmd --permanent --add-port=%d/tcp\", p)\n    }\n    host.Command(\"firewall-cmd --reload\")\n    return nil\n  })\n}\n```\n\n\nThis piece of code opens some ports on all hosts labeled with `smtp`\nusing `firewall-cmd`.\n\nWatermelon communicates with the modules you write using gRPC and\nJSON. When you run your infrastructure code, watermelon builds your\ncode, runs it, and communicates with it over gRPC. You don't need to\nbuild your code separately. This lets you work with your infrastructure\ncode as if it is a script--you can write code, and run it.\n\nFrom your module, you can call the functions provided by the client\nlibrary as well as functions in other modules. The other modules can\nbe written in languages other than Go, provided there is a client\nruntime for that language.\n\nWatermelon is procedural, not declarative. However, if you can write\nidempotent infrastructure code, you can use it as a declarative tool\nas well. It is very hard to write declarative rules for things such as\ninstall packages, reboot, then continue with other tasks.\n\n\n## Install\n\nUse `go get`:\n\n```\ngo get github.com/bserdar/watermelon\n```\n\nThese are some existing modules you can use:\n\n```\ngit clone github.com/bserdar/watermelon-modules\n```\n\nYou will need to pass the directory of these modules to `watermelon` to use\nthem, or define WM_MODULES environment variable:\n\n```\nexport WM_MODULES=\"~/go/src/github.com/bserdar/watermelon-modules\"\n```\n\n## Running\n\nUsing watermelon you run functions exported in modules. \n\nRun watermelon using:\n\n```\nwatermelon run --inv inventory.yml --mdir /dir-to-modules/watermelon-modules --mdir /other/module/dir someModule someFunc\n```\n\n * --inv inventory.yml: This will load `inventory.yml` which contains\nhost definitions and configuration options. The configuration items\ncan be accessed from the running scripts using JSON pointers.\n * --mdir dir: Each --mdir option will define a directory under which\n   modules can be found. Each module has the name of the last\n   component of the directory it is in.\n * someModule someFunc: The module and function to run. This will\n   build and load the module `someModule` under one of the `--mdir`s,\n   and then execute the function `someFunc` in that module.\n   \n\n\n## Inventory and Configurations\n\nThe inventory contains remote hosts and host specific or global\nconfiguration options. Inventory file looks like this:\n\n```\n# The ssh private key file to use to ssh remote hosts\n# If omitted, username and password for each host must\n# be provided.\nprivateKey: /home/user/.ssh/id_rsa\n\n# Passphrase for the private key. If omitted, passphrase will\n# be asked via stdin first time it is needed\npassphrase: 123abcdef\n\n# Hosts section lists all remote hosts\nhosts:\n  - \n    # id is the unique host id. This is how scripts\n    # identify hosts\n    id: host1\n    \n    # Addresses of this host. This host has two\n    # addresses, a primary one and a private one.\n    addresses:\n      - address: xxx.xxx.xxx.xxx\n        name: primary\n      - address: xxx.2xxx.xxx.xxx\n        name: private\n\n    # ssh parameters for the host. These are hidden from the \n    # scripts. ssh tries all given authentication schemes to \n    # connect a host. Specifying both password and private key\n    # will allow ssh using an initial password, then disabling\n    # password authorization in favor of private key auth.\n    ssh:\n      # The name or the IP of the host\n      hostname: xxx.xxx.xxx.xxx\n      # The user to ssh as\n      user: root\n      # Password\n      password: \"pwd01\"\n      # This will add \"sudo\" to all commands, so \n      # you can login as non-root\n      become: sudo\n      \n    # Labels assigned to the host. You can selects groups\n    # of hosts using their labels\n    labels:\n      - db\n  \n    # Node specific properties, key=value\n    properties:\n      dbPassword: \"pwd\"\n      \n    # Node specific configuration options. YAML object. These override \n    # matching global configuration items\n    configuration:\n\n\n# Labels section assign labels to hosts\nlabels:\n  # This assigns 'controller' label to 'host1' and 'host2'\n  controller:\n     - host1\n     - host2\n  # This assigns 'worker' label to 'host3' and 'host4'\n  worker:\n     - host3\n     - host4\n     \n# Configuration section contains global configuration options\nconfiguration:\n  endpoint: http://myendpoint\n  dbport: 2222\n```\n\n\nWatermelon merges all configuration files and the contents of the\n`configuration` item in the inventory, and serves them as a common\nconfiguration tree. You can query individual items using JSON\npointers, and unmarshal them into structs.\n\nFor example, the following\nwill unmarshal the contents of JSON location `/endpoint` :\n\n```\nvar endpoint string\nsession.GetCfg(\"/endpoint\", \u0026endpoint)\n// endpoint is now http://myendpoint\n```\n\n\nWatermelon client runtime provides APIs to access inventory items by\ntheir labels or ids. For example:\n\n```\n// Select all hosts that has 'controller' label, and run \n// the function for each host. Each func runs in it own\n// goroutine.\nsession.ForAllSelected(client.Has(\"controller\"), func(host client.Host) error {\n\t\tvar cfg MyConfig\n        // Unmarshal contents of /configuration/myconfig from\n        // the inventory file, or /myconfig\n        // from one of the configuration files\n\t\tsession.GetCfg(\"/myconfig\", \u0026cfg)\n        ...\n```\n        \n## Logs\n\nFor each run, Watermelon server creates a log directory containing the\ntimestamp of the run. Under that directory, each remote host and if\nyou use it `localhost` has separate log files. Any logs generated for\nremote hosts and localhost will be written to its corresponding file.\n\n## Modules\n\nYou write your infrastructure code as `modules`. A module is an\nexecutable program that communicates with the watermelon server using\ngRPC. Currently watermelon has a Go client runtime, but other runtimes\ncan be written for any language that supports gRPC.\n\nA module source tree should look like the following:\n\n```\nmoduleroot/\n   moduleName/\n      module.w\n      ...\n```\n\nThe `moduleroot` will be given to `watermelon` with the `--mdir` flag. Each\ndirectory with a file called `module.w` is a module, and the module\nname is the directory name it is under.\n\n`module.w` is a shell script that is run in the module directory.\nIt can be executed in one of the following forms:\n\n```\n./module.w buildrun :port --log \u003cloglevel\u003e\n```\n\n`module.w` should build and run the module. When run, the module \nconnects to the Watermelon server at `localhost:port`.\n\n```\n./module.w run :port --log \u003cloglevel\u003e\n```\n\n`module.w` should run the module, it doesn't have to build it.\n\nFor example, `pkg/module.w` file looks like the following. It runs\n`make` if necessary, and then runs the binary with the arguments:\n\n```\n#!/bin/sh\n\nif test \"$#\" -lt 2; then\n    return 2\nfi\n\nif [ \"$1\" == \"buildrun\" ]; then\n    make || exit 1;\nfi\n\nshift\n\n./pkg $*\n```\n\n\n\nWatermelon server executes the module with a host:port\nargument that the module uses to connect to the server. \n\nThe following show how this is done in Go:\n```\npackage main\n\nimport (\n\t\"os\"\n\n\t\"github.com/bserdar/watermelon/client\"\n)\n\nfunc main() {\n\tclient.Run(os.Args[1:], nil, nil)\n}\n```\n\n\nIn order to make a function acessible from other modules, you have to\n*export* it:\n\n```\n// Export the function SetupDB as smtp.SetupDB.\n// Callers will have to call this function as module.smtp.SetupDB\nvar _ = client.Export(\"smtp.SetupDB\", SetupDB)\n\n// The SetupDB function, \nfunc SetupDB(session *client.Session) {\n  ...\n}\n```\n\nExported functions can have one of the following signatures:\n\n```\nfunc WithInputAndOutput(*Session,InStruct) (OutStruct,error)\nfunc WithInputOnly(*Session,InStruct) error\nfunc WithInputNoError(*Session,InStruct)\nfunc OnlySession(*Session)\n```\n\nThe input and output structures must be JSON marshalable.\n\nThe `Session` provides the interface to the watermelon server. Using\nthe `Session`, the function can select hosts, and run commands on them.\n\n```\nsession.ForAllSelected(client.Has(\"controller\"), func(host client.Host) error {\n        var ca []byte\n        ca:=getCertificate(host)\n        \n        // Write the byte array to the remote host\n        host.WriteFile(\"/certs\", 0644, ca)\n        // Run systemctl on the remote host\n        host.Command(\"systemctl restart myservice\")\n\n        // Ensure directory exists on remote host\n   \t    host.Ensure(\"/etc/myapp/config\", client.Ensure{}.EnsureDir())\n\n        // Evaluate the Go template from local directory using templateData, and write it to remote host if different.\n  \t    changed, err := node.WriteFileFromTemplateFile(\"/etc/config\", 0644, \"templates/config\", templateData)\n  }\n}\n```\n\nYou can call an exported function from a module:\n\n```\nresponse:=session.Call(\"myModule\",\"smtp.SetupDB\",nil)\nif !response.Success {\n   return errors.New(response.ErrorMsg)\n}\n```\n\nIf the exported function gets an argument, you have to send an object\nthat can be JSON-marshaled:\n\n```\nsession.Call(\"pkg\",\"func\",map[string]interface{}{\"hostId\":host.ID,\"pkg\":\"ntpd\"})\n```\n\n### Using gRPC to Export Functions\n\nYou can implement a gRPC server for the modules. The functions\nexported using the gRPC server can be accessed using the\n`session.Call` method as well as using direct gRPC.\n\nTo implement the module as a gRPC server, change the main as follows:\n\n```\nimport (\n\t\"os\"\n\n\t\"google.golang.org/grpc\"\n\n\t\"github.com/bserdar/watermelon-modules/pkg/yum\"\n\t\"github.com/bserdar/watermelon/client\"\n)\n\nfunc main() {\n    // Create the grpc server \n\tyumServer := yum.Server{}\n    // Register the grpc server to runtime \n\tclient.Run(os.Args[1:], nil, func(server *grpc.Server, rt *client.Runtime) {\n        // Register the grpc server to the runtime with name \"yum\".\n        // This allows other modules to call the gRPC functions of this module \n        // using session.Call(\"pkg\",\"yum.funcName\")\n\t\trt.RegisterGRPCServer(\u0026yumServer, \"yum\")\n        \n        // Register the gRPC server to the watermelon runtime gRPC server\n\t\tyum.RegisterYumServer(server, yumServer)\n\t})\n}\n```\n\nThis does two things: first it registers the gRPC server of the\nimplementation to the watermelon client runtime so it can proxy\nnon-gRPC calls to the gRPC server functions. Second, it registers a\ngRPC server to the module's gRPC implementation.\n\nThe `main` function can register as many gRPC servers as you need.\n\nThe actual implementation of the module functions is as follows:\n\n```\npackage yum\n\nimport (\n\t\"github.com/bserdar/watermelon/client\"\n\t\"github.com/bserdar/watermelon/server/pb\"\n)\n\n// This is the gRPC server for the yum module\ntype Server struct {\n\tclient.GRPCServer\n}\n\n// The gRPC implementation of the Install function\nfunc (s Server) Install(ctx context.Context, req *PackageParams) (*pb.Response, error) {\n    // Get the watermelon session from the server\n\tsession := s.SessionFromContext(ctx)\n    // Do the work\n    ...\n    // Return the response\n\treturn \u0026pb.Response{Data: stdout, ErrorMsg: stderr, Modified: true}, nil\n}\n```\n\nThe session information is send using gRPC metadata, and\n`s.SessionFromContext(ctx)` retrieves that.\n\nIf a module implements a gRPC server, then you can call the functions\neither by `session.Call`, or by the gRPC bridge:\n\n```\n// Using session.Call\nresponse := node.S.Call(\"pkg\", \"yum.Ensure\", map[string]interface{}{\"hostId\": node.ID,\n  \"pkgs\":    []string{\"mongodb-org\", \"firewalld\"},\n  \"version\": \"installed\"})\n```\n\n\nWhen calling a module function using thr gRPC bridge, make sure you pass in a \ncontext obtained from `session.Context()`:\n```\n// Using gRPC bridge\n\n// This will load the \"pkg\" module, and return a gRPC connection to it\npkgConn, err := session.ConnectModule(\"pkg\")\nif err != nil {\n   return err\n}\nyumCli := yum.NewYumClient(pkgConn)\n// You have to pass in the context you obtained from the session, otherwise the\n// receiving end will not receive session info\nyumCli.Ensure(node.S.Context(), \u0026yum.EnsureParams{HostId: host.ID,\n                Pkgs:    []string{\"wget\", \"ntpdate\", \"iptables-services\"},\n                Version: \"installed\"})\n```\n\n\n\n## Examples\n\nInstall ETCD to all nodes if it is not already installed:\n\n```\nfunc ETCDBootstrap(session *client.Session) {\n  // Select all hosts that are labeled with etcd\n  session.ForAllSelected(client.Has(\"etcd\"), func(host client.Host) error {\n      // This section runs for all hosts with label \"etcd\" concurrently\n      \n     // Check if etcd already exists\n     rsp := host.Commandf(\"/usr/local/bin/etcd -version\")\n     if strings.Index(string(rsp.Stdout), \"etcd Version\") == -1 {\n        // Download etcd tarfile to /tmp and install\n        rsp := host.Command(\"mktemp -d\")\n        tempDir := strings.TrimSpace(string(rsp.Stdout))\n        rsp = host.Commandf(\"wget %s -O %s/tarfile\", ETCDTarFile, tempDir)\n        if rsp.ExitCode != 0 {\n          return fmt.Errorf(\"Cannot wget etcd: %s\", string(rsp.Stderr))\n        }\n        // Print log message for this host\n        host.Logf(\"Downloaded etcd\")\n        host.Commandf(\"sh -c 'cd %s \u0026\u0026 tar  --strip-components=1 -x -f tarfile \u0026\u0026 mv etcd* /usr/local/bin'\", tempDir)\n     }\n     return nil\n  })\n}\n```\n\nReset mariadb root password:\n\n```\nfunc ResetRootPWD(session *client.Session) {\n  // Run this for all hosts labeled with \"mariadb\"\n  session.ForAllSelected(client.Has(\"mariadb\"), func(host client.Host) error {\n    // Get mariadb root password from host properties in inventory\n    password, ok := host.GetInfo().Properties[\"mariadb_root_password\"]\n    if !ok {\n      return fmt.Errorf(\"Root password required in mariadb_root_password\")\n    }\n    host.Command(\"systemctl stop mariadb\")\n    host.Command(`nohup mysqld_safe --skip-grant-tables \u003c/dev/null \u003e/dev/null 2\u003e/dev/null \u0026`)\n    time.Sleep(2 * time.Second)\n    host.Commandf(`cat \u003c\u003cEOF | mysql -u root\nuse mysql;\nupdate user SET PASSWORD=PASSWORD(\"%s\") WHERE USER='root';\nflush privileges;\nexit\nEOF\n`, password)\n    host.Commandf(\"mysqladmin -u root --password=%s shutdown\", rootPwd)\n    host.Command(\"systemctl start mariadb\")\n  })\n}\n```\n\nWrite /etc/hosts from a template:\n\n```\nfunc WriteHosts(session *client.Session) {\n  // For all hosts\n  session.ForAllSelected(client.AllHosts,func(host client.Host) error {\n    // Get host information for all hosts\n    hosts := host.S.GetHosts(client.AllHosts)\n    // Write /etc/hosts from the template. The templates are relative to the module root\n    return node.WriteFileFromTemplateFile(\"/etc/hosts\", 0644, \"templates/hosts\", hosts)\n  })\n}\n```\n\nwhere the hosts template is:\n```\n27.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4\n::1         localhost localhost.localdomain localhost6 localhost6.localdomain6\n\n{{range $index,$host := .}}\n{{range .Addresses}}\n{{.Address}} {{$host.ID}}\n{{end -}}\n{{end -}}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbserdar%2Fwatermelon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbserdar%2Fwatermelon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbserdar%2Fwatermelon/lists"}