{"id":18620599,"url":"https://github.com/simonsobs/sisock","last_synced_at":"2025-11-03T12:30:22.679Z","repository":{"id":45646641,"uuid":"110147275","full_name":"simonsobs/sisock","owner":"simonsobs","description":"Sisock ('saɪsɒk): streaming of Simons Obs. data over websockets for quicklook","archived":false,"fork":false,"pushed_at":"2019-11-14T15:53:51.000Z","size":1163,"stargazers_count":2,"open_issues_count":14,"forks_count":0,"subscribers_count":32,"default_branch":"master","last_synced_at":"2024-12-27T04:25:48.197Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simonsobs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-11-09T17:43:32.000Z","updated_at":"2020-10-10T05:01:35.000Z","dependencies_parsed_at":"2022-09-15T02:50:57.461Z","dependency_job_id":null,"html_url":"https://github.com/simonsobs/sisock","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2Fsisock","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2Fsisock/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2Fsisock/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2Fsisock/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simonsobs","download_url":"https://codeload.github.com/simonsobs/sisock/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239413920,"owners_count":19634339,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T04:07:09.707Z","updated_at":"2025-11-03T12:30:22.542Z","avatar_url":"https://github.com/simonsobs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sisock\n\n[![travis badge](https://travis-ci.com/simonsobs/sisock.svg?branch=master)](https://travis-ci.com/simonsobs/sisock)\n[![Documentation Status](https://readthedocs.org/projects/sisock/badge/?version=latest)](https://sisock.readthedocs.io/en/latest/?badge=latest)\n[![Coverage Status](https://coveralls.io/repos/github/simonsobs/sisock/badge.svg?branch=master)](https://coveralls.io/github/simonsobs/sisock?branch=master)\n\n## Brief Overview\n\n**Sisock** is a suite of software for serving **Si**mons Observatory data over\nweb**sock**ets.\n\nThe key components are:\n- *A WAMP server* \u0026mdash; This runs on just one computer; all data are routed\n  through it.\n- *Data node servers* \u0026mdash; These can be spread across the internet, located\n  on machines from which data are to be served (*data nodes*).\n- A *hub* \u0026mdash; The hub keeps track of which data node servers are online.\n  Only one hub is run, by default on the computer running the WAMP server).\n- *Consumers* \u0026mdash; These are clients that request data from data node\n  servers. They connect to the hub in order to know which data node servers are\n  available and how to access them.\n  - The `components/grafana_server` is an example of a consumer. It reads data\n    via `sisock` and then passes it to Grafana over HTTP.\n\nThe following diagram illustrates the above:\n![Diagram of stuff](docs/_static/diagram.png)\n\n## Installing and Running\n\n### 1. Setting Up Crossbar\n\nFirst, we need to set up the crossbar configuration. For this, you need access\nto the [ocs-site-configs](https://github.com/simonsobs/ocs-site-configs)\nrepository.\n\nEverything you need to do to run `sisock` can be done from the `templates` \ndirectory of this `ocs-site-configs` repo; if you are modifying anything for\nlocal development, you can make a copy of this directory elsewhere.\n\n\u003e **Important**: if you make a copy of these files, for security reasons do\n\u003e **not** do it in the root directory of your sisock clone, since it will put \n\u003e the encryption certificates in your docker image. This is not necessarily a \n\u003e problem on your local machine, but would compromise security if the image\n\u003e gets pushed to the public DockerHub.\n\nA default configuration can be found in the `templates` directory.\nYou will need first to run `setup_tls.py` to generate encryption certificates in\n`templates/.crossbar`\n\n\u003e **Important**: the ‘common name’ (CN) of the certificate should be\n\u003e `localhost` if you are doing local development, or else the hostname of your\n\u003e server if you are in production mode and will be serving over the internet.\n\u003e The certificates will be rejected if their CN does not match the real-life\n\u003e hostname. You will have to modify the above script if you do not want\n\u003e `localhost`. See `templates/.crossbar/README.md` for instructions on manually\n\u003e generating the certificates.\n\nThe `.crossbar/config.json` file should work out of the box, but advanced users\ncan tweak settings as they see fit.\n\n### 2. Setting up to use docker\n\n#### Get Dependencies\n\nYou will need to have `docker` and `docker-compose` installed on your system.\nThere are plenty of websites online that give instructions on how to do so, but\n\n```bash\n$ pip install docker docker-compose\n```\n\nshould do the trick.\n\n#### Create `docker-compose.yaml`\n\nWe will presume that you want to do local development in the following\ninstructions.\n\n\u003e If not, instead of using `templates/docker-compose_dev-mode.yaml` from\n\u003e `ocs-site-configs`, you could use\n\u003e `templates/docker-compose.yaml`. In this case, you actually\n\u003e don't need any of the `sisock` code locally, since it pulls all the docker \n\u003e images from `grumpy.physics.yale.edu`: see below for how to access images\n\u003e from this computer.\n\nSisock is normally run in `docker` containers using `docker-compose`. You will\nneed to create a YAML file for configuring this. Again, you can find an example\nin the [ocs-site-configs](https://github.com/simonsobs/ocs-site-configs) in\n`templates/docker-compose_dev-mode.yaml`.\n\nIf you want to use this file ‘out of the box’, define the following environment\nvariable:\n- `SISOCK_DIR` to point to the root of your `sisock` clone.\n\nYou can also customise which ports are used for various services, though the\ndefault values are probably fine on most machines.\n- `SISOCK_SQL_PORT` (default = 3308; chosen in order not to conflict with 3306\n  if you are already running an SQL server).\n- `SISOCK_HTTP_PORT` (default = 5000)\n- `SISOCK_GRAFANA_PORT` (default = 3000)\n\nAdditionally, if you want to use certain data node servers that read data from\nyour local disc, you should also define:\n- For `g3-reader`, define `${SISOCK_HK_DIR}`.\n- For `radiometer-server`, define `${SISOCK_RADIOMETER_DIR}`.\n- For `apex-weather-server`, define `${SISOCK_APEX_DIR}`.\n\n##### Make Sure You Have Access to the `so3g` Docker Image\n\nThe `g3_reader` data node server uses the `so3g` library, whose image is pulled\nby `components/g3_file_scanner/Dockerfile`. By default, this image is pulled\nfrom `grumpy` at Yale. You will need to do the following to enable this:\n\n```bash\n$ docker login grumpy.physics.yale.edu\n```\n\nThe username/password is a SO standard: ask Brian Koopman if you don't know it.\n\nAlternatively, you can create your own image locally and alter\n`components/g3file_scanner/Dockerfile` as appropriate.\n\n### 3. Go!\n\n#### Firing Things Up\n\nIf it doesn't yet exist, create the overarching network for the system:\n\n```bash\n$ docker network create --driver bridge sisock-net\n```\n\nTo run *all* the services, simply do:\n\n```bash\n$ docker-compose up\n```\n\n\u003e Note that this and all subsequent `docker-compose` commands assume your YAML\n\u003e file is named `docker-compose.yaml`. If you are using the out-of-the-box\n\u003e configuration from `ocs-site-configs`, use the `-f` flag to specify the\n\u003e filename.\n\nOr, you can select which services listed in `docker-compose.yaml` you want to\nrun. For instance, the following is a nice, minimal check that things are\nworking:\n\n```bash\n$ docker-compose up grafana sisock-http sensors-server\n```\n\n\u003e The commands above attach the `stdout` of all the containers to your\n\u003e terminal; you can terminate with good ol' ctrl-c. This mode is helpful for\n\u003e debugging. However, if you'd like to background this, throw the `-d` flag.\n\nGrafana should now be running: access it at `localhost:3000`. You'll need to \nconfigure the Grafana data source as the SimpleJson type with a URL of\n`http://sisock-http:5000`. (The user defined bridge network, `sisock-net`,\nenables DNS resolution by container name, in this case `sisock-http`, as is \ndefined in the `docker-compose.yaml` file.)\n\n#### Rebuilding\n\nIf you make modifications to the code, `docker-compose up` won't automatically\nupdate the images. You will need to do this yourself with `docker-compose\nbuild`. For instance, if you were working on the code in\n`components/data_node_servers/sensors/`, you would do:\n\n```bash\n$ docker-compose build sensors-server\n```\n\nTo rebuild *all* the images (normally not necessary), just do:\n\n```bash\n$ docker-compose build\n```\n\n#### Clean-up\n\nTo shutdown and cleanup, run:\n\n```bash\n$ docker-compose down\n```\n\nIf you'd like to remove the images as well run:\n\n```bash\n$ docker-compose down --rmi all\n```\n\nThis will not remove `sisock-net`. To do so:\n\n```bash\n$ docker network rm sisock-net\n```\n\nFurthermore, if you really don't want your saved grafana configuration, you can\nremove the grafana-storage container:\n\n```bash\n$ docker volume rm grafana-storage\n```\n\n## License\n\nThis project is licensed under the BSD 2-Clause License - see the [LICENSE.txt](LICENSE.txt) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonsobs%2Fsisock","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimonsobs%2Fsisock","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonsobs%2Fsisock/lists"}