{"id":18829820,"url":"https://github.com/kartta-labs/project","last_synced_at":"2025-04-14T03:41:00.011Z","repository":{"id":47232993,"uuid":"219325633","full_name":"kartta-labs/Project","owner":"kartta-labs","description":"Infrastructure","archived":false,"fork":false,"pushed_at":"2021-09-07T19:11:13.000Z","size":1282,"stargazers_count":47,"open_issues_count":4,"forks_count":13,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-03-27T17:46:15.046Z","etag":null,"topics":["configuration","docker","kubernetes-deployment"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kartta-labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-11-03T15:54:25.000Z","updated_at":"2023-06-11T16:38:13.000Z","dependencies_parsed_at":"2022-09-23T17:21:45.732Z","dependency_job_id":null,"html_url":"https://github.com/kartta-labs/Project","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kartta-labs%2FProject","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kartta-labs%2FProject/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kartta-labs%2FProject/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kartta-labs%2FProject/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kartta-labs","download_url":"https://codeload.github.com/kartta-labs/Project/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248818525,"owners_count":21166438,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["configuration","docker","kubernetes-deployment"],"created_at":"2024-11-08T01:46:40.532Z","updated_at":"2025-04-14T03:40:59.988Z","avatar_url":"https://github.com/kartta-labs.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kartta Labs Suite\n\nThis project contains everything you need to spin up a complete suite of Kartta\nLabs web applications for development purposes on your local system in docker.  It also\ncontains configuration files and scripts needed to launch the applications in\nGKE (Google Kubernetes Engine).\n\nThis project does not contain the code for the individual applications -- it\njust contains the overall scripts and configuration files for launching and\nmanaging the applications together.  The code for each application is stored in\nits own repository in http://github.com/kartta-labs.  The scripts in this\nproject will take care of cloning the applications repos for you.\n\n\n## Initial Setup\n\nDo these things just once to get set up.\n\n1. Install Docker\n\n\n2. Clone this repo and cd into it\n\n   ```\n   git clone https://github.com/kartta-labs/Project\n   cd Project\n   ```\n\n3. Generate a secrets file by running (optionally in an isolated python environment).\n\n   ```\n   pip install -r requirements.txt\n   ./makesecrets\n   ```\n   This will create the file `./container/secrets/secrets.env` by copying `example-secrets.env`.\n   For running locally with docker for development, you don't need to edit the generated file,\n   although you may customize the configuration by editing it if you want.\n\n   You can provide a set of intial values to `makesecrets` by passing it a YAML file if you want:\n   ```\n   ./makesecrets values.yml\n   ```\n   This will cause `makesecrets` to substitute values found in `values.yml` for the corresponding\n   variable when generating `./container/secrets/secrets.env`.  Note: you can't use `values.yml` to\n   add additional variables -- the substitution only happens for variables already present in\n   `example-secrets.env`.\n\n   If you want to use code from a repository other than the offical http://github.com/kartta-labs\n   repository for one or more of the applications, change the value of the appropriate *_REPO\n   variable(s) in ./container/secrets/secrets.env before continuing.\n\n4. Run the bootstrap script\n   ```\n   ./bootstrap.sh\n   ```\n   This will take 45-60 minutes or so but only has to be done once.  It might ask you to\n   enter your password at various points (because it uses sudo, since many docker commands\n   need to run as root), so check on it occasionally.\n\n\n## Running\n\nOnce `bootstrap.sh` finishes, to run the suite of applications, do\n\n```\n./start\n```\nLeave this command running in one terminal; use a separate terminal for everything else below.  To shut everything down\nwhen you are done, or so you can restart everything, hit Ctrl-C.\n\nOnce all the containers have started (usually 1 or 2 minutes), visit http://localhost/e/ in your browser\nfor editor, or http://localhost/w/ for warper.\n\n### Things to note\n\n1. See the file ARCHITECTURE.md in this directory for a general overview of\n   how these applications are configure.\n\n1. As currently configured this will only work on port 80 on localhost, which\n   unfortunately means you can't port-forward to your local machine if you're\n   accessing the host remotely, since only root can forward privileged ports.\n   That means that in order to view this in a browser you'll have to\n   remotedesktop to the host machine and run the browser there.  It's possible\n   to reconfigure everything to use a public port other than 80, though, and/or\n   a hostname other than 'localhost', but that will involve some careful\n   tweaking of the various nginx docker configs.\n\n1. The apps currently run on port 8888 using http (not https).  If you want\n   to view using a local browser, use ssh to tunnel port 8888 from your remote\n   host to your local machine: `ssh -L 8888:localhost:8888 remote.machine.net`.\n\n1. Run `docker ps` to see a list of all the running containers; the output will look similar to this:\n   ```\n   CONTAINER ID   IMAGE                  COMMAND                  PORTS                                            NAMES\n   cd904223a9d1   project_fe             \"nginx -g 'daemon of…\"   0.0.0.0:80-\u003e80/tcp                               project_fe_1\n   3539235734cd   project_editor         \"/editor-container-s…\"   0.0.0.0:32936-\u003e80/tcp                            project_editor_1\n   34d109561bca   project_warper      \"/warper-containe…\"   0.0.0.0:32938-\u003e80/tcp, 0.0.0.0:32937-\u003e3000/tcp   project_warper_1\n   c2916d2bb811   project_cgimap         \"/cgimap-container-s…\"   0.0.0.0:32935-\u003e8000/tcp                          project_cgimap_1\n   1d039d94f71d   project_oauth-proxy    \"/root/go/bin/oauth2…\"   0.0.0.0:4180-\u003e4180/tcp                           project_oauth-proxy_1\n   afdeaf8b1429   mdillon/postgis        \"docker-entrypoint.s…\"   5432/tcp                                         project_warper-db_1\n   750b55e3ae1f   redis                  \"docker-entrypoint.s…\"   6379/tcp                                         project_redis_1\n   707c200a13e8   postgres:11            \"docker-entrypoint.s…\"   5432/tcp                                         project_editor-db_1\n   ```\n\n1. Each container corresponds to a \"service\" entry in the `docker-compose.yml` file.\n   Each one is like a single server, or VM.  The ones whose IMAGE name start with 'project_' correspond\n   to services Kartta runs in production:\n\n     * fe: frontend nginx server, receives all incoming requests and routes them\n       internally to the relevant other service\n     * oauth-proxy: handles all authentication\n     * editor: editor-website rails app\n     * warper: warper rails app\n\n   The other containers are local instances of postgres/postgis/redis servers which are needed by\n   the project_ services.  In production we use GCP managed versions of these services.\n\n   You can use the `./dsh` script to start a bash shell in any of the\n   containers; it takes a single arg which is the name of the service (without\n   the 'project_' prefix or '_1' suffix).  For example `./dsh editor`.  This is like\n   \"ssh-ing\" to the VM; once in the shell, you can poke around with the\n   filesystem and running processes for the container.  In particular, you can\n   use this to view the nginx config, application log files, or running processes.\n\n1. All the config files and application code for the editor and warper apps,\n   as well as all nginx conf files, are \"mounted\" inside the containers from\n   their corresponding location in the 'Project' directory on your workstation (the\n   docker \"host\").  You can edit these files with an editor on the host, and the\n   changes are immediately visible from inside the running containers.  The mappings\n   that determine which files/dirs get mounted in each container, and where, are\n   given by the \"volumes\" directives in the `docker-compose.yml` file.\n\n1. Inside any of the nginx containers (editor, warper, fe), run\n   `nginx -s reload`\n   to restart nginx after changing either the nginx conf or any of the\n   application files.\n\n1. `docker-compose` creates a local IP network in which each container is known\n   by its service name from the `docker-compose.yml` file.  For example, the\n   editor container can be referred to by the name `editor` when constructing\n   hostnames and/or urls for use in any of the containers.  These names are not\n   available on the host -- only inside the containers.\n\n1. The `docker-compose.yml` file in the top level `Project` directory references various\n   environment variables which must be set in order for it to work.  These variables\n   are set in the secrets file `./container/secrets/ecrets.env`, and the `./start`\n   script takes care of loading these values into the environment automatically.\n   You should always use `./start` to launch the containers -- it runs `docker-compose up`\n   for you, after loading the environment variables from the secrets file.\n   If you need to otherwise run `docker-compose` (e.g. to re-build an image for an\n   application you're developing), run the `./dcwrapper` script rather than running\n   `docker-compose` directly.\n\n## Misc helpful commands\n\n* Shut down any remaining running containers:\n  ```\n  sudo docker-compose down\n  ```\n\n* Rebuild the docker images before starting the containers:\n  ```\n  docker-compose up --build\n  ```\n  This uses docker's build cache so it usually runs much faster than the 30 min\n  or so needed to run the bootstrap script initially.  Docker is pretty good\n  (but not perfect) about knowing what parts of each build need to be re-done\n  based on which files have changed.\n\n* Completely remove all docker artifacts, including built images and cache data:\n  ```\n  sudo docker system prune -a\n  ```\n  Do this if you want to force docker to rebuild EVERYTHING the next time you\n  run `docker-compose up` (this is like a much stronger version of\n  `docker-compose up --build`).\n\n* The database containers store their data in files in the `tmp` dir (the one at the top level of the `Project` dir).\n  You can clear the data from a database by removing the corresponding file from `tmp`.  Note that if you do that,\n  the database will need to be re-initialized by re-running the relevant commands from `bootstrap.sh`.\n  If you are at all unsure about what's needed to do this, it's probably safer to start over by doing\n  `sudo docker system prune -a` to clear everything out of docker, clone a new copy of this `Project` repo,\n  and re-run `bootstrap.sh`.\n\n\n## Kubernetes Deployments\n\nThe 'k8s' directory contains configuration files and scripts for deploying the\nsuite of applications in GKE (Google Kubernetes Engine).  This is intended for\nrunning production servers, and/or for testing the production deployment\nprocess.  If you just want to run the suite so you can work on one or more of\nthe applications, use the docker process described above.\n\n\n1. Create a new GCP project, or decide on an existing one to use.\n\n2. Clone a fresh copy of this 'Project' repo.  This should be a new copy -- not one that you\n   have previously used with docker for local development.\n\n3. Generate a secrets file by running\n\n   ```\n   ./makesecrets\n   ```\n\n4. Edit `./container/secrets/secrets.env` and set all the required values at the top of the file;\n   see the comments in the file for details.\n\n5. Warper has an optional feature that causes it to run OCR on uploaded maps in an attempt to initially\n   place them in an approximate location.  This feature is disabled by default.  If you want to\n   enable this feature:\n     * Go to https://pantheon.corp.google.com/apis/credentials and create a new API key\n     * Edit `./container/secrets/secrets.env` to insert the value of that API key for\n       the variable MAPWARPER_GOOGLE_MAPS_KEY.  Also change the value of MAPWARPER_ENABLE_OCR_JOB\n       to \"true\".\n\n6. Run `./k8s/kbootstrap.sh`.  It may take up to an hour to run.  When it finishes, it will\n   print out a message with the IP address of the running server (this is the IP of the load\n   balancer).\n\n7. Create a DNS entry which associates the generated IP address with the SERVER_NAME\n   you chose above.\n\n8. The vector tile server deployment is handled by a separate script.  If you also want to\n   create a vector tile server deployment, run `./k8s/kvector-bootstrap.sh`.  Do this _after_\n   running `./k8s/kbootstrap.sh`; `./k8s/kvector-bootstrap.sh` will not work correctly if\n   `./k8s/kbootstrap.sh` has not been run first.\n\n9. Note that both `./k8s/kbootstrap.sh` and `./k8s/kvector-bootstrap.sh` edit the secrets file\n   (container/secrets/secrets.env) to add information such as IP addresses and names of\n   resources they create, or values for passwords they generate.  You should guard that secrets\n   file carefully -- it will be needed when making any changes to the deployment.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkartta-labs%2Fproject","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkartta-labs%2Fproject","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkartta-labs%2Fproject/lists"}