{"id":31045123,"url":"https://github.com/dativebase/old-pyramid-deployment","last_synced_at":"2025-09-14T16:57:59.625Z","repository":{"id":55108076,"uuid":"290682801","full_name":"dativebase/old-pyramid-deployment","owner":"dativebase","description":"OLD Pyramid Deployment (OPD) using Docker and Babashka","archived":false,"fork":false,"pushed_at":"2021-01-11T03:47:07.000Z","size":27,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-04-14T23:58:11.077Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dativebase.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-27T05:17:24.000Z","updated_at":"2021-03-31T04:49:11.000Z","dependencies_parsed_at":"2022-08-14T12:10:49.435Z","dependency_job_id":null,"html_url":"https://github.com/dativebase/old-pyramid-deployment","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dativebase/old-pyramid-deployment","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dativebase%2Fold-pyramid-deployment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dativebase%2Fold-pyramid-deployment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dativebase%2Fold-pyramid-deployment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dativebase%2Fold-pyramid-deployment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dativebase","download_url":"https://codeload.github.com/dativebase/old-pyramid-deployment/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dativebase%2Fold-pyramid-deployment/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275136757,"owners_count":25411709,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-14T02:00:10.474Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-14T16:57:56.986Z","updated_at":"2025-09-14T16:57:59.603Z","avatar_url":"https://github.com/dativebase.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"================================================================================\n  OLD Pyramid Deployment (OPD) using Docker and Babashka\n================================================================================\n\n.. warning:: I am currently (2021-01-09) considering moving away from this\n             heavily automated approach and starting fresh, but building on this.\n\nThis project deploys the Online Linguistic Database (OLD) using Docker images\nand containers, and Babashka for shell scripting and service deployment\norchestration.\n\nIt should ultimately be able to automate the installation of both Pylons and\nPyramid OLD installations. Note that while the \"original\" (Pylons) version of\nthe OLD is officially deprecated, some OLD instances are still running on it as\nof this writing (2020-08-26.)\n\n\nTODOs\n================================================================================\n\n- Write deployment scripts for the Pyramid OLD.\n- Move this code to the OLD repository itself and set up CI/CD with CircleCI.\n- Set up log shipping.\n- Set up automated backups of MySQL and filesystem data.\n- Set up monitoring for downtime.\n- Ensure reboots bring the system back up.\n- Store images in Docker Hub.\n\n\nHow to Deploy the OLD using the OPD (this tool)\n================================================================================\n\nThe OPD can be used to deploy both versions of the OLD, the Pylons version and\nthe Pyramid one. Some steps are common to both versions. These are given below.\nWhen the steps diverge, new sections begin.\n\nFirst, clone this repo (the OPD source)::\n\n    $ git clone https://github.com/dativebase/old-pyramid-deployment\n    $ cd old-pyramid-deployment\n\nNext, clone the source code for the Pyramid and Pylons OLDs to\n``src/old-pyramid/`` and ``src/old/``, respectively::\n\n    $ git submodule update --init --recursive\n\nTODO: fix the following paragraph to reflect the Pylons/Pyramid split reality.\n\nThis project assumes that the ``private/`` directory contains two sub-directories\nwith analogous structures. The ``private/old-stores/`` directory should contain\nsubdirectories whose names are the OLD instances (and simultaneously the names of\nthe databases of those instances) of each OLD being deployed. The\n``private/old-mysqldumps/`` directory should contain one MySQL dump file for each\nOLD instance directory mentioned above; each file should have the same name as\nthe directory, but with the ``.dump`` extension. Example::\n\n    $ ls private/old-stores/\n    blaold\n    dpold\n    $ ls private/old-mysqldumps/\n    blaold.dump\n    dpold.dump\n\n\nHow to Deploy the Pylons OLD\n================================================================================\n\nThe following command will build Docker images for MySQL, the Pylons OLD and\nNginx and then bring up a container running MySQL, one running Nginx, and one\nfor each Pylons OLD under ``private/old-stores/``::\n\n    $ bb opd.clj deploy-new-pylons \u003cENV\u003e\n\nThe value of ``\u003cENV\u003e`` determines which top-level key under ``:environments`` in\nthe config map of ``CONFIG.edn`` will be used.\n\nThe above command is equivalent to running the following commands in sequence.\n\n1. Create the ``opd`` Docker network idempotently::\n\n       $ bb opd.clj create-network \u003cENV\u003e\n\n2. Build the MySQL image and bring up a container based on it named\n   ``opd-mysql``. This container will have access to ``private/mysqldumps/`` from\n   which it can ingest the source OLD MySQL dump files. It will also write its\n   MySQL persistence files to ``private/mysql-data`` on the host. To create the\n   MySQL container idempotently::\n\n       $ bb opd.clj up-build-mysql \u003cENV\u003e\n\n3. Create the Pylons OLD MySQL databases in the running MySQL container. Running\n   the following will create one database for each dump file in\n   ``private/mysqldumps/``. Warning: this will drop any database in the MySQL\n   container that has the same name as any of the dump files in the\n   aforementioned directory.::\n\n       $ bb opd.clj bootstrap-dbs \u003cENV\u003e\n\n4. Build the OLD Docker image::\n\n       $ bb opd.clj build-old \u003cENV\u003e\n\n5. Repair the Pylons OLD config files. These are the ``production.ini`` files\n   under each subdirectory of ``private/old-stores``, e.g.,\n   ``private/old-stores/blaold/production.ini``. This sets the host value to\n   ``0.0.0.0`` and the ``sqlalchemy.url`` value to\n   ``mysql://\u003cMYSQL_USER\u003e:\u003cMYSQL_PASSWORD\u003e@\u003cMYSQL_CONTAINER\u003e:3306/\u003cOLD_DB_NAME\u003e?charset=utf8``.::\n\n       $ bb opd.clj repair-old-configs \u003cENV\u003e\n\n6. Bring up the OLD containers. Each OLD container will have a name like\n   ``opd-old-\u003cOLD_NAME\u003e``, where ``\u003cOLD_NAME\u003e`` is the name of the OLD dump\n   file. Note that this is idempotent insofar as a container will not be spun up\n   if one with the same name already exists::\n\n       $ bb opd.clj up-olds \u003cENV\u003e\n\n7. Build the Nginx Docker image and create a container from it named\n   ``opd-nginx``. The Nginx process within this container will be configured to\n   serve OLDs (using the HTTPS scheme at port 443) at subpaths mathing the OLD\n   names, i.e., ``https://\u003cDOMAIN\u003e:443/\u003cSUBPATH\u003e/``, e.g.,\n   ``https://oldpyramid:443/blaold/``. This Nginx config file is visible on the\n   host at ``src/nginx/etc/old``. Its contents are determined by inspecting the\n   OLD config files that were repaired in a previous step above::\n\n        $ bb opd.clj up-build-nginx \u003cENV\u003e\n\n\nHow to Deploy the Pyramid OLD\n================================================================================\n\nThe following command will build Docker images for MySQL, the Pyramid OLD and\nNginx and then bring up a container running MySQL, one running Nginx, and one\nfor each Pyramid OLD under ``private/old-stores-pyr/``::\n\n    $ bb opd.clj deploy-new-pyramid \u003cENV\u003e\n\nThe value of ``\u003cENV\u003e`` determines which top-level key under ``:environments`` in\nthe config map of ``CONFIG.edn`` will be used.\n\nThe above command is equivalent to running the following commands in sequence.\n\n1. Create the ``opd`` Docker network idempotently::\n\n       $ bb opd.clj create-network \u003cENV\u003e\n\n2. Build the MySQL image and bring up a container based on it named\n   ``opd-mysql-pyr``. This container will have access to\n   ``private/mysqldumps-pyr/`` from which it can ingest the source OLD MySQL dump\n   files. It will also write its MySQL persistence files to\n   ``private/mysql-data-pyr`` on the host. To create the MySQL container idempotently::\n\n       $ bb opd.clj up-build-mysql-pyr \u003cENV\u003e\n\n3. Create the Pyramid OLD MySQL databases in the running MySQL container. Running\n   the following will create one database for each dump file in\n   ``private/old-mysqldumps-pyr/``. Warning: this will drop any database in the MySQL\n   container that has the same name as any of the dump files in the\n   aforementioned directory.::\n\n       $ bb opd.clj bootstrap-dbs-pyr \u003cENV\u003e\n\n4. Build the OLD Pyramid Docker image::\n\n       $ bb opd.clj build-old-pyr \u003cENV\u003e\n\n5. Repair the Pyramid OLD config files. These are the ``production.ini`` files\n   under each subdirectory of ``private/old-stores-pyr``, e.g.,\n   ``private/old-stores-pyr/cooold/production.ini``. This sets the host value to\n   ``0.0.0.0`` and the ``sqlalchemy.url`` value to\n   ``mysql://\u003cMYSQL_USER\u003e:\u003cMYSQL_PASSWORD\u003e@\u003cMYSQL_CONTAINER\u003e:3306/\u003cOLD_DB_NAME\u003e?charset=utf8``.::\n\n       $ bb opd.clj repair-old-pyr-configs \u003cENV\u003e\n\n   .. note:: I do not believe the above step is necessary with Pyramid OLDs.\n             These OLDs use the in-source configuration file ``config.ini`` and\n             customize it with environment variables.\n\n6. Bring up the OLD containers. Each OLD container will have a name like\n   ``opd-old-pyr-\u003cOLD_NAME\u003e``, where ``\u003cOLD_NAME\u003e`` is the name of the OLD dump\n   file. Note that this is idempotent insofar as a container will not be spun up\n   if one with the same name already exists::\n\n       $ bb opd.clj up-olds-pyr \u003cENV\u003e\n\n7. Build the Nginx Docker image and create a container from it named\n   ``opd-pyr-nginx``. The Nginx process within this container will be configured\n   to serve OLDs (using the HTTPS scheme at port 443) at subpaths mathing the OLD\n   names, i.e., ``https://\u003cDOMAIN\u003e:443/\u003cSUBPATH\u003e/``, e.g.,\n   ``https://oldpyramid:443/blaold/``. This Nginx config file is visible on the\n   host at ``src/nginx/etc/old``. Its contents are determined by inspecting the\n   OLD config files that were repaired in a previous step above::\n\n        $ bb opd.clj up-build-nginx-pyr \u003cENV\u003e\n\n        Network connectivity blocked by security group rule: DefaultRule_DenyAllInBound\n\n\nNotes about the Deployments\n================================================================================\n\nOLD instance container logs can be found under::\n\n    $ private/old-stores/\u003cOLD_NAME\u003e/application.log\n\ne.g.,::\n\n    $ private/old-stores/gitold/application.log\n\n.. warning:: It appears to me that the types of paths listed above do not\n             actually contain any useful logs. I believe the most interesting\n             logs (at least in the Pylons OLDs) are the Nginx access logs. See\n             below.\n\nThe Nginx container's logs are under ``/var/lib/docker/containers/`` in a\ndirectory and file named using the container's ID, which can be found as\nfollows::\n\n    $ docker ps --no-trunc -aqf \"name=opd-nginx\"\n    1b7623bda2621d6135ad71cbaaed515f0bb373a88d875fd568ce8cd2beeb5edc\n\nIn this case, the logs are at::\n\n    /var/lib/docker/containers/1b7623bda2621d6135ad71cbaaed515f0bb373a88d875fd568ce8cd2beeb5edc/1b7623bda2621d6135ad71cbaaed515f0bb373a88d875fd568ce8cd2beeb5edc-json.log\n\nNew filesystem data (e.g., audio files, phonologies) will be stored under::\n\n    $ ls private/old-stores/\n\nThe MySQL data directory is mounted to the host at::\n\n    $ ls private/mysql-data/\n\n\n\n\nWARNING: Beware Below!\n================================================================================\n\nTake all of the documentation below with a grain of salt. It needs review.\n\n\nUsage\n================================================================================\n\nThis tutorial assumes that you are running on a system with Docker installed.\n\nHigh-level Steps:\n\n1. Clone the OPD source.\n2. Clone the OLD Pyramid source.\n2. Build and run the ``opd`` Docker daemon container\n3. Use ``opd`` to download and install the Babashka ``bb`` script to the host.\n4. Use ``bb`` to:\n\n   a. Build Docker images for the required services:\n\n      i. MySQL\n      ii. OLD Pyramid\n      iii. Nginx (web server)\n\n   d. Launch long-running dockerized processes for the configuration defined in\n      opd.edn. In overview, launch processes running:\n\n      i. MySQL (listening on port 3306)\n      ii. Nginx (listening on ports 80 and 443)\n\n\nBuild and run the ``opd`` Docker daemon container\n--------------------------------------------------------------------------------\n\nBuild the ``opd:1.0`` (OLD Pyramid Deployment) image which brings in Babashka\n(bb) and other system tools::\n\n    $ docker build -t opd:1.0 .\n\nNow run the container in daemon mode, mounting your home directory (which in this\nexample is /home/rancher/) to a mirror path in the container , and naming it\n``opd``::\n\n    $ docker run --rm -d -v \"/home/rancher:/home/rancher\" --name opd opd:1.0\n\nNow you can execute tools such as ``bb`` and ``tree`` from within the ``opd``\ncontainer against files under your home directory. Example::\n\n    $ docker exec -it opd bb\n    Babashka v0.1.3 REPL.\n    Use :repl/quit or :repl/exit to quit the REPL.\n    Clojure rocks, Bash reaches.\n    user=\u003e (* 8 8)\n    64\n\nOptionally, set your shell profile to alias ``opd`` to ``docker exec -it opd``.\nIn RancherOS, for example, this would mean modifying ~/.profile as follows::\n\n    $ cat ~/.profile\n    alias opd=\"docker exec -it opd\"\n    $ source ~/.profile\n\nNow we can just call ``opd \u003cCMD\u003e``, e.g.,::\n\n    $ opd bb\n    user=\u003e\n\nNow we can download the Babashka ``bb`` binary to the host machine, using cURL\nfrom the OPD container::\n\n    $ opd curl -s -L https://github.com/borkdude/babashka/releases/download/v0.1.3/babashka-0.1.3-linux-static-amd64.zip -o /home/rancher/downloads/bb.zip\n    $ cd /home/rancher/downloads\n    $ unzip bb.zip\n    $ mv bb /usr/bin/bb\n\n\nBuild the MySQL Docker image\n--------------------------------------------------------------------------------\n\nUse ``bb`` to build the MySQL Docker image::\n\n    $ bb opd.clj build-mysql\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdativebase%2Fold-pyramid-deployment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdativebase%2Fold-pyramid-deployment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdativebase%2Fold-pyramid-deployment/lists"}