{"id":23201600,"url":"https://github.com/mitodl/ol-infrastructure","last_synced_at":"2025-08-19T00:31:59.949Z","repository":{"id":36990065,"uuid":"267145677","full_name":"mitodl/ol-infrastructure","owner":"mitodl","description":"Infrastructure automation code for use by MIT Open Learning","archived":false,"fork":false,"pushed_at":"2024-12-16T15:53:47.000Z","size":75416,"stargazers_count":48,"open_issues_count":109,"forks_count":4,"subscribers_count":24,"default_branch":"main","last_synced_at":"2024-12-16T16:45:42.047Z","etag":null,"topics":["aws","devops","infrastructure"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mitodl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-26T20:31:47.000Z","updated_at":"2024-12-16T14:59:23.000Z","dependencies_parsed_at":"2023-10-13T09:05:00.847Z","dependency_job_id":"3b901935-9ade-4323-be7a-2e85c369503e","html_url":"https://github.com/mitodl/ol-infrastructure","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitodl%2Fol-infrastructure","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitodl%2Fol-infrastructure/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitodl%2Fol-infrastructure/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mitodl%2Fol-infrastructure/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mitodl","download_url":"https://codeload.github.com/mitodl/ol-infrastructure/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230292713,"owners_count":18203650,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","devops","infrastructure"],"created_at":"2024-12-18T15:16:06.481Z","updated_at":"2024-12-18T15:16:06.961Z","avatar_url":"https://github.com/mitodl.png","language":"Python","readme":"# Overview\nThis repository is a monorepo for managing the configuration and deployment of services managed by MIT Open Learning engineering. It uses a combination of Pulumi and PyInfra to build a pure-python deployment stack to enable more developer-friendly access to creating and modifying the systems that power the applications that we build and serve.\n\nAll infrastructure provisioning performed via Pulumi is located under `src/ol-infrastructure/` and all configuration management written in PyInfra lives under `src/bilder`.\n\n# Infrastructure Management\n\n## Getting Started\nThis is a Pulumi project, so the first step is to install the Pulumi CLI along with the relevant language\nprotocol. Instructions [here](https://www.pulumi.com/docs/get-started/install/).\n\nTo install and manage the relevant dependencies this project uses [Poetry](https://python-poetry.org/). After installing\nthe Poetry CLI simply run `poetry install`.\n\nWe use the S3 state backend for Pulumi, so after installing the Pulumi CLI run `pulumi login s3://mitol-pulumi-state`.\n\n## Structure\n\nThis is a monorepo of the Pulumi code that we use to manage the infrastructure that powers MIT Open Learning\n\nComponent resources are located under `src/ol-infrastructure/components/` with a descriptive name that makes it evident what the intent of the\ncomponent is.\n\nConcrete implementations of infrastructure are located under `src/ol-infrastructure/infrastructure/` with a descriptive name that makes it\nevident what is contained in that project.\n\nManagement of resources that rely on foundational infrastructure being provisioned, but\nwhich supports the operation of applications is located under\n`src/ol-infrastructure/substructure`. An Example of this includes Consul prepared queries.\n\nProvisioning of all of the resources needed to support and deploy a specific application\nis located under `src/ol-infrastructure/applications/` with a name that indicates the\napplication being managed (e.g. `concourse`).\n\nEach component or concrete infrastructure that is more complex than a single resource will include a `diagram.py` file\nthat uses the [diagrams](https://diagrams.mingrammer.com/) package to illustrate the system structure that it creates.\n\n## Nomenclature\n\nPulumi organizes code into `Projects` which represent a deployable unit. Within a project they have a concept of\n`Stacks` which are often used as a mapping for different environments. Each module underneath `src/ol-infrastructure/infrastructure/` and\n`src/ol-infrastructure/applications/` is its own `Project`, meaning that it will have a `Pulumi.yaml` definition of that project. Each `stack`\nhas its own yaml file in which the configuration for that stack is defined.\n\n## Conventions\n\nStack names should be a dot-separated namespaced representation of the project path, suffixed with an environment\nspecifier in the form of QA or Production. The capitalization is important as it will be used directly to interpolate\ninto tag objects. It is easier to start with QA and Production and then call `.lower()` than it is to build a dictionary\nmapping the lowercase versions to their properly capitalized representation.\n\nThe dotted namespace allows for peaceful coexistence of multiple projects within a single state backend, as well as\nallowing for use of [stack references](https://www.pulumi.com/docs/tutorials/aws/aws-py-stackreference/) between\nprojects.\n\nThe infrastructure components should be properly namespaced to match the stack names. For example, the project for\nmanaging VPC networking in AWS is located at `src/ol-infrastructure/infrastructure/aws/network/` and the corresponding stacks are defined as\n`aws.network.QA` and `aws.network.Production`.\n\n\n## Executing\n\nIn order to run a deployment, you need to specify the project and the stack that you would like to deploy. From the root\nof the repository, you can run `pulumi -C src/ol_infrastructure/path/to/module/ up`. If you haven't already selected the\nstack, it will ask you to interactively select the stack which you are deploying.\n\n\n## Adding a new Project\n\nFor each deployable unit of work we need to have a Pulumi project defined. The Pulumi CLI has a `new` command, but that\nintroduces extra files that we don't want. The minimum necessary work to signal that a given directory is a project is\nthe presence of a `Pulumi.yaml` file and a `__main__.py` where the deployment code is located. The contents of the\n`Pulumi.yaml` file should follow this structure:\n\n```\nname: ol-infrastructure-dagster-application # Change the name here to be descriptive of the purpose of this project\nruntime: python # not necessary to change\ndescription: Pulumi project for deploying the stack of services needed by the Dagster ETL framework # update the description accordingly\nbackend:\n  url: s3://mitol-pulumi-state/ # should not be changed\n```\n\nCreate the directory path according to the conventions detailed above and then create these files. Once that is done you\nwill need to create the stack definitions (again according to the above conventions). To do this, `cd` to the target\ndirectory and run the command `pulumi stack init --secrets-provider=awskms://alias/infrastructure-secrets-qa \u003cyour.dotted.stack.name.QA\u003e`\n\nNow you're ready to start writing the code that will define the target deployment.\n\n\n# Configuration Management\nIn order to streamline the management of infrastructure required to support the services\nrun by MIT Open Learning we have implemented our configuration management logic using\nPyInfra. This has the benefit of being pure Python, allowing us to take advantage of the\nbroad ecosystem that it provides including linting, testing, and integrations.\n\n## Project Structure\nAny given service is unlikely to be composed of a single service, instead requiring a\nvariety of components to be used together. In order to make the composition of a running\nsystem easier to reason about, we decompose the responsibilities for any given piece of\ntechnology into a self-contained component. These components live under the\n`src/bilder/components/` directory. Each component provide at minimum the logic needed to\ninstall and configure the associated service. In addition, each component should also\nprovide the logic needed to upgrade an instance of the service, the SystemD unit files\nneeded to register the service, and any logic required to perform maintenance on the\nassociated technology.\n\n## Configuration Objects\nA common requirement across all configuration management systems is a method to provide\nsettings at build/runtime to populate config files, determine which logical paths to\ntake, etc. A problem that often comes from managing these settings is uncertainty around\nthe constraints for each value, and the ability to easily merge default values with user\nsupplied information. In order to reduce the complexity of this situation and provide\nuseful constraints we use [Pydantic](https://pydantic-docs.helpmanual.io/) models with\nrelevant validators to contain the values needed by each component.\n\nEach model defined for a given component will inherit from\n`ol_configuration_management.lib.model_helpers.OLBaseSettings`. This class inherits from\nthe Pydantic `BaseSettings` class and sets the `case_sensitive` model setting to\n`False`. This allows us to easily use environment variables to override attributes of a\nmodel at build time.\n\n## Directory Locations\nWhen faced with a decision of where to locate any given files or directories we default\nto following the [Filesystem Hierarchy\nStandard](https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard). This typically\nmeans that applications or services which are deployed by downloading an archive or\ncloning a software repository should go under `/opt`, configuration files under `/etc/`,\nand data files under `/var/lib`.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmitodl%2Fol-infrastructure","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmitodl%2Fol-infrastructure","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmitodl%2Fol-infrastructure/lists"}