{"id":34774868,"url":"https://github.com/cloudtruth/config-catalyst","last_synced_at":"2025-12-25T08:13:54.627Z","repository":{"id":224112893,"uuid":"762383533","full_name":"cloudtruth/config-catalyst","owner":"cloudtruth","description":"Extract parameters from your existing config and import to your CloudTruth organization","archived":false,"fork":false,"pushed_at":"2024-05-30T20:35:55.000Z","size":149,"stargazers_count":0,"open_issues_count":2,"forks_count":1,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-05-30T22:53:15.127Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloudtruth.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-23T17:03:14.000Z","updated_at":"2024-05-30T20:35:58.000Z","dependencies_parsed_at":"2024-05-13T20:49:10.952Z","dependency_job_id":"027e7db7-6b7e-4384-920f-c915e0598925","html_url":"https://github.com/cloudtruth/config-catalyst","commit_stats":null,"previous_names":["cloudtruth/dynamic-importer","cloudtruth/config-catalyst"],"tags_count":24,"template":false,"template_full_name":null,"purl":"pkg:github/cloudtruth/config-catalyst","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudtruth%2Fconfig-catalyst","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudtruth%2Fconfig-catalyst/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudtruth%2Fconfig-catalyst/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudtruth%2Fconfig-catalyst/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloudtruth","download_url":"https://codeload.github.com/cloudtruth/config-catalyst/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudtruth%2Fconfig-catalyst/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28024398,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-25T02:00:05.988Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-25T08:13:52.934Z","updated_at":"2025-12-25T08:13:54.620Z","avatar_url":"https://github.com/cloudtruth.png","language":"Python","readme":"![Logo for Config Catalyst by CloudTruth](https://github.com/cloudtruth/config-catalyst/blob/main/docs/img/repo-logo.png?raw=true)\n\n# Config Catalyst\n\n![CI](https://github.com/cloudtruth/config-catalyst/actions/workflows/ci.yaml/badge.svg)\n![Codecov](https://img.shields.io/codecov/c/github/cloudtruth/config-catalyst)\n![Docker pulls](https://img.shields.io/docker/pulls/cloudtruth/config-catalyst)\n![version](https://img.shields.io/docker/v/cloudtruth/config-catalyst)\n\nConfig Catalyst automatically converts static config files in your repos into parameterized templates.\nIt's the easiest way to \"pay down config tech debt\" with a single command.\n\nWe :heart: feedback, [bugs](https://github.com/cloudtruth/config-catalyst/issues/new), and [enhancement suggestions](https://github.com/cloudtruth/config-catalyst/issues/new).\n\nWe also have a #config-catalyst channel [on our Discord](https://discord.com/invite/eBZXm9Tzr7).\n\n# Motivation\nConfig Catalyst exists to solve this problem:\n\n\u003e\"I need to take this weird network config that Bob (who left five years ago) roughed out by hand and turn it into a crisp little YAML template with parameterized variables.\"\n\nStatic, hard-coded config is a form of \"tech debt\" many teams want to eliminate, but the \"pay it down\" process is tedious and time-consuming. Until now.\n\n# How it works\nConfig Catalyst works off of a local copy of your repo folder structure.\n1. Scans your repos and finds all the configuration files (JSON, YAML, ENV, HCL, TF_Var, etc..)\n1. Automatically creates parameterized templates with linked parameters and secrets to the parameter/secret store.\n1. Result: Finally, parameter and secret changes can be reflected in all the config files that use/depend on them.\n\nYou don't need to build all the plumbing anymore. Config file changes are independent of param/secret variable changes.\n\nNeed to manage values across multiple environments with one template? No problem!\n\nBonus: The variables and secrets can optionally be synced to Azure Key Vault, AWS Secrets Manager, ParameterStore, or Vault.\n\n# Features and supported file types\n\n**Current file type support includes**:\n* JSON\n* YAML\n* dotenv\n* tfvars\n* variables.tf\n\n**Not yet supported (but planned)**\n* ini\n* pkl\n\n# Installation\nThis utility is distributed as a Docker container and can be pulled from cloudtruth/config-catalyst on Docker Hub\n\nClone your repo(s) to local disk to allow Config Catalyst to find the supported file types.\n\nTo complete the process, you will need a CloudTruth [API token](https://app.cloudtruth.io/organization/api).\n\nTo get an API token, use your existing account or create a free [CloudTruth account](https://app.cloudtruth.io/signup).\n\n\n## Processing a directory tree (the easy method for fully automatic mode)\nYou can feed a directory of files into the `walk-directories` command, which will find all files matching the supplied types and parse them into CloudTruth config formats. If you supply your CLOUDTRUTH_API_KEY via docker, the data will be uploaded to your CloudTruth account.\n\n```\ndocker run --rm -it -e CLOUDTRUTH_API_KEY=\"myverysecureS3CR3T!!\" -v ${PWD}/files:/app/files cloudtruth/config-catalyst walk-directories --config-dirs /app/samples/ -t dotenv -t json -t tf\n```\n\n## Processing a single file\nAn example of how to process a .env file. This example assumes you have one .env file in the current directory that will be mounted through to the container\n\n```\ndocker run --rm -v ${PWD}:/app/files cloudtruth/config-catalyst walk-directories --config-dirs /app/files/ --file-type dotenv --output-dir /app/files/\n```\n\n# Advanced Usage\nThese examples break down the directory walking method into its individual components.\n\n## Processing several files\nAn example of how to process several .env files and create values for each environment\n```\ndocker run --rm -v ${PWD}/files:/app/files cloudtruth/config-catalyst process-configs \\\n    -p myproj -t dotenv \\\n    --default-values /app/samples/dotenvs/.env.default.sample \\\n    --env-values development:/app/samples/dotenvs/.env.dev.sample \\\n    --env-values staging:/app/samples/dotenvs/.env.staging.sample \\\n    --env-values production:/app/samples/dotenvs/.env.prod.sample \\\n    --output-dir /app/files/\n```\n\n## Editing template references\nSometimes this utility is too aggressive or you want a variable to remain hard-coded in your CloudTruth template. In that case, you can remove the references from the generated `.ctconfig` file and re-generate the template.\n\n```\ndocker run --rm -v ${PWD}/files:/app/files cloudtruth/config-catalyst regenerate-template --input-file /app/samples/.env.sample --file-type dotenv --data-file /app/files/.env.ctconfig --output-dir /app/files/\n```\n\n## Uploading data to CloudTruth\nOnce you are comfortable with your template and associated data, you're ready to upload to CloudTruth! Be sure to provide your CloudTruth API Key as an environment variable\n\n```\ndocker run --rm -e CLOUDTRUTH_API_KEY=\"myverysecureS3CR3T!!\" -v ${PWD}/files:/app/files cloudtruth/config-catalyst create-data --data-file /app/files/.env.ctconfig -m /app/files/.env.cttemplate -p \"Meaningful Project Name\"\n```\n\n# Command help\n\n**Automatic mode processes a directory tree and creates parameters, values, and templates.**\n```\nwalk-directories --help\n```\n\nThis command walks a directory, constructs templates and config data, and uploads to CloudTruth. It is an interactive version of the process_configs and create_data commands. As files are walked, the user will be prompted for project and environment names.\n\nOptions:\n\n--config-dirs - Full path to the directory to walk and locate configs\n\n-t, --file-types - Type of file to process. Must be one of 'dotenv', 'json', 'tf', 'tfvars', 'yaml'\n  \n--exclude-dirs - Directory to exclude from walking. Can be specified multiple times\n\n--create-hierarchy - If specified, project hierarchy will be created based on the directory hierarchy\n\n--parse-descriptions - Detect comments in the input file and use them for parameter descriptions\n\n-k - Ignore SSL certificate verification\n\n-c - Create missing projects and environments\n\n-u - Upsert values\n\n**Manual mode step 1 - Find and convert**\n```\nprocess-configs --help\n```\nOptions:\n\n-t, --file-type - Type of file to process. Must be one of: 'dotenv', 'json', 'tf', 'tfvars', 'yaml'\n\n--default-values - Full path to a file containing default values for the config data\n\n--env-values - Full path to a file containing environment specific values for the config data. Should be in the format of `env:file_path`\n\n-o, --output-dir - Directory to write processed output to. Default is current directory\n\n--parse-descriptions - Detect comments in the input file and use them for parameter descriptions\n\n-p, --project - CloudTruth project to import data into\n\n**Manual mode step 2 - Upload**\n```\ncreate-data --help\n```\nOptions:\n\n-d, --data-file Full path to config data file generated from process_configs command\n\n-m, --template-file Full path to template file generated from process_configs command\n\n-k Ignore SSL certificate verification\n\n-c Create missing projects and environments\n\n-u Upsert values\n\n**Manual mode step 3 - Edit template prior to upload**\n```\nregenerate-template --help\n```\nOptions:\n\n--default-values - Full path to a file containing default values for the config data\n\n--env-values - Full path to a file containing environment specific values for the config data. Should be in the format of `env:file_path`\n\n-t, --file-type - Type of file to process. Must be one of: 'dotenv','json', 'tf', 'tfvars', 'yaml'\n\n-d, --data-file - Full path to config data file generated from process_configs command\n\n# Development\n1. Set up a virtualenv\n1. From your checkout of the code, `pip install -e .[dev]`\n\n# Testing\nTest code lives in `src/tests` and uses [click.testing](https://click.palletsprojects.com/en/8.1.x/testing/) as the entrypoint for all commands and processors. There are additional unit tests for the api client code, which heavily leverages mocks for the CloudTruth API. See examples in `tests.fixures.requests` for more.\n\nTo run unittests, run `pytest` from within your virtualenv.\n\nPre-commit is installed in this repo and should be used to verify code organization and formatting. To set it up, run `pre-commit install` in your virtualenv\n\n# Known issues\n1. On Mac OSX the \"{PWD}\" in the \"docker run -v ${PWD}/files:/app/files\" command argument may not be reliable. Replace \"{PWD}\" with a fully qualified path.\n\n# Contributing\nIssues, pull requests, and discussions are welcomed. Please vote for any issues tagged with [needs votes](https://github.com/cloudtruth/config-catalyst/issues?q=is%3Aissue+is%3Aopen+label%3A%22needs+votes%22)\n\nSee `dynamic_importer.processors` and the subclasses within for examples of the current design. TL;DR - if you can convert the source into a dict, `BaseProcessor._traverse_data` should handle most of the heavy lifting.\n\n# Further reading\nCloudTruth [documentation](https://docs.cloudtruth.com/)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudtruth%2Fconfig-catalyst","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloudtruth%2Fconfig-catalyst","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudtruth%2Fconfig-catalyst/lists"}