Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dreitier/conditional-regex-search-and-replace-action
Executes conditional search and replace operations on a set of files. Strings are replaced by matching regular expressions.
https://github.com/dreitier/conditional-regex-search-and-replace-action
argocd continuous-delivery continuous-deployment continuous-integration github-action github-actions gitops glob regex regular-expression search-and-replace structure
Last synced: 5 days ago
JSON representation
Executes conditional search and replace operations on a set of files. Strings are replaced by matching regular expressions.
- Host: GitHub
- URL: https://github.com/dreitier/conditional-regex-search-and-replace-action
- Owner: dreitier
- License: mit
- Created: 2022-11-12T12:51:58.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-02-10T12:30:48.000Z (about 2 years ago)
- Last Synced: 2024-10-31T10:43:47.910Z (4 months ago)
- Topics: argocd, continuous-delivery, continuous-deployment, continuous-integration, github-action, github-actions, gitops, glob, regex, regular-expression, search-and-replace, structure
- Language: PHP
- Homepage: https://dreitier.com
- Size: 193 KB
- Stars: 1
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
# conditional-regex-search-and-replace-action
This GitHub action executes conditional search and replace operations on multiple files. Strings in files are replaced, if they match a given regular expressions.
Originally, this action has been created to update strings - like Docker image tag and Git branches - in Helm charts and Kustomize templates inside of Git repositories used for GitOps projects.
When using the [environment-per-folder strategy](https://dreitier.com/knowledge-base/continuous-delivery-and-deployment/using-environment-per-folder-directory-structure-for-gitops-projects-with-argo-cd-and-helm), updating the designated file for one or multiple environments can be cumbersome.
If you want to use this action indeed for updating GitOps repositories, please look at [this article](https://dreitier.com/knowledge-base/continuous-delivery-and-deployment/promoting-new-docker-images-to-helm-charts-in-github-repositories-with-github-actions) on how to execute it with `repository_dispatch`.
## Usage
```yaml
uses: dreitier/conditional-regex-search-and-replace-action
env:
with:
mappings: "docker_image_tag==main.* {THEN_UPDATE_FILES} **dev/values.yaml=docker_image_tag_regex"
docker_image_tag: "${{ github.sha }}"
docker_image_tag_regex: "imageTag: \\\"(?.*)\\\""
```
### Mandatory arguments
| Argument | Description |
| --- | --- |
| `mappings` | DSL to define search-and-replace operations, see below |
### Optional arguments
| Argument | Default | Description |
|-----------------------------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
| `directory` | `$CWD` | Directory, in which to operate. By default, the current working directory is used. |
| `dump` | `` | If `1`, it dumps the provided configuration |
| `if_no_match_fail` | `0` | If the action has not modified any file and `if_no_match_fail` is `1`, it will fail with exit code `3` |
| `if_well_known_vars_missing_fail` | `0` | If `1`, it fails with exit code `2` if none of docker_image_tag, git_branch or git_tag is provided |
| `updated_file_suffix` | `` | If set, any changes will be written to another file the path of the original file and that suffix |
| `register_custom_regexes` | `` | A comma-separated list of custom regular expressions to register |
| `register_custom_variables` | `` | A comma-separated list of custom variables to register |
| `commit` | `0` | If there has been any modified files (see below, `outputs.total_modified_files`), it will commit the changes to the Git repository |
| `commit_template` | `` | Template to use for the commit message. You can use the [Blade template engine](https://laravel.com/docs/9.x/blade) to dynamically specify the message. |
| `committer_name` | `` | Name of committer, only relevant if `commit` is present |
| `committer_email` | `` | E-mail of committer, only relevant if `commit` is present |
| `skip-regexes-autodetect` | `0` | Skip auto-detection of regexes for environment variables ending with `_REGEX` |
| `skip-variables-autodetect` | `0` | Skip auto-detection of variables for environment variables ending with `_VAR` |
### Well-known variables and regular expressions
Due the original requirement of this action, you can use the following built-in variables:
| Argument | Description |
| --- | --- |
| `__always__` and `__true__` | Both variables are automatically set to `1` |
| `docker_image_tag` | Docker image tag created by upstream repository |
| `docker_image_tag_regex` | Regular expression to modify occurences of Docker image tags in globbed files |
| `git_tag` | Git tag created by upstream repository |
| `git_tag_regex` | Regular expression to modify occurences of Git tags in globbed files |
| `git_branch` | Git branch modified in upstream repository |
| `git_branch_regex` | Regular expression to modify occurences of Git branch in globbed files |
There is __no__ need to use `docker_image_tag_regex`, `git_tag_regex` and `git_branch_regex`, those regular expressions are automatically registered. You can register additional regular expressions as you like.
#### Registering additional regular expressions
Additional regular expressions are automatically detected if the environment variable has the suffix (`_REGEX`). If your regular expression has not that suffix, you can use the option `register_custom_regexes`:
```yaml
uses: dreitier/conditional-regex-search-and-replace-action
with:
# ...
register_custom_regexes: check_for_customer_myregex
env:
# this one is automatically detected
MY_CUSTOM_REGEX: "/a: b/"
# this is detected by using `register_custom_regexes`
CHECK_FOR_CUSTOMER_MYREGEX: "/customer: (\d+)/"
```
To register multiple regular expressions without the `_REGEX` suffix, use a comma (`,`) for separation:
```yaml
uses: dreitier/conditional-regex-search-and-replace-action
with:
# ...
register_custom_regexes: check_for_customer_myregex, project_id_myregex
env:
CHECK_FOR_CUSTOMER_MYREGEX: "/customer: (?\d+)/"
PROJECT_MYID_REGEX: "/project: (?\d+)/"
```
#### Registering additional variables
To provide additional variables aside from `docker_image_tag`, `git_tag` and `git_branch` you have to register those variables.
- Variables are automatically detected for environment variables having the suffix `_VAR`.
- If your variable cannot have the suffix `_VAR`, you have to use `register_custom_variables`.
For referencing the `customer_number` variable of the previous section, you can use either:
```yaml
uses: dreitier/conditional-regex-search-and-replace-action
with:
# ...
env:
# this one is automatically detected by using the _VAR suffix.
CUSTOMER_NUMBER_VAR: "555"
```
or
```yaml
uses: dreitier/conditional-regex-search-and-replace-action
with:
# ...
register_custom_variables: customer_number
env:
# custom registered variable
CUSTOMER_NUMBER: "555"
```
In both cases, you can reference the variable later on with `customer_number`. The `_VAR` suffix is removed.
To register multiple variables without the `_VAR` suffix, use a comma (`,`) for separation:
```yaml
uses: dreitier/conditional-regex-search-and-replace-action
with:
# ...
register_custom_variables: customer_number, project_id
env:
CUSTOMER_NUMBER: "555"
PROJECT_ID: "5550555
```
#### Git commit message templates
When using `commit_template`, you can either put in some static string or you can use the [Blade template engine](https://laravel.com/docs/9.x/blade) to dynamically specify the message.
Each of the well-known and custom variables is passed as variable to the template. If you have a custom variable `customer_number`, you can access the value with `{{ $customer_number }}`.
Use a static string:
```yaml
uses: dreitier/conditional-regex-search-and-replace-action
with:
# ...
commit_template: 'A new version has been born.'
```
Reference a well-known variable:
```yaml
uses: dreitier/conditional-regex-search-and-replace-action
with:
# ...
docker_image_tag: '1.0.0'
commit_template: 'chore: Update to Docker image tag {{ $docker_image_tag }}'
```
Reference a custom variable:
```yaml
uses: dreitier/conditional-regex-search-and-replace-action
with:
# ...
docker_image_tag: '1.0.0'
commit_template: 'chore: Update to Docker image tag {{ $docker_image_tag } for customer {{ $customer_number))'
env:
CUSTOMER_NUMBER_VAR: "555" # or set it to an empty string
```
Expressions:
```yaml
uses: dreitier/conditional-regex-search-and-replace-action
with:
# ...
docker_image_tag: '1.0.0'
commit_template: 'chore: Update to Docker image tag {{ $docker_image_tag }}@if(!empty($customer_number)) for customer {{ $customer_number }}@endif'
env:
CUSTOMER_NUMBER_VAR: "" # or a non-empty string
```
#### Mappings
Mappings specify, if one or multiple regular expressions match, how other regular expressions are applied. It's basically
```yaml
uses: dreitier/conditional-regex-search-and-replace-action
with:
mappings: ' OR {THEN_UPDATE_FILES} = {AND} =&'
```
In each regular expression to apply, you have access to all well-known and custom variables through named capturing groups:
```yaml
uses: dreitier/conditional-regex-search-and-replace-action
docker_image_tag: '1.0.0'
docker_image_tag_regex: 'imageTag: \"(?.*)\".*'
```
### Outputs
| Output | Default | Description |
| --- | --- |----------------------------------------------------|
| `total_modified_files` | `0` | Number of modified (target) files. |
| `$your_variable` | `` | Any of your provded well-known or custom variables |
You can reference the output variables in your GitHub Action workflow like this:
```yaml
jobs:
my_job_with_dedicated_commit_and_push:
runs-on: ubuntu-latest
steps:
- name: Update repository
id: search_and_replace_op
uses: dreitier/conditional-regex-search-and-replace-action
with:
# ...
# instead of using this action's commit hook, we use the more flexible EndBug/add-and-commit if any file has been modified
- name: Commit and push
if: ${{ steps.search_and_replace.outputs.total_modified_files >= 1 }}
uses: EndBug/add-and-commit@v7
with:
author_name: build@internalcom
author_email: build@internal
message: "There have been {{ steps.search_and_replace.outputs.total_modified_files }} modified files"
```
To access e.g. the `customer_number` from before
```yaml
jobs:
my_job_with_dedicated_commit_and_push:
runs-on: ubuntu-latest
steps:
- name: Update repository
id: search_and_replace_op
uses: dreitier/conditional-regex-search-and-replace-action
with:
docker_image_tag: '1.0.0'
# ...
env:
CUSTOMER_NUMBER_VAR: "555"
- name: Commit and push
if: ${{ steps.search_and_replace.outputs.total_modified_files >= 1 }}
uses: EndBug/add-and-commit@v7
with:
author_name: build@internal
author_email: build@internal
message: "Customer number {{ steps.search_and_replace.outputs.customer_number }} has {{ steps.search_and_replace.outputs.total_modified_files }} modified files for tag {{ steps.search_and_replace.outputs.docker_image_tag }}"
```
## Examples
### Updating strings in multiple files
In a folder structure like this
```
├── asia
│ ├── dev
│ │ ├── values.yaml
│ └── prod
│ ├── values.yaml
├── eu
│ ├── dev
│ │ └── values.yaml
│ └── prod
```
the files `eu/dev/values.yaml` and `asia/dev/values.yaml` have be modified. The original content of both `values.yaml` files looks like this:
```yaml
custom_parameter: custom_value
imageTag: v0.1.0
other_parameter: other_custom_value
```
We want to update the `imageTag` value in both `values.yaml` files from `v0.1.0` to `v0.2.0`:
```yaml
uses: dreitier/conditional-regex-search-and-replace-action
env:
with:
mappings: "docker_image_tag==v.* {THEN_UPDATE_FILES} **dev/values.yaml=docker_image_tag_regex"
docker_image_tag: "v0.2.0"
docker_image_tag_regex: "imageTag: \\\"(?.*)\\\""
```
After running `conditional-regex-search-and-replace-action`, both files will look like this:
```yaml
custom_parameter: custom_value
imageTag: v0.2.0
other_parameter: other_custom_value
```
#### Transformed into pseudo code
Sometimes it's easier to read pseudo code instead of skimming through YAML files. The above YAML's definition can be read as
```
if $docker_image_tag =~ /v.*/ then
$files = glob_files_with_matcher("**dev/values.yaml")
foreach $files as $file
foreach $line in $file
if $line =~ /imageTag: \"(?.*)\"/ then
$line = "imageTag: \"$docker_image_tag\""
write_line_to_file($file, $line)
fi
endforeach
endforeach
endif
```
### Do not check a regex for a given pattern
If you do not have to check one or multiple known variable for a regex, you can simply use the well-known variables `__always__` or `__true__`:
```yaml
with:
mappings: "__always__ {THEN_UPDATE_FILES} **dev/values.yaml=docker_image_tag_regex"
docker_image_tag: "v0.2.0"
docker_image_tag_regex: "imageTag: \\\"(?.*)\\\""
```
By using `__always__` or `__true__` the replacement will be applied, regardless of one of the variable's value.
### Updating multiple values at the same time
If you need to apply multiple regexes at the same time, you can chain those regexes with:
```yaml
with:
mappings: "docker_image_tag==v.* {THEN_UPDATE_FILES} **dev/values.yaml=docker_image_tag_regex&git_branch_regex"
git_branch: "feature/165-integrate-kerberos-auth"
git_branch_regex: "branch: \\\"(?.*)\\\""
docker_image_tag: "v0.2.0"
docker_image_tag_regex: "imageTag: \\\"(?.*)\\\""
```
A `values.yaml` of this:
```yaml
custom_parameter: custom_value
imageTag: v0.2.0
branch: old-branch
```
will be transformed into this:
```yaml
custom_parameter: custom_value
imageTag: v0.2.0
branch: feature/165-integrate-kerberos-auth
```
#### Transformed into pseudo code
```
if $docker_image_tag =~ /v.*/ then
$files = glob_files_with_matcher("**dev/values.yaml")
foreach $files as $file
foreach $line in $file
if $line =~ /imageTag: \"(?.*)\"/ then
$line = "imageTag: \"$docker_image_tag\""
write_line_to_file($file, $line)
fi
if $line =~ /branch: \"(?.*)\"/ then
$line = "git_branch: \"$git_branch\""
write_line_to_file($file, $line)
fi
endforeach
endforeach
endif
```
### Check, if at least one variable matches
Sometimes you want to update files if at least one of multiple conditions is valid. `mappings` support a simple `{OR}` operator.
```yaml
with:
mappings: "docker_image_tag==v.* {OR} git_branch==feature\/.* {THEN_UPDATE_FILES} **dev/values.yaml=docker_image_tag_regex&git_branch_regex"
git_branch: "feature/165-integrate-kerberos-auth"
git_branch_regex: "branch: \\\"(?.*)\\\""
docker_image_tag: "latest"
docker_image_tag_regex: "imageTag: \\\"(?.*)\\\""
```
#### Transformed into pseudo code
```
if $docker_image_tag =~ /v.*/ or $git_branch =~ /feature\/.*/ then
$files = glob_files_with_matcher("**dev/values.yaml")
foreach $files as $file
foreach $line in $file
if $line =~ /imageTag: \"(?.*)\"/ then
$line = "imageTag: \"$docker_image_tag\""
write_line_to_file($file, $line)
fi
if $line =~ /branch: \"(?.*)\"/ then
$line = "git_branch: \"$git_branch\""
write_line_to_file($file, $line)
fi
endforeach
endforeach
endif
```
### Multiple mappings
This action's DSL for defining mappings does __not__ feature a `{AND}` condition in the left-hand part of `{THEN_UPDATE_FILES}`. Instead, you can chain multiple mappings together. Use De Morgan's law for complex conditions ;-)
Each of those mappings will be separately evaluated:
```yaml
with:
mappings: "docker_image_tag==v.* {THEN_UPDATE_FILES} **dev/values.yaml=docker_image_tag_regex {NEXT_MAPPING} git_branch==feature\/.* {THEN_UPDATE_FILES} **dev/values.yaml=git_branch_regex"
git_branch: "feature/165-integrate-kerberos-auth"
git_branch_regex: "branch: \\\"(?.*)\\\""
docker_image_tag: "v2.0.0"
docker_image_tag_regex: "imageTag: \\\"(?.*)\\\""
```
#### Transformed into pseudo code
```
if $docker_image_tag =~ /v.*/ then
$files = glob_files_with_matcher("**dev/values.yaml")
foreach $files as $file
foreach $line in $file
if $line =~ /imageTag: \"(?.*)\"/ then
$line = "imageTag: \"$docker_image_tag\""
write_line_to_file($file, $line)
fi
endforeach
endforeach
endif
if $git_branch =~ /feature\/.*/ then
$files = glob_files_with_matcher("**dev/values.yaml")
foreach $files as $file
foreach $line in $file
if $line =~ /branch: \"(?.*)\"/ then
$line = "git_branch: \"$git_branch\""
write_line_to_file($file, $line)
fi
endforeach
endforeach
endif
```
## FAQ
### Why not using something more GitOps-esk like Argo CD Image Updater?
Using Argo CD Image Updater is totally fine but might have some drawbacks:
- By default, Argo CD Image Updater checks the Docker registries every 2 minutes. There might be a good chance of hitting API request limits, e.g. with Docker Hub.
- Setting up Argo CD Image Updater might be difficult, depending upon the environment.
- Argo CD Image Updater does not support complex search-and-replace operations.
When using `conditional-regex-search-and-replace-action`, you can either configure Webhooks in your GitOps repository to notify Argo CD or let Argo CD pull the latest version.
### Why are you not using Bash/$your_favorite_language_here for this action?
Before introducing this action, I've developed a Bash script for updating various GitOps repositories. Different projects had different requirements: The Bash script was no longer maintainable.
Using Laravel Zero and Pest for developing testable GitHub Actions looks fine to me.
### But I can do everything with awk & sed.
Just keep using `awk` & `sed`, it's fine!
### Why looks the DSL like it does?
Due to the nature of complexity and how parameters are passed from the GitHub workflow to single actions, it was the first approach I came up with.
I also thought about using Blade templates for doing the mapping stuff.
## DSL spec
```
next_mapping = "{NEXT_MAPPING}"
if_match_then_execute = "{THEN_UPDATE_FILES}"
glob = $valid_glob
regex = $valid_quoted_regex
variable_reference = (lower_case_chars | digits | '_')+
regex_reference = (lower_case_chars | digits | '_')+
matches = "=="
matcher = variable_reference (matches regex)?
or_matcher = "{OR}"
matchers = matcher (or_matcher matcher)+
regex_references = regex_reference ('&' regex_reference)+
assign_to = '='
replacer = glob assign_to regex_references
replacers = replacer (and_replacer replacer)+
and_replacer = "{AND}"
mapping = matchers if_match_then_execute replacers
multiple_mappings = mapping (next_mapping mapping)+
```
## Support
This software is provided as-is. You can open an issue in GitHub's issue tracker at any time. But we can't promise to get it fixed in the near future. If you need professionally support, consulting or a dedicated feature, please get in contact with us through [our website](https://dreitier.com).
## Contribution
Feel free to provide a pull request.
## License
This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details.