{"id":13531008,"url":"https://github.com/0xVavaldi/ruleprocessorY","last_synced_at":"2025-04-01T19:31:05.359Z","repository":{"id":59998704,"uuid":"439478685","full_name":"0xVavaldi/ruleprocessorY","owner":"0xVavaldi","description":"Rule Processor Y is a next-gen Rule processor with complex multibyte character support built to support Hashcat","archived":false,"fork":false,"pushed_at":"2024-04-23T21:08:06.000Z","size":604,"stargazers_count":30,"open_issues_count":3,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-05-21T13:49:32.373Z","etag":null,"topics":["cybersecurity","hashcat","hashcat-rules","johntheripper","jtr","mdxfind","password-analysis","password-cracking"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/0xVavaldi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-17T22:49:07.000Z","updated_at":"2024-08-01T07:32:50.644Z","dependencies_parsed_at":"2024-01-07T13:09:16.859Z","dependency_job_id":"1ae075b1-9180-4293-b1e6-4cada950a968","html_url":"https://github.com/0xVavaldi/ruleprocessorY","commit_stats":null,"previous_names":["0xvavaldi/ruleprocessory"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xVavaldi%2FruleprocessorY","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xVavaldi%2FruleprocessorY/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xVavaldi%2FruleprocessorY/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xVavaldi%2FruleprocessorY/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/0xVavaldi","download_url":"https://codeload.github.com/0xVavaldi/ruleprocessorY/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246700207,"owners_count":20819834,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cybersecurity","hashcat","hashcat-rules","johntheripper","jtr","mdxfind","password-analysis","password-cracking"],"created_at":"2024-08-01T07:00:58.985Z","updated_at":"2025-04-01T19:31:00.349Z","avatar_url":"https://github.com/0xVavaldi.png","language":"C++","funding_links":[],"categories":["Hashcat"],"sub_categories":["Rule tools"],"readme":"# ruleprocessorY\n![Main Build](https://github.com/0xVavaldi/ruleprocessorY/actions/workflows/cmake.yml/badge.svg) \n![CodeQL](https://github.com/0xVavaldi/ruleprocessorY/actions/workflows/codeql-analysis.yml/badge.svg)\n\nRule Processor Y is a next-gen Rule processor with multibyte character support built for hashcat. It applies rules to wordlists in order to transform them in whichever way the user pleases.\nThe key feature of this ruleprocessor is that it allows a user to quickly do multibyte or multi-character replacements such as replacing the e with é or the other way around for normalization of wordlists.\n\n## Requirements\n```\nsudo apt-get install build-essential cmake git\n```\n\n## Quickstart\nIf you receive an error regarding your cmake version, edit CMakeLists.txt and lower the cmake_minimum_required to match your version, this will generally not cause an issue. \n```\ngit clone https://github.com/0xVavaldi/ruleprocessorY\ncd ruleprocessorY\ncmake .\nmake\n./ruleprocessorY -h\n./ruleprocessorY -w rockyou.txt -r example_rules.rule\n./ruleprocessorY -w rockyou.txt -r rules/best66.rule --hashcat-input --hashcat-output\n```\nIf you want to use this with hashcat, you must specify the `--hashcat-input` flag to accept hashcat-style input files and `--hashcat-output` if you wish to use the output of the tool in hashcat.\n\n## Rule writing\nRules are stored using a tab separated format (TSV), which is CSV, but with tabs. Within each line you can utilize the standard rules you might be familiar with in Hashcat (https://hashcat.net/wiki/doku.php?id=rule_based_attack). An example is shown below:\n```tsv\nc\nu   $1  $2  $3\nl\n$1  $2\n$2  $0  $0  $0\n^a  ^m\n```\n\nAdditionally, we support multi-character rules. Allowing the appending, prepending or replacing of multiple characters. To do this we utilize the / delimiter, similar to some unix tools.\nA known issue is being unable to use the / character. This is a planned feature where we will make use of $HEX[]\nBelow is a sample file with comments explaining the construct in an example. \n```tsv\nl   $2022\nu   $2000\n^prefix     $suffix\ns/a/4   sa@     # this is two different formats replacing one character with another, both are supported\ns/alpha/beta\no0beta\no/0/beta\n```\n\n### Hashcat cross-comptability\nFinally, using the `--hashcat-input` and/or `--hashcat-output` flag we support hashcat formatted rules (space/no delimiter). This will automatically attempt to parse the rules and convert them into the TSV format.\nIn doing so it will replace tabs with \\x09 and spaces with \\x20. Hashcat supports this notation and the rules will be cross compatible if you were to replace all tabs in the output file with spaces. (or removing tabs entirely).\n\n### Note on duplicate candidates\nCandidates matching the original word are never printed unless the `:` rule is specified. This is done to prevent duplicates. Example: Using `l` will only print candidates that have an uppercase character and as a result are different from the original plaintext. This can be unfavorable when working with rejection rules. In that case a `:` must be added as a first rule. An example is shown below where the goal is to reject all candidates containing the word \"test\". To match case toggled candidates the `l` rule is added before the match test. To ensure all candidates are printed and not just rules with uppercase the `:` rule is added, which will force all candidates to be printed.\n```bash\nruleprocessorY.exe -r rule.txt --optimize-no-op --hashcat-input --hashcat-output \u003e optimized_rule.txt\n```\n```tsv\n:\nl\n!test\n$1 $2 $3\n```\n\n\n## Rule Optimizing\nRules generated or used by hashcat can contain contradictions or operations that do not make sense. RuleProcessorY is capable of cleaning up your rules and structuring it out for you. In total we current support 3 forms of optimization. Starting off we can look at some operations that 'do nothing', we refer to this as a no-op (no-operation). Using the `--optimize-no-op` we remove these. \n```tsv\nT0      T0\n$1      ]\n^1      [\n```\n\nRules generated or used by hashcat can contain partial contradictions or can be rewritten to be more efficient. This can happen in different ways, but for computational sake we won't entirely rewrite rules. Instead, we will look if the rule can be performed using less operations. The `--optimize-same-op` will remove these.\n\n```bash\nruleprocessorY.exe -r rule.txt --optimize-same-op \u003e optimized_rule.txt\n```\n```tsv\n$a      $b      ]\n{       }       $b\n[       *97     O57\nT6      $n      O65\n,0      ,6      Y4\n```\nThe resulting rules could look like this, where operations that don't contribute have been removed:\n```tsv\n$a\n$b\n[   O57\n$n  O65\n,6  Y4\n```\n\nFinally we will look through all rules and find two rules that perform the same action. This can be a very computational intensive operation and requires 2-3x the size of the wordlist in RAM. A warning is displayed for this. An extra flag has been added to support large rule files, but will take exponentially long to complete.\n`--optimize-similar-op` will perform the optimization, and `--optimize-slow` as an EXTRA flag, will utilize the memory-limited variant. Included is an example of a rule file before and after optimization. It will keep the rule with the least operations it comes across that performs change that has not been seen before. If two rules perform the same operations with the same actions it will take the first occurrence. \n\nReplace rules `s` that replace one word with another are skipped (`s/alpha/beta`), `s/a/beta` is taken into account.\n\n```bash\nruleprocessorY.exe -r rule.txt --optimize-similar-op \u003e optimized_rule.txt\nruleprocessorY.exe -r rule.txt --optimize-all \u003e optimized_rule.txt\n```\n```tsv\n$a  $b\n$ab\n$abc\n$a  ^a\n^a  $a\n$$  Z2\n$$  $$  $$\nD1  $1  $2  $3  D0\n$1  D0  $3  D0  $3\n[   [   $1  $2  $3\n$6  [   $9\n[   $6  $9\n```\n\nThe optimized version:\n```tsv\n$ab\n$abc\n$a  ^a\n$$      Z2\n[       [       $1      $2      $3\n[       $6      $9\n```\n\n## Rule Optimizing / Comparison\nAdditionally, you can compare one rule against another and optimize rule files against each other. Removing rules from file A that also appear in file B. To do so we can use the `--optimize-compare` flag. Example command to remove all rules from fordy10k.txt that also appear in best64.rule.\n```bash\nruleprocessorY.exe --hashcat-input --hashcat-output --optimize-all -r fordy10k.txt --optimize-compare best64.rule\n```\n\n## Rule Optimizing with wordlists\nAlthough technically supported - the use is heavily discouraged. Due to the nature of the computing problem, it is nearly impossible to optimize rules for a specific wordlist without spending a significant amount of money or time. The following command will optimize the `dive.rule` file for the rockyou.txt wordlist. \nHowever, expect to use about 350PB of RAM. \n```bash\nruleprocessorY.exe --hashcat-input --hashcat-output --optimize-all -r dive.rule -w rockyou.txt\n```\n\nAlternatively you can have it be computed on the fly. This uses approximately 7-8GB of RAM, but also takes significantly longer to process.\nIn the end you will gain relatively little extra performance for all the time invested. If you wish to optimize using this, use a very small wordlist.\n```bash\nruleprocessorY.exe --hashcat-input --hashcat-output --optimize-all -r dive.rule -w rockyou.txt --optimize-slow\n```\n\n## Rule Optimizing more\nIt is possible (alternatively to specifying a wordlist as described above) to optimize a wordlist more. This will use a smaller 'validation dictionary'.\nUsing this method will result in some loss of cracks / founds. In return, you will optimize the rules more. This can be favorable in a few scenarios when working with generated rules that work on edge-cases like `@\\x02` and do not apply to 'normal passwords' commonly.\n\n**Use this with caution.**\n```bash\nruleprocessorY.exe --hashcat-input --hashcat-output --optimize-all -r dive.rule -w rockyou.txt --optimized-words\n```\n\n\n## Optimize debugging\nTo debug what changes have been made, the `--optimize-debug` flag can be used. This will display what changes are made to STDERR by default. Adding a file name can force the output to a text-file. `--optimize-debug debug.txt`.\nExample output:\n```yml\nBefore: $!      o9H     x27\nAfter:  $!      x27\nBefore: $*      @*      +3\nAfter:  @*      +3\nBefore: $/      @/      i3k\nAfter:  @/      i3k\nBefore: $1      +7      D7\nAfter:  $1      D7\nBefore: $1      D6      *45     '6\nAfter:  $1      *45     '6\nBefore: $1      DA      @1\nAfter:  DA      @1\n\nKept:   $$      Z2\nDeleted:        $$      $$      $$\nKept:   D1      $1      $2      $3      D0\nDeleted:        [       [       $1      $2      $3\nKept:   $6      [       $9\nDeleted:        [       $6      $9\nKept:   $1      +0      Z1\nDeleted:        +0      $1      $1\nKept:   o2a\nDeleted:        $5      o2a     ]\nKept:   $3      $2      $1      D0\nDeleted:        [       $3      $2      $1\nKept:   $2      $0      $1      $1      [\nDeleted:        [       $2      $0      $1      $1\nKept:   $l      $o      $l\nDeleted:        $l      $l      $o      K\n```\n\nThe action when a new rule is added is not displayed to not overload the debug output. This can be re-enabled in the code by searching for \"Kept new\" in main.cpp\n\n**Credits**\np͞é͜ng̸u̡͘iń͢͞k̴è͢͜e̛p͠è͢r for helping test, debug, and help write the hashcat rule parser.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0xVavaldi%2FruleprocessorY","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F0xVavaldi%2FruleprocessorY","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0xVavaldi%2FruleprocessorY/lists"}