{"id":41142744,"url":"https://github.com/dataunitylab/semantic-regex","last_synced_at":"2026-01-22T18:51:50.533Z","repository":{"id":139414115,"uuid":"491668306","full_name":"dataunitylab/semantic-regex","owner":"dataunitylab","description":null,"archived":false,"fork":false,"pushed_at":"2026-01-12T21:42:20.000Z","size":561,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-13T01:55:41.995Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dataunitylab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-05-12T21:07:39.000Z","updated_at":"2026-01-12T21:42:24.000Z","dependencies_parsed_at":null,"dependency_job_id":"a461c414-0d7d-4061-91d1-67921b96771a","html_url":"https://github.com/dataunitylab/semantic-regex","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dataunitylab/semantic-regex","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataunitylab%2Fsemantic-regex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataunitylab%2Fsemantic-regex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataunitylab%2Fsemantic-regex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataunitylab%2Fsemantic-regex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dataunitylab","download_url":"https://codeload.github.com/dataunitylab/semantic-regex/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataunitylab%2Fsemantic-regex/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28668542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-22T17:07:18.858Z","status":"ssl_error","status_checked_at":"2026-01-22T17:05:02.040Z","response_time":144,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-22T18:51:50.400Z","updated_at":"2026-01-22T18:51:50.482Z","avatar_url":"https://github.com/dataunitylab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Learning from Uncurated Regular Expressions\n\n[![CI](https://github.com/dataunitylab/semantic-regex/actions/workflows/ci.yml/badge.svg)](https://github.com/dataunitylab/semantic-regex/actions/workflows/ci.yml)\n[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/dataunitylab/semantic-regex/main.svg)](https://results.pre-commit.ci/latest/github/dataunitylab/semantic-regex/main)\n\nDependencies of all Python code are managed with [Pipenv](https://pipenv.pypa.io/en/latest/) and can be installed with `pipenv install`.\nNote that the dataset from the [Sherlock](https://github.com/mitmedialab/sherlock-project) project should be available in a copy of the repository in alongside the directory for this project.\n[`jq`](https://jqlang.github.io/jq/) is also required for some JSON processing.\n\n## Model training\n\n1. Download all regular expressions from regex101\n\n`./download_patterns.sh`\n\nThis will create a directory `regex101` which has the individual regular expressions and `patterns.json` which contains only the expressions strings.\n\n2. Compile a database of all the downloaded regular expressions\n\n`pipenv run python compile_db.py \u003c patterns.json \u003e patterns_final.json`\n\n`patterns_final.json` is a subset of the expressions in `patterns.json` which are supported by Hyperscan.\nThis step will also create `hs.db` which are the compiled regular expressions that can be used during preprocessing.\n\n3. Preprocess the data to generate feature vectors\n\n`pipenv run python preprocess.py train`\n\nThis will generate `preprocessed_train.txt` which contains all the feature vectors extracted using the regular expression extracted using the regular expressions.\n\n4. Train the model on the extracted features\n\n`pipenv run python train.py`\n\nThe model architecture will be stored in `nn_model_sherlock.json` with the weights in `nn_model_sherlock.weights.keras`.\n\n## Evaluation\n\nFirst, the test data must be preprocessed.\n\n`pipenv run python preprocess.py test`\n\nThen, the model can be evaluated.\n\n`pipenv run python test.py`\n\n## Model explanation\n\nExplains for predictions for an individual class can be generated using [SHAP](https://shap.readthedocs.io/en/latest/).\nFirst, follow the steps for training the model above.\nThe file `patterns_final.json` will be used to match the patterns back to the original regular expressions.\n\n`pipenv run python find_patterns.py \u003e pattern_ids.txt`\n\nThis file of pattern IDs will then be used to label the SHAP plot with the ID of the regular expression.\nTo generate the SHAP plot in `shap.png`, run the command below where `\u003cclass_name\u003e` is one of the semantic types defined by Sherlock.\n\n`pipenv run python explain.py \u003cclass_name\u003e`\n\nThe IDs displayed in the SHAP plot can be used to reference the regular expressions by ID in the `regex101/patterns` directory or viewing it directly on regex101 at the URL `https://regex101.com/library/\u003cID\u003e`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdataunitylab%2Fsemantic-regex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdataunitylab%2Fsemantic-regex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdataunitylab%2Fsemantic-regex/lists"}