{"id":31778823,"url":"https://github.com/iamdevnd/transformer-robustness-benchmark","last_synced_at":"2026-02-16T16:44:37.156Z","repository":{"id":304723209,"uuid":"1019739886","full_name":"iamdevnd/Transformer-Robustness-Benchmark","owner":"iamdevnd","description":"Transformer Robustness Benchmark – DistilBERT under Adversarial \u0026 Stress Attacks","archived":false,"fork":false,"pushed_at":"2025-07-14T20:01:59.000Z","size":2110,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-18T03:30:24.051Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iamdevnd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-14T19:48:29.000Z","updated_at":"2025-09-17T22:47:51.000Z","dependencies_parsed_at":"2025-07-15T00:00:53.400Z","dependency_job_id":"b4f028cd-a285-463f-bc7e-f3c3f116b232","html_url":"https://github.com/iamdevnd/Transformer-Robustness-Benchmark","commit_stats":null,"previous_names":["doddanikhil/transformer-robustness-benchmark","iamdevnd/transformer-robustness-benchmark"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/iamdevnd/Transformer-Robustness-Benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamdevnd%2FTransformer-Robustness-Benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamdevnd%2FTransformer-Robustness-Benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamdevnd%2FTransformer-Robustness-Benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamdevnd%2FTransformer-Robustness-Benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iamdevnd","download_url":"https://codeload.github.com/iamdevnd/Transformer-Robustness-Benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamdevnd%2FTransformer-Robustness-Benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002962,"owners_count":26083491,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-10T06:58:20.466Z","updated_at":"2025-10-10T06:58:26.694Z","avatar_url":"https://github.com/iamdevnd.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Benchmarking Sentiment Classifiers Against Adversarial and Stress Attacks\n\n## Objective\nThis project benchmarks the robustness of a DistilBERT-based sentiment classifier against various adversarial threats, including typographical errors, flood attacks with long junk texts, and simulated DDoS attacks. The goal is to evaluate model stability, response time, and security under real-world attack conditions.\n\n## Datasets\n- **IMDB Movie Reviews**: Complex, natural movie reviews, ideal for evaluating typo robustness.\n- **Amazon Mobile Apps Reviews**: Short, bursty reviews, perfect for testing stress and system load impacts.\n\n## Why These Datasets?\nIMDB reviews offer long-form natural language testing, while Amazon Mobile Apps reviews simulate real-world short bursts of text traffic, common in production ML deployments.\n\n## Model Used\n- **DistilBERT-base-uncased-finetuned-sst-2-english**  \n- Chosen for speed, efficiency, and strong benchmark sentiment classification performance.\n\n## Attack Methods\n- **Typo Adversarial Attack**: Introducing random character errors.\n- **Flood Attack**: Long text flooding to simulate sequence overflow.\n- **DDoS Attack Simulation**: Rapid injection of small random junk inputs to simulate system overload.\n\n## Key Results\n| Dataset | Clean Accuracy | Typo Attack Accuracy |\n|:--|:--|:--|\n| IMDB | 49.4% | 52.8% |\n| Amazon Mobile Apps | 88.0% | 71.3% |\n\nResponse time increased significantly under flood and DDoS conditions.\n\n## Key Takeaways\nDespite reasonable typo robustness, NLP models like DistilBERT degrade significantly under flood and DDoS-style attacks, exposing vulnerabilities for real-world deployment scenarios.\n\n---\n\nOutputs:\n- Full benchmarking notebook (Benchmarking.ipynb)\n- Adversarial Results (CSV, HTML)\n- Graphs (Accuracy Comparison, Response Time)\n- Full HTML Final Report\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiamdevnd%2Ftransformer-robustness-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiamdevnd%2Ftransformer-robustness-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiamdevnd%2Ftransformer-robustness-benchmark/lists"}