https://github.com/griffincancode/scantron
A file scanner for secret management and more -- faster than the speed of light.
https://github.com/griffincancode/scantron
devops elm git leak-detection password-protection protection rust scan scss secrets zig
Last synced: 13 days ago
JSON representation
A file scanner for secret management and more -- faster than the speed of light.
- Host: GitHub
- URL: https://github.com/griffincancode/scantron
- Owner: GriffinCanCode
- Created: 2025-09-28T18:27:19.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2025-09-29T01:56:17.000Z (7 months ago)
- Last Synced: 2025-10-03T06:08:54.421Z (7 months ago)
- Topics: devops, elm, git, leak-detection, password-protection, protection, rust, scan, scss, secrets, zig
- Language: Rust
- Homepage:
- Size: 2.36 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Scantron - Universal Code & Configuration Scanner
**Grep on steroids**
Scantron is a sophisticated scanning engine that crawls ANY text-based files on your system and extracts structured information based on customizable patterns. Perfect for DevOps engineers managing multiple projects with different tech stacks.
## Core Use Cases
- **Security Audits**: Find hardcoded passwords, API keys, and secrets across all projects
- **Migration Planning**: Identify all references to deprecated APIs, old database servers, or legacy systems
- **Dependency Management**: Track which versions of libraries are in use across your infrastructure
- **Compliance Reporting**: Generate audit reports for security and compliance requirements
- **Technical Debt Analysis**: Identify code smells and problematic patterns across codebases
## Architecture
### Backend: Perl (Mojolicious)
- **High-performance text processing** with parallel scanning
- **RESTful API** for all operations
- **WebSocket support** for real-time updates
- **Type-safe data structures** using Type::Tiny
- **Extensible scanner engine** with plugin architecture
- **Job queue system** with Minion for background processing
### Frontend: Elm Web UI
- **Functional reactive programming** for reliable UI
- **Strong typing** eliminates runtime errors
- **Real-time dashboard** with live scan progress
- **Interactive result visualization** with filtering and export
- **Responsive design** works on desktop and mobile
### CLI: Rust
- **Fast startup** and cross-platform binaries
- **Modern argument parsing** with rich help
- **Auto-server management** starts backend as needed
- **Multiple output formats** (table, JSON, CSV, plain)
- **CI/CD integration** with exit codes and quiet mode
## Features
### Scanning Capabilities
- **Pattern Matching**: Regex patterns with customizable flags
- **Data Extraction**: Parse YAML, JSON, ENV files, and more
- **File Filtering**: Include/exclude by extension, path patterns, file size
- **Multi-target Support**: Local files, Git repos, Docker containers, SSH remotes, S3 buckets
- **Parallel Processing**: Configurable worker threads for fast scanning
- **Background Job Queue**: Reliable async processing with retry and monitoring
### Scan Profiles
- **Reusable Configurations**: Save and share scanning profiles
- **Pattern Libraries**: Built-in patterns for common security issues
- **Custom Extractors**: Define structured data extraction rules
- **Profile Inheritance**: Extend existing profiles with new patterns
### Results & Reporting
- **Timeline View**: See when patterns were introduced
- **Heatmap Visualization**: Identify problem areas at a glance
- **Dependency Graphs**: Understand relationships between findings
- **Export Options**: JSON, CSV, HTML reports
- **Trend Analysis**: Track technical debt over time
## Project Structure
```
scantron/
├── backend/ # Perl/Mojolicious API server
│ ├── lib/Scantron/
│ │ ├── API/ # REST API controllers
│ │ ├── Scanner/ # Core scanning engine
│ │ ├── Types/ # Type definitions
│ │ └── Utils/ # Utilities and storage
│ ├── t/ # Perl tests
│ ├── script/ # Startup scripts
│ └── cpanfile # Perl dependencies
├── frontend/ # Elm web application
│ ├── src/
│ │ ├── Types/ # Elm type definitions
│ │ ├── Pages/ # UI pages/views
│ │ ├── Components/# Reusable UI components
│ │ └── API.elm # Backend communication
│ ├── public/ # Static assets
│ ├── tests/ # Elm tests
│ └── elm.json # Elm configuration
├── cli/ # Rust CLI application
│ ├── src/
│ │ ├── cli/ # Command implementations
│ │ ├── client.rs # HTTP client for backend
│ │ ├── output.rs # Output formatters
│ │ ├── server.rs # Server management
│ │ └── types.rs # Data structures
│ ├── Cargo.toml # Rust dependencies
│ └── README.md # CLI documentation
│ ├── security-audit.json
│ ├── aws-migration.json
│ └── dependency-audit.json
└── README.md
```
## Installation & Setup
### Quick Start (Recommended)
```bash
# Install all dependencies and build everything
make install
# Build CLI and web app
make build
# Start development servers (web UI)
make dev
# Or use the fast CLI
./cli/target/release/scantron --help
```
### CLI Setup (Fast Path)
```bash
cd cli
# Build CLI
cargo build --release
# Install to system (optional)
cargo install --path .
# Quick security scan
./target/release/scantron scan --profile security-audit ./
```
### Web App Setup (Full Experience)
```bash
# Backend setup
cd backend
cpanm --installdeps .
./script/scantron daemon -l http://localhost:3000
# Frontend setup
cd frontend
npm install -g elm
elm make src/Main.elm --output=public/elm.js --debug
# Open http://localhost:3000 in browser
```
### Production Deployment
```bash
# Build everything
make build
# Start server and background workers
make start
# Or manually:
# Backend with Hypnotoad (production PSGI server)
cd backend
./script/scantron prefork &
# Start background job workers
./script/scantron-worker --queues default &
# CLI is ready to use
./cli/target/release/scantron scan --profile security-audit ./
```
### Worker Management
```bash
# Start/stop everything
make start # Start server + workers
make stop # Stop server + workers
# Individual control
make start-workers # Start job queue workers only
make stop-workers # Stop job queue workers only
make worker-logs # View worker logs
```
## Configuration
### Environment Variables
```bash
# Optional: Custom data directory
SCANTRON_DATA_DIR=/var/lib/scantron
# Optional: Log level
MOJO_LOG_LEVEL=info
```
### Scan Profile Example
```json
{
"name": "Security Audit",
"description": "Find security issues in code",
"patterns": [
{
"name": "hardcoded-password",
"regex": "password\\s*=\\s*[\"']([^\"']+)[\"']",
"description": "Hardcoded passwords"
},
{
"name": "api-keys",
"regex": "api[_-]?key\\s*[=:]\\s*[\"']?([a-zA-Z0-9]{20,})",
"description": "API keys and tokens"
}
],
"extractors": [
{
"type": "yaml",
"path": "database.password",
"description": "Extract DB passwords from YAML"
}
],
"file_filter": {
"extensions": ["py", "js", "yaml", "json"],
"exclude_patterns": ["node_modules", "*.min.js"],
"max_size": 1048576
}
}
```
## Testing
### Backend Tests
```bash
cd backend
prove -l t/
```
### Frontend Tests
```bash
cd frontend
elm-test
```
### Integration Tests
```bash
# Start backend in test mode
cd backend
./script/scantron test
# Run API tests
curl -X GET http://localhost:3000/api/v1/scans
```
## Usage Examples
### CLI Quick Start
```bash
# 1. Security audit with built-in profile
scantron scan --profile security-audit ./my-project
# 2. Find AWS references for migration
scantron scan --profile aws-migration ./infrastructure
# 3. Track dependencies across projects
scantron scan --profile dependency-audit ./
# 4. Custom pattern search
scantron scan --pattern "TODO|FIXME" --ext js,py,ts ./src
# 5. CI/CD integration (fail if secrets found)
scantron scan --profile security-audit --fail-on-match --quiet ./
```
### Web UI Workflows
1. **Open http://localhost:3000** in your browser
2. **Create scan profiles** or use built-in ones
3. **Start scans** and watch real-time progress
4. **Explore results** with interactive filtering
5. **Export reports** for compliance and analysis
### Profile Examples
Built-in profile templates available in `backend/profile-templates/`:
**Security Audit** (`security-audit.json`)
- Hardcoded passwords and API keys
- Weak cryptographic algorithms
- SQL injection patterns
- Debug code and TODOs
**AWS Migration** (`aws-migration.json`)
- AWS service references
- EC2 instance IDs and S3 buckets
- Lambda functions and IAM roles
- CloudFormation templates
**Dependency Audit** (`dependency-audit.json`)
- NPM, pip, maven, gradle dependencies
- Version constraints and outdated packages
- Security vulnerability patterns
## Contributing
We welcome contributions! Please see our guidelines:
1. **Code Style**: Follow existing patterns and naming conventions
2. **Testing**: Add tests for new features
3. **Documentation**: Update README and inline docs
4. **Type Safety**: Maintain strong typing in both Perl and Elm
### Development Principles
- **Minimize Tech Debt**: Every file and function should be concise but sophisticated
- **Strong Typing**: Use Type::Tiny in Perl and native types in Elm
- **Testability**: Design for easy testing and mocking
- **Readability**: One-word, memorable, relevant names for all files
- **Extensibility**: Plugin architecture for custom scanners and exporters
## License
MIT License - see LICENSE file for details.
## Roadmap
- [ ] **Plugin System**: Custom scanner plugins for specialized file types
- [ ] **Cloud Integration**: Native support for AWS, Azure, GCP scanning
- [ ] **AI-Powered Patterns**: Machine learning to suggest scan patterns
- [ ] **Continuous Monitoring**: Git hooks and CI/CD integration
- [ ] **Team Features**: User management and shared scan profiles
- [ ] **Advanced Visualizations**: Graph analysis and trend charts
- [ ] **Mobile App**: Native iOS/Android apps for monitoring
---
**Scantron** - Making large-scale code analysis simple, fast, and beautiful.