Scantron - Universal Code & Configuration Scanner

Grep on steroids

Scantron is a sophisticated scanning engine that crawls ANY text-based files on your system and extracts structured information based on customizable patterns. Perfect for DevOps engineers managing multiple projects with different tech stacks.

Core Use Cases

• Security Audits: Find hardcoded passwords, API keys, and secrets across all projects
• Migration Planning: Identify all references to deprecated APIs, old database servers, or legacy systems
• Dependency Management: Track which versions of libraries are in use across your infrastructure
• Compliance Reporting: Generate audit reports for security and compliance requirements
• Technical Debt Analysis: Identify code smells and problematic patterns across codebases

Architecture

Backend: Perl (Mojolicious)

• High-performance text processing with parallel scanning
• RESTful API for all operations
• WebSocket support for real-time updates
• Type-safe data structures using Type::Tiny
• Extensible scanner engine with plugin architecture
• Job queue system with Minion for background processing

Frontend: Elm Web UI

• Functional reactive programming for reliable UI
• Strong typing eliminates runtime errors
• Real-time dashboard with live scan progress
• Interactive result visualization with filtering and export
• Responsive design works on desktop and mobile

CLI: Rust

• Fast startup and cross-platform binaries
• Modern argument parsing with rich help
• Auto-server management starts backend as needed
• Multiple output formats (table, JSON, CSV, plain)
• CI/CD integration with exit codes and quiet mode

Features

Scanning Capabilities

• Pattern Matching: Regex patterns with customizable flags
• Data Extraction: Parse YAML, JSON, ENV files, and more
• File Filtering: Include/exclude by extension, path patterns, file size
• Multi-target Support: Local files, Git repos, Docker containers, SSH remotes, S3 buckets
• Parallel Processing: Configurable worker threads for fast scanning
• Background Job Queue: Reliable async processing with retry and monitoring

Scan Profiles

• Reusable Configurations: Save and share scanning profiles
• Pattern Libraries: Built-in patterns for common security issues
• Custom Extractors: Define structured data extraction rules
• Profile Inheritance: Extend existing profiles with new patterns

Results & Reporting

• Timeline View: See when patterns were introduced
• Heatmap Visualization: Identify problem areas at a glance
• Dependency Graphs: Understand relationships between findings
• Export Options: JSON, CSV, HTML reports
• Trend Analysis: Track technical debt over time

Project Structure

scantron/
├── backend/           # Perl/Mojolicious API server
│   ├── lib/Scantron/
│   │   ├── API/       # REST API controllers
│   │   ├── Scanner/   # Core scanning engine
│   │   ├── Types/     # Type definitions
│   │   └── Utils/     # Utilities and storage
│   ├── t/             # Perl tests
│   ├── script/        # Startup scripts
│   └── cpanfile       # Perl dependencies
├── frontend/          # Elm web application
│   ├── src/
│   │   ├── Types/     # Elm type definitions
│   │   ├── Pages/     # UI pages/views
│   │   ├── Components/# Reusable UI components
│   │   └── API.elm    # Backend communication
│   ├── public/        # Static assets
│   ├── tests/         # Elm tests
│   └── elm.json       # Elm configuration
├── cli/               # Rust CLI application
│   ├── src/
│   │   ├── cli/       # Command implementations
│   │   ├── client.rs  # HTTP client for backend
│   │   ├── output.rs  # Output formatters
│   │   ├── server.rs  # Server management
│   │   └── types.rs   # Data structures
│   ├── Cargo.toml     # Rust dependencies
│   └── README.md      # CLI documentation
│   ├── security-audit.json
│   ├── aws-migration.json
│   └── dependency-audit.json
└── README.md

Installation & Setup

Quick Start (Recommended)

# Install all dependencies and build everything
make install

# Build CLI and web app
make build

# Start development servers (web UI)
make dev

# Or use the fast CLI
./cli/target/release/scantron --help

CLI Setup (Fast Path)

cd cli

# Build CLI
cargo build --release

# Install to system (optional)
cargo install --path .

# Quick security scan
./target/release/scantron scan --profile security-audit ./

Web App Setup (Full Experience)

# Backend setup
cd backend
cpanm --installdeps .
./script/scantron daemon -l http://localhost:3000

# Frontend setup  
cd frontend
npm install -g elm
elm make src/Main.elm --output=public/elm.js --debug

# Open http://localhost:3000 in browser

Production Deployment

# Build everything
make build

# Start server and background workers
make start

# Or manually:
# Backend with Hypnotoad (production PSGI server)
cd backend
./script/scantron prefork &

# Start background job workers
./script/scantron-worker --queues default &

# CLI is ready to use
./cli/target/release/scantron scan --profile security-audit ./

Worker Management

# Start/stop everything
make start    # Start server + workers
make stop     # Stop server + workers

# Individual control
make start-workers  # Start job queue workers only
make stop-workers   # Stop job queue workers only
make worker-logs    # View worker logs

Configuration

Environment Variables

# Optional: Custom data directory
SCANTRON_DATA_DIR=/var/lib/scantron

# Optional: Log level
MOJO_LOG_LEVEL=info

Scan Profile Example

{
  "name": "Security Audit",
  "description": "Find security issues in code",
  "patterns": [
    {
      "name": "hardcoded-password",
      "regex": "password\\s*=\\s*[\"']([^\"']+)[\"']",
      "description": "Hardcoded passwords"
    },
    {
      "name": "api-keys",
      "regex": "api[_-]?key\\s*[=:]\\s*[\"']?([a-zA-Z0-9]{20,})",
      "description": "API keys and tokens"
    }
  ],
  "extractors": [
    {
      "type": "yaml",
      "path": "database.password",
      "description": "Extract DB passwords from YAML"
    }
  ],
  "file_filter": {
    "extensions": ["py", "js", "yaml", "json"],
    "exclude_patterns": ["node_modules", "*.min.js"],
    "max_size": 1048576
  }
}

Testing

Backend Tests

cd backend
prove -l t/

Frontend Tests

cd frontend
elm-test

Integration Tests

# Start backend in test mode
cd backend
./script/scantron test

# Run API tests
curl -X GET http://localhost:3000/api/v1/scans

Usage Examples

CLI Quick Start

# 1. Security audit with built-in profile
scantron scan --profile security-audit ./my-project

# 2. Find AWS references for migration 
scantron scan --profile aws-migration ./infrastructure

# 3. Track dependencies across projects
scantron scan --profile dependency-audit ./

# 4. Custom pattern search
scantron scan --pattern "TODO|FIXME" --ext js,py,ts ./src

# 5. CI/CD integration (fail if secrets found)
scantron scan --profile security-audit --fail-on-match --quiet ./

Web UI Workflows

1. Open http://localhost:3000 in your browser
2. Create scan profiles or use built-in ones
3. Start scans and watch real-time progress
4. Explore results with interactive filtering
5. Export reports for compliance and analysis

Profile Examples

Built-in profile templates available in backend/profile-templates/:

Security Audit (security-audit.json)

• Hardcoded passwords and API keys
• Weak cryptographic algorithms
• SQL injection patterns
• Debug code and TODOs

AWS Migration (aws-migration.json)

• AWS service references
• EC2 instance IDs and S3 buckets
• Lambda functions and IAM roles
• CloudFormation templates

Dependency Audit (dependency-audit.json)

• NPM, pip, maven, gradle dependencies
• Version constraints and outdated packages
• Security vulnerability patterns

Contributing

We welcome contributions! Please see our guidelines:

1. Code Style: Follow existing patterns and naming conventions
2. Testing: Add tests for new features
3. Documentation: Update README and inline docs
4. Type Safety: Maintain strong typing in both Perl and Elm

Development Principles

• Minimize Tech Debt: Every file and function should be concise but sophisticated
• Strong Typing: Use Type::Tiny in Perl and native types in Elm
• Testability: Design for easy testing and mocking
• Readability: One-word, memorable, relevant names for all files
• Extensibility: Plugin architecture for custom scanners and exporters

License

MIT License - see LICENSE file for details.

Roadmap

Plugin System: Custom scanner plugins for specialized file types
Cloud Integration: Native support for AWS, Azure, GCP scanning

AI-Powered Patterns: Machine learning to suggest scan patterns
Continuous Monitoring: Git hooks and CI/CD integration
Team Features: User management and shared scan profiles
Advanced Visualizations: Graph analysis and trend charts
Mobile App: Native iOS/Android apps for monitoring

Scantron - Making large-scale code analysis simple, fast, and beautiful.