{"id":28826482,"url":"https://github.com/michael-borck/academic-auto-grader","last_synced_at":"2026-04-24T23:31:21.392Z","repository":{"id":299474082,"uuid":"1003013996","full_name":"michael-borck/academic-auto-grader","owner":"michael-borck","description":"Configuration-driven automated grading system for programming assignments with multi-LLM evaluation, GitHub integration, and comprehensive    cost tracking","archived":false,"fork":false,"pushed_at":"2025-06-16T19:29:12.000Z","size":115,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-16T19:34:52.170Z","etag":null,"topics":["academic-tools","anthropic","automated-grading","education","github-integration","jupyter-notebooks","llm","openai","programming-assessment","python","yaml-configuration"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michael-borck.png","metadata":{"files":{"readme":"docs/README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-16T13:43:48.000Z","updated_at":"2025-06-16T19:29:16.000Z","dependencies_parsed_at":"2025-06-20T00:01:34.552Z","dependency_job_id":null,"html_url":"https://github.com/michael-borck/academic-auto-grader","commit_stats":null,"previous_names":["michael-borck/academic-auto-grader"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/michael-borck/academic-auto-grader","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michael-borck%2Facademic-auto-grader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michael-borck%2Facademic-auto-grader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michael-borck%2Facademic-auto-grader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michael-borck%2Facademic-auto-grader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michael-borck","download_url":"https://codeload.github.com/michael-borck/academic-auto-grader/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michael-borck%2Facademic-auto-grader/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32244994,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-24T13:21:15.438Z","status":"ssl_error","status_checked_at":"2026-04-24T13:21:15.005Z","response_time":64,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["academic-tools","anthropic","automated-grading","education","github-integration","jupyter-notebooks","llm","openai","programming-assessment","python","yaml-configuration"],"created_at":"2025-06-19T03:04:37.952Z","updated_at":"2026-04-24T23:31:21.386Z","avatar_url":"https://github.com/michael-borck.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Automated Grading System\n\nA generalized, configuration-driven automated grading system for programming assignments. Supports multiple assignments through YAML configurations and Jinja2 templates.\n\n## 🚀 Quick Start\n\n```bash\n# List available assignments\n./grader list\n\n# Complete workflow for WeatherWise assignment\n./grader process-all submissions/ --assignment weatherwise --roster students.csv\n\n# Dry run to estimate costs\n./grader grade submissions/ --assignment weatherwise --dry-run\n```\n\n## 📁 Project Structure\n\n```\nautomated-grader/\n├── core/                           # Core grading functionality\n│   ├── auto_grader.py             # Main grading engine (generalized)\n│   ├── config_loader.py           # YAML configuration loader\n│   ├── prompt_builder.py          # Jinja2 template-based prompt generation\n│   └── github_scanner.py          # GitHub repository scanner\n├── preprocessing/                  # Pre-processing pipeline\n│   ├── consolidate_assignments.py # Group student files by ID\n│   └── clean_zip_files.py         # Remove non-enrolled students\n├── templates/                      # Jinja2 prompt templates\n│   ├── base_prompt.j2             # Main grading prompt template\n│   ├── github_analysis.j2         # GitHub repository analysis\n│   ├── function_analysis.j2       # Required function checking\n│   ├── notebook_structure.j2      # Notebook organization analysis\n│   ├── conversation_analysis.j2   # AI conversation evaluation\n│   ├── additional_documents.j2    # Additional document processing\n│   ├── readme_analysis.j2         # README evaluation\n│   └── code_sample.j2             # Code sample display\n├── assignments/                    # Assignment configurations\n│   └── weatherwise/\n│       └── config.yaml            # WeatherWise assignment configuration\n├── tools/                         # Utility tools\n│   ├── assignment_generator.py    # Interactive assignment creator\n│   └── config_validator.py        # Configuration validator\n├── grader*                        # Unified CLI interface\n└── docs/\n    ├── README.md                  # This file\n    ├── USAGE.md                   # Detailed usage guide\n    └── ASSIGNMENTS.md             # Creating new assignments\n```\n\n## 🎯 Core Features\n\n### ✅ Preserved from Original System\n- **Multi-LLM grading** (OpenAI + Anthropic) with token tracking\n- **Cost analysis** and dry-run capabilities\n- **GitHub integration** with commit analysis and repository structure evaluation\n- **Multiple document formats** (PDF, DOCX, Jupyter notebooks, text files)\n- **Second-person feedback style** for personalized student communication\n- **Comprehensive error handling** and progress tracking\n- **Pre-processing workflow** integration for Blackboard downloads\n\n### 🆕 New Generalized Features\n- **Configuration-driven assignments** - Create new assignments without touching code\n- **Template-based prompt generation** - Jinja2 templates for flexible, reusable prompts\n- **Unified CLI interface** - Single command for complete workflows\n- **Assignment generator tools** - Interactive creation of new assignment configs\n- **Validation system** - Ensures assignment configurations are correct\n- **Backward compatibility** - WeatherWise assignment still works exactly as before\n\n## 🔧 Installation\n\n1. **Clone or copy the automated-grader directory**\n2. **Install dependencies:**\n   ```bash\n   pip install -r requirements.txt\n   ```\n3. **Set up environment variables:**\n   ```bash\n   export OPENAI_API_KEY=\"your-openai-key\"\n   export ANTHROPIC_API_KEY=\"your-anthropic-key\"\n   export GITHUB_TOKEN=\"your-github-token\"  # Optional, for GitHub integration\n   ```\n4. **Make CLI executable:**\n   ```bash\n   chmod +x grader\n   ```\n\n## 📋 Usage Examples\n\n### List Available Assignments\n```bash\n./grader list\n```\n\n### Complete Workflow (Recommended)\n```bash\n# Process WeatherWise submissions with full pipeline\n./grader process-all submissions/ \\\n  --assignment weatherwise \\\n  --roster students.csv\n```\n\n### Individual Steps\n\n#### 1. Preprocessing Only\n```bash\n./grader preprocess submissions/ --roster students.csv\n```\n\n#### 2. GitHub Scanning Only\n```bash\n./grader github-scan submissions/\n```\n\n#### 3. Grading Only\n```bash\n# With GitHub integration\n./grader grade submissions/ --assignment weatherwise --github\n\n# Without GitHub integration (faster)\n./grader grade submissions/ --assignment weatherwise --no-github\n\n# Dry run (estimate costs)\n./grader grade submissions/ --assignment weatherwise --dry-run\n```\n\n## 📊 Assignment Configuration\n\nEach assignment is defined by a YAML configuration file in `assignments/\u003cassignment_name\u003e/config.yaml`.\n\n### Example Configuration Structure\n```yaml\nassignment:\n  name: \"WeatherWise\"\n  description: \"Intelligent Weather Analysis \u0026 Advisory System\"\n  type: \"python_application\"\n  total_points: 100\n\ncontext:\n  requirements:\n    - \"Weather data retrieval and display\"\n    - \"Natural language interface for weather queries\"\n    - \"At least 2 data visualizations\"\n  \n  required_functions:\n    - name: \"get_weather_data\"\n      signature: \"get_weather_data(location, forecast_days=5)\"\n      description: \"Retrieve weather data from API\"\n  \n  file_expectations:\n    notebook: required\n    conversation_files: required\n    readme: optional\n    github: optional\n\nrubric:\n  functionality:\n    weight: 15\n    max_score: 15\n    criteria: \"Application works as expected; core functions implemented correctly\"\n  \n  intentional_prompting:\n    weight: 30\n    max_score: 30\n    criteria: \"Documentation of AI interactions; evidence of strategic prompting techniques\"\n  \n  # ... more criteria\n\ngithub_integration:\n  enabled: true\n  minimum_commits: 15\n  repository_analysis: true\n  readme_comparison: true\n```\n\n## 🛠️ Creating New Assignments\n\n### Option 1: Interactive Generator (Recommended)\n```bash\npython tools/assignment_generator.py\n```\n\n### Option 2: Copy and Modify Existing\n```bash\ncp -r assignments/weatherwise assignments/new_assignment\n# Edit assignments/new_assignment/config.yaml\n```\n\n### Option 3: Manual Creation\nCreate `assignments/\u003cname\u003e/config.yaml` following the structure above.\n\n### Validate Configuration\n```bash\npython tools/config_validator.py assignments/new_assignment/config.yaml\n# Or validate all assignments\npython tools/config_validator.py --all\n```\n\n## 📈 Token Usage and Cost Tracking\n\nThe system provides comprehensive token usage tracking:\n\n- **Real-time cost estimation** during dry runs\n- **Per-student cost breakdown** in results\n- **Provider comparison** (OpenAI vs Anthropic efficiency)\n- **Detailed token reports** saved alongside grading results\n\n### Cost Estimation Example\n```bash\n./grader grade submissions/ --assignment weatherwise --dry-run\n```\n\nOutput includes:\n- Estimated tokens per submission\n- Cost breakdown by provider\n- Total estimated cost for all submissions\n- Content analysis summary\n\n## 🔄 Migration from Original System\n\nYour existing WeatherWise workflow continues to work:\n\n### Before (Original)\n```bash\npython auto_grader.py submissions/ --no-github\n```\n\n### After (Generalized)\n```bash\n./grader grade submissions/ --assignment weatherwise --no-github\n```\n\nThe results are identical, but now you can:\n- Create new assignments without code changes\n- Use templates for consistent prompt structure\n- Benefit from the unified CLI interface\n- Access improved validation and error handling\n\n## 🐛 Troubleshooting\n\n### Common Issues\n\n1. **\"Assignment not found\"**\n   ```bash\n   # List available assignments\n   ./grader list\n   # Validate assignment config\n   python tools/config_validator.py assignments/your_assignment/config.yaml\n   ```\n\n2. **\"Missing API keys\"**\n   ```bash\n   export OPENAI_API_KEY=\"your-key\"\n   export ANTHROPIC_API_KEY=\"your-key\"\n   ```\n\n3. **\"Template errors\"**\n   ```bash\n   # Validate templates\n   python core/prompt_builder.py\n   ```\n\n4. **\"GitHub integration issues\"**\n   ```bash\n   # Run GitHub scanner separately\n   ./grader github-scan submissions/\n   # Or disable GitHub integration\n   ./grader grade submissions/ --assignment name --no-github\n   ```\n\n### Debug Mode\nAdd `--dry-run` to any grading command to see what would happen without making API calls.\n\n## 📚 Additional Documentation\n\n- **[USAGE.md](USAGE.md)** - Detailed usage guide with advanced examples\n- **[ASSIGNMENTS.md](ASSIGNMENTS.md)** - Complete guide to creating and configuring assignments\n- **Original documentation** in `docs/` folder for reference\n\n## 🤝 Contributing\n\n1. Create new assignments in `assignments/` folder\n2. Add new templates in `templates/` folder\n3. Validate configurations with `tools/config_validator.py`\n4. Test with dry runs before live grading\n\n## 📝 License\n\nThis project inherits the license from the original grading system.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichael-borck%2Facademic-auto-grader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichael-borck%2Facademic-auto-grader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichael-borck%2Facademic-auto-grader/lists"}