{"id":38231699,"url":"https://github.com/acmutd/acmutd-api","last_synced_at":"2026-01-17T01:01:01.636Z","repository":{"id":304209465,"uuid":"1013494258","full_name":"acmutd/acmutd-api","owner":"acmutd","description":"A centralized way for students and ACM products to get data about UTD","archived":false,"fork":false,"pushed_at":"2025-12-01T21:29:08.000Z","size":272,"stargazers_count":1,"open_issues_count":5,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-04T11:45:01.804Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/acmutd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-04T02:13:12.000Z","updated_at":"2025-12-01T21:29:12.000Z","dependencies_parsed_at":"2025-07-11T18:40:02.913Z","dependency_job_id":"c0d1d6aa-e5d0-40f0-b166-6484d0e5aef4","html_url":"https://github.com/acmutd/acmutd-api","commit_stats":null,"previous_names":["acmutd/acmutd-api"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/acmutd/acmutd-api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acmutd%2Facmutd-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acmutd%2Facmutd-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acmutd%2Facmutd-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acmutd%2Facmutd-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/acmutd","download_url":"https://codeload.github.com/acmutd/acmutd-api/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acmutd%2Facmutd-api/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28490895,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T00:50:05.742Z","status":"ssl_error","status_checked_at":"2026-01-17T00:43:11.982Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-17T01:00:30.754Z","updated_at":"2026-01-17T01:01:01.616Z","avatar_url":"https://github.com/acmutd.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ACM UTD API\n\nA comprehensive REST API for accessing University of Texas at Dallas course data, including course information, grade distributions, and professor ratings. Built with Go, Firebase, and Python scrapers.\n\n## Architecture Overview\n\nThis project consists of several key components:\n\n- **Go API Server** (`cmd/api/`) - Main REST API with authentication, rate limiting, and CORS support. This is the main entry point for running the API.\n- **Go Scraper Service** (`cmd/scraper/`) - Orchestrates data collection from various sources. This is used to scrape the data from the various sources and store it in different potential locations.\n- **Python Scrapers** (`scripts/`) - Individual scrapers for different data sources\n- **Firebase Integration** - Cloud Firestore for data storage and Cloud Storage for file management\n\n### Data Sources\n\n- **Coursebook** - UTD's official course catalog and scheduling system\n- **Grade Distributions** - Historical grade data from UTD\n- **RMP** - Professor ratings and reviews\n- **Integration Service** - Combines data from multiple sources\n\n## Quick Start\n\n### Prerequisites\n\n- **Go** - [Download here](https://golang.org/dl/)\n- **Python** - [Download here](https://www.python.org/downloads/)\n- **Firebase Project** - Set up a Firebase project with Firestore and Storage. You will need to set up a service account and download the service account key.\n- **Chrome/ChromeDriver** - Required for web scraping. This should be handled by the scripts\n\n### Installation\n\n1. **Clone the repository**\n\n   ```bash\n   git clone https://github.com/acmutd/acmutd-api.git\n   cd acmutd-api\n   ```\n\n2. **Run the setup script**\n\n   ```bash\n   ./setup.sh\n   ```\n\n   This script will:\n   - Install Go dependencies\n   - Create a Python virtual environment\n   - Install all Python requirements\n   - Copy the `.env.example` file to `.env` for configuration\n\n3. **Configure environment variables**\n  Your `.env` file should look like this:\n\n   ```env\n   # Server Configuration\n   PORT=8080\n\n   # Firebase Configuration\n   FIREBASE_CONFIG=path/to/your/firebase-service-account.json\n\n   # Scraper Configuration\n   SCRAPER=coursebook  # Options: coursebook, grades, rmp-profiles, integration\n   SAVE_ENVIRONMENT=local  # Options: local, dev, prod\n\n   # Integration Scraper Configuration\n   INTEGRATION_SOURCE=local  # Options: local, dev, prod\n   INTEGRATION_RESCRAPE=false  # Options: true, false\n\n   # UTD Credentials (for coursebook scraper)\n   NETID=your_netid\n   PASSWORD=your_password\n\n   # Terms to scrape (comma-separated)\n   CLASS_TERMS=24f,25s,25f\n   ```\n\n4. **Start the API server**\nThe API will be available at `http://localhost:8080`\n\n   ```bash\n   go run cmd/api/main.go\n   ```\n\n5. **Run the scraper**\n\n    The scraper will run depending on the `SCRAPER` environment variable.\n    Depending on the `SAVE_ENVIRONMENT` environment variable, the data will be saved locally or uploaded to Firebase.\n    When running the integration scraper, the `INTEGRATION_SOURCE` environment variable determines the data source (local files, dev Firebase, or prod Firebase), and `INTEGRATION_RESCRAPE` determines whether to run scrapers first before processing the data.\n\n    For detailed scraper documentation, see [`SCRAPER.md`](./SCRAPER.md).\n\n   ```bash\n   go run cmd/scraper/main.go\n   ```\n\n## 📖 API Documentation\n\nComprehensive API documentation is available in [`API_DOCUMENTATION.md`](./API_DOCUMENTATION.md).\n\n### Quick Examples\n\n```bash\n# Health check (no auth required)\ncurl http://localhost:8080/health\n\n# Get all courses for Fall 2024 (requires API key)\ncurl -H \"X-API-Key: your-api-key\" \\\n     http://localhost:8080/api/v1/courses/24f\n\n# Search for Computer Science courses\ncurl -H \"X-API-Key: your-api-key\" \\\n     \"http://localhost:8080/api/v1/courses/24f?prefix=cs\"\n```\n\n### Authentication\n\nAll API endpoints (except `/health`) require an API key. Admin users can create API keys via the admin endpoints. See the [API Documentation](./API_DOCUMENTATION.md) for details.\n\n## 🛠️ Development\n\n### Project Structure\n\n```bash\nacm-api/\n├── cmd/                    # Main applications\n│   ├── api/               # REST API server\n│   └── scraper/           # Scraper orchestrator\n├── internal/              # Private application code\n│   ├── firebase/          # Firebase integration\n│   ├── scraper/           # Scraper implementations\n│   ├── server/            # HTTP server and middleware\n│   └── types/             # Data models\n├── scripts/               # Python scrapers\n│   ├── coursebook/        # UTD Coursebook scraper\n│   ├── grades/            # Grade distribution processor\n│   ├── integration/       # Data integration service\n│   ├── professors/        # Professor data aggregator\n│   └── rmp-profiles/      # Rate My Professor scraper\n└── setup.sh              # Development environment setup\n```\n\n### Data Flow\n\n1. **Coursebook Scraper** → Extracts course data from UTD's coursebook system\n2. **Grade Processor** → Processes Excel files containing grade distributions\n3. **RMP Scraper** → Collects professor ratings from Rate My Professor\n4. **Integration Service** → Combines all data sources and uploads to Firebase\n5. **API Server** → Serves integrated data via REST endpoints\n\n## 🔧 Configuration\n\n### Environment Variables\n\n| Variable | Description | Required | Default |\n|----------|-------------|----------|---------|\n| `PORT` | API server port | No | `8080` |\n| `FB_CONFIG` | Firebase service account JSON filename | Yes | `acmutd-api.json` |\n| `SCRAPER` | Which scraper to run (coursebook/grades/rmp-profiles/integration) | Yes (for scraper) | - |\n| `SAVE_ENVIRONMENT` | Where to save data (local/dev/prod) | No | `local` |\n| `NETID` | UTD NetID for coursebook access | Yes (for coursebook) | - |\n| `PASSWORD` | UTD password for coursebook access | Yes (for coursebook) | - |\n| `CLASS_TERMS` | Comma-separated terms to scrape (e.g., 24f,25s,25f) | Yes (for scrapers) | - |\n| `INTEGRATION_SOURCE` | Data source for integration scraper (local/dev/prod) | No | `local` |\n| `INTEGRATION_RESCRAPE` | Whether to run scrapers before integration (true/false) | No | `false` |\n\n### Term Format\n\nTerms use a specific format: `{YY}{season}` where:\n\n- `YY` is the 2-digit year (e.g., `24` for 2024)\n- `season` is `f` (Fall), `s` (Spring), or `u` (Summer)\n\nExamples: `24f` (Fall 2024), `25s` (Spring 2025), `24u` (Summer 2024)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Facmutd%2Facmutd-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Facmutd%2Facmutd-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Facmutd%2Facmutd-api/lists"}