{"id":15096121,"url":"https://github.com/ecshreve/jepp","last_synced_at":"2026-01-20T20:11:17.910Z","repository":{"id":176365432,"uuid":"653439389","full_name":"ecshreve/jepp","owner":"ecshreve","description":"api fun with jeopardy!","archived":false,"fork":false,"pushed_at":"2024-02-23T05:02:03.000Z","size":122752,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-01T00:25:51.177Z","etag":null,"topics":["api","colly","data","gin-gonic","github-actions","go","golang","jeopardy","mysql","scrape","trivia"],"latest_commit_sha":null,"homepage":"https://jepp.app","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ecshreve.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"custom":["https://www.buymeacoffee.com/ecshreve"]}},"created_at":"2023-06-14T04:02:31.000Z","updated_at":"2023-09-20T20:31:31.000Z","dependencies_parsed_at":null,"dependency_job_id":"5d3d225a-aab5-4bf4-add7-c3abbc4430c3","html_url":"https://github.com/ecshreve/jepp","commit_stats":null,"previous_names":["ecshreve/jepp"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/ecshreve/jepp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ecshreve%2Fjepp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ecshreve%2Fjepp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ecshreve%2Fjepp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ecshreve%2Fjepp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ecshreve","download_url":"https://codeload.github.com/ecshreve/jepp/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ecshreve%2Fjepp/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28612163,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-20T18:56:40.769Z","status":"ssl_error","status_checked_at":"2026-01-20T18:54:26.653Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","colly","data","gin-gonic","github-actions","go","golang","jeopardy","mysql","scrape","trivia"],"created_at":"2024-09-25T15:45:43.952Z","updated_at":"2026-01-20T20:11:17.881Z","avatar_url":"https://github.com/ecshreve.png","language":"Go","funding_links":["https://www.buymeacoffee.com/ecshreve"],"categories":[],"sub_categories":[],"readme":"# [jepp](https://jepp.app)\n\nAPI fun with Jeopardy! Access \u003e100k Jeopardy clues scraped from [j-archive] via a simple api.\n\n\n[![CI](https://github.com/ecshreve/jepp/actions/workflows/ci.yml/badge.svg?branch=main\u0026event=push)](https://github.com/ecshreve/jepp/actions/workflows/ci.yml)\n![GitHub go.mod Go version](https://img.shields.io/github/go-mod/go-version/ecshreve/jepp)\n[![Go Report Card](https://goreportcard.com/badge/github.com/ecshreve/jepp)](https://goreportcard.com/report/github.com/ecshreve/jepp)\n[![GoDoc](https://godoc.org/github.com/ecshreve/jepp?status.svg)](https://godoc.org/github.com/ecshreve/jepp)\n![GitHub release (release name instead of tag name)](https://img.shields.io/github/v/release/ecshreve/jepp)\n\n---\n\n![jepp](static/repo/jepp-ui.png)\n\n# API\n\nThe api is backed by a go web server built with [gin] that exposes a few endpoints to access historical Jeopardy data.\n\n## Types\n\nThe shape of the data returned from the api aligns with the db schema, this is accomplished via various struct tags on the type definitions.\n\n### Struct tags quick reference\n\n- `db` tag is used by the [sqlx] library to map the db columns to the struct fields\n- `json` tag is used by the [gin] library to map the struct fields to the json response\n- `example` tag is used by the [swaggo] library to generate example responses for the swagger docs\n- `form` and `binding` tags are used by [gin] to map query arguments to a struct with some basic validation\n\n\nfor example, the `pkg.models.Clue` type is defined as follows:\n```{golang}\ntype Clue struct {\n\tClueID     int64  `db:\"clue_id\" json:\"clueId\" example:\"804002032\"`\n\tGameID     int64  `db:\"game_id\" json:\"gameId\" example:\"8040\"`\n\tCategoryID int64  `db:\"category_id\" json:\"categoryId\" example:\"804092001\"`\n\tQuestion   string `db:\"question\" json:\"question\" example:\"This is the question.\"`\n\tAnswer     string `db:\"answer\" json:\"answer\" example:\"This is the answer.\"`\n}\n```\n\n\n- Struct tags also appear on some helper structs like the `pkg.server.Filter` type:\n```{golang}\n// Filter describes the query parameters that can be used to filter the results of an API query.\ntype Filter struct {\n\tRandom *bool  `form:\"random\"`\n\tID     *int64 `form:\"id\"`\n\tPage   *int64 `form:\"page,default=0\" binding:\"min=0\"`\n\tLimit  *int64 `form:\"limit,default=1\" binding:\"min=1,max=100\"`\n}\n```\n\n\n## Frontend / UI\n\n- The ui is served from the `/` endpoint and is an html template that displays the swagger docs, some\n\tgeneral information, and a sample request.\n- The embedded swagger ui provides runnable request / response examples and type references.\n\n## Swagger Docs\n\n- Swagger documentation is generated with [swaggo] and embedded in the homepage as part of the html template.\n- Figuring out the right build/deploy configuration was challenging here, I ran into some problems in my Taskfile task dependencies. The main problem seemed to be multiple tasks with the same set of files listed as `sources` causing a watched task to continuously rebuild because of some circular dependencies.\n- These problems seem to be solved after breaking up and organizing the taskfiles better.\n\n# DB\n\nCurrently the app uses a file based sqlite database. Below are some notes on the deprecated mysql setup.\nAll in all, the 15 seasons of data currently in the DB only end up as ~25 MB .sql file. Using\nsqlite removed the need to run a mysql server and made the app easier to deploy and test.\n\n## Notes on deprecated mysql setup\n\nGetting the data into the database started as a manual process, and hasn't been automated yet because the data is all there and I haven't needed to import / export it recently.\n\nHere's how I went about doing it initially:\n- For local development I set the `DB_HOST`, `DB_USER`, `DB_PASS`, `DB_NAME` environment variables to target a `mariadb/mysql` server running in my home lab.\n- Most of the time I play with that local copy of the data, but the public api uses a mysql db hosted on [digital ocean](https://www.digitalocean.com/products/managed-databases-mysql)\n- Initially to populate the prod db I just manually created a backup of my local database and restored it to the prod database, both via an [adminer](https://hub.docker.com/_/adminer/) instance running in my home lab.\n- Currently the `task sql:dump` command will create a dump of the database defined by the environment variables and write it to `data/dump.sql.gz`.\n- Recent dumps of the prod database are available in the [data](data/) directory or as downloads on repository's [Releases](https://github.com/ecshreve/jepp/releases) page.\n\n\n\n## Data Scraping\n\nnote: all the scraping was done against the mysql databse, not the current sqlite setup (though I did \nsome brief testing and things seemed to still work for the most part _ymmv_)\n\nThe [scraper](pkg/scraper/) package contains the code to scrape [j-archive] for jeopardy clues and write the data to a mysql database. [Colly] is the package use to scrape the data and [sqlx] is used to write the data to the db. The scraping happened in a few passes, more or less following these steps:\n\nGet all the seasons and populate the seasons table.\n\n- This scrape targeted the season [summary page on j-archive](https://www.j-archive.com/listseasons.php) and pulled the season number, start date, end date for each season\n\nGet all the games for each season and populate the game table.\n\n- This scrape targets the individual [season show pages on j-archive](https://www.j-archive.com/showseason.php?season=1) and pulls the game number, air date, taped date for each season\n \nGet all the clues for each game in each season and populate the category and clue tables\n\n- This scrape targeted the individual [game pages on j-archive](https://www.j-archive.com/showgame.php?game_id=7040) and pulls the clue data from the `\u003ctable\u003e` elements on the page\n\n\n## references / prior art\n\n- [jservice](https://jservice.io/)\n- [jservice repo](https://github.com/sottenad/jService)\n- [jeppy](https://github.com/ecshreve/jeppy)\n- [illustrated sqlx](https://jmoiron.github.io/sqlx/)\n\n[sqlx]: \u003chttps://github.com/jmoiron/sqlx\u003e\n[gin]: \u003chttps://github.com/gin-gonic/gin\u003e\n[swaggo]: \u003chttps://github.com/swaggo/swag\u003e\n[j-archive]: \u003chttps://www.j-archive.com/\u003e\n[colly]: \u003chttps://github.com/gocolly/colly\u003e\n\n\n\u003chr\u003e\n\u003chr\u003e\n\n![cf](https://img.shields.io/badge/Cloudflare-F38020?style=for-the-badge\u0026logo=Cloudflare\u0026logoColor=white)\n![do](https://img.shields.io/badge/Digital_Ocean-0080FF?style=for-the-badge\u0026logo=DigitalOcean\u0026logoColor=white)\n![ga](https://img.shields.io/badge/GitHub_Actions-2088FF?style=for-the-badge\u0026logo=github-actions\u0026logoColor=white)\n![mysql](https://img.shields.io/badge/MySQL-005C84?style=for-the-badge\u0026logo=mysql\u0026logoColor=white)\n![mariadb](https://img.shields.io/badge/MariaDB-003545?style=for-the-badge\u0026logo=mariadb\u0026logoColor=white)\n![dock](https://img.shields.io/badge/Docker-2CA5E0?style=for-the-badge\u0026logo=docker\u0026logoColor=white)\n![swag](https://img.shields.io/badge/Swagger-85EA2D?style=for-the-badge\u0026logo=Swagger\u0026logoColor=white)\n![golan](https://img.shields.io/badge/Go-00ADD8?style=for-the-badge\u0026logo=go\u0026logoColor=white)\n\n\n\u003ca href=\"https://www.buymeacoffee.com/ecshreve\" target=\"_blank\"\u003e\u003cimg src=\"https://cdn.buymeacoffee.com/buttons/v2/default-blue.png\" alt=\"Buy Me A Coffee\" style=\"height: 25px !important;width: 100px !important;\" \u003e\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fecshreve%2Fjepp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fecshreve%2Fjepp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fecshreve%2Fjepp/lists"}