{"id":45348597,"url":"https://github.com/lightning-dabbler/sportscrape","last_synced_at":"2026-04-01T21:27:40.342Z","repository":{"id":278981160,"uuid":"937020039","full_name":"lightning-dabbler/sportscrape","owner":"lightning-dabbler","description":"A tool for collecting and transforming sports statistics from various sources into standardized formats. ⚾️ 🏀","archived":false,"fork":false,"pushed_at":"2026-03-16T05:26:31.000Z","size":509,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-16T09:28:53.156Z","etag":null,"topics":["mlb","mma","nba","odds","sports","ufc","wnba"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lightning-dabbler.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-02-22T06:18:38.000Z","updated_at":"2026-03-16T05:22:28.000Z","dependencies_parsed_at":"2025-03-10T22:25:28.056Z","dependency_job_id":"1b70f7a6-a1ea-45cf-bdc3-fb91dd4a06cb","html_url":"https://github.com/lightning-dabbler/sportscrape","commit_stats":null,"previous_names":["lightning-dabbler/sportscrape"],"tags_count":48,"template":false,"template_full_name":null,"purl":"pkg:github/lightning-dabbler/sportscrape","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightning-dabbler%2Fsportscrape","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightning-dabbler%2Fsportscrape/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightning-dabbler%2Fsportscrape/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightning-dabbler%2Fsportscrape/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lightning-dabbler","download_url":"https://codeload.github.com/lightning-dabbler/sportscrape/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightning-dabbler%2Fsportscrape/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31292258,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T21:15:39.731Z","status":"ssl_error","status_checked_at":"2026-04-01T21:15:34.046Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mlb","mma","nba","odds","sports","ufc","wnba"],"created_at":"2026-02-21T12:00:46.597Z","updated_at":"2026-04-01T21:27:40.333Z","avatar_url":"https://github.com/lightning-dabbler.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sportscrape\nA Go package for collecting and transforming sports statistics from various sources into standardized formats.\n\n[![Deploy][sportscrape-ci-status]][sportscrape-ci]\n[![Go Report Card][go-report-status]][go-report]\n[![Go Reference][goref-sportscrape-status]][goref-sportscrape]\n[![Releases][release-status]][releases]\n\n## CLI\n\nThe `sportscrape` CLI extracts sports data from all supported providers and exports to JSONL or Parquet, locally or to S3-compatible storage.\n\n### Installation\n```console\ngo install github.com/lightning-dabbler/sportscrape/cmd/sportscrape@latest\n```\n\n### Commands\n\n| Command | Subcommand | Provider |\n|---------|------------|----------|\n| `sportscrape baseballsavant` | | baseballsavant.mlb.com |\n| `sportscrape foxsports` | `mlb`, `nba`, `wnba` | foxsports.com |\n| `sportscrape espn` | `ufc` | espn.com/mma |\n| `sportscrape nba` | | nba.com |\n\nRun `sportscrape \u003ccommand\u003e --help` for feeds, flags, and defaults per provider.\n\n### Example\nExtract `2025-06-05` NBA traditional box score data from https://nba.com and store in `./tmp/nba-traditional-box-score-2025-06-05.jsonl` in JSONL format\n```console\nsportscrape nba \\\n  --feed traditional-box-score \\\n  --date 2025-06-05 \\\n  --destination ./tmp/nba-traditional-box-score-2025-06-05.jsonl\n```\n\n## Go Package\n\n### Installation\n```console\ngo get github.com/lightning-dabbler/sportscrape\n```\n\n### Quick start\nRetrieve and output `2025-06-05` NBA traditional box score data from https://nba.com\n```go\npackage main\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"log\"\n\t\"time\"\n\n\t\"github.com/lightning-dabbler/sportscrape/dataprovider/nba\"\n\t\"github.com/lightning-dabbler/sportscrape/dataprovider/nba/model\"\n\t\"github.com/lightning-dabbler/sportscrape/runner\"\n)\n\nfunc main() {\n\tmatchupScraper := nba.NewMatchupScraper(\n\t\tnba.WithMatchupDate(\"2025-06-05\"),\n\t\tnba.WithMatchupTimeout(2*time.Minute),\n\t)\n\tmatchupScraper.NetworkHeaders = nba.NetworkHeaders\n\tmatchuprunner := runner.NewMatchupRunner(\n\t\trunner.MatchupRunnerConfig[model.Matchup]{\n\t\t\tScraper:   matchupScraper,\n\t\t\tKeepAlive: true,\n\t\t},\n\t)\n\n\tmatchups, err := matchuprunner.Run()\n\tif err != nil {\n\t\tmatchupScraper.Close()\n\t\tpanic(err)\n\t}\n\n\tboxscorescraper := nba.NewBoxScoreTraditionalScraper(\n\t\tnba.WithBoxScoreTraditionalTimeout(2*time.Minute),\n\t\tnba.WithBoxScoreTraditionalPeriod(nba.Full),\n\t)\n\tboxscorescraper.DocumentRetriever = matchupScraper.DocumentRetriever\n\n\tboxscorerunner := runner.NewEventDataRunner(\n\t\trunner.EventDataRunnerConfig[model.Matchup, model.BoxScoreTraditional]{\n\t\t\tScraper:     boxscorescraper,\n\t\t\tConcurrency: 1,\n\t\t},\n\t)\n\n\trecords, err := boxscorerunner.Run(matchups)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\t// Output each statline as pretty json\n\tfor _, record := range records {\n\t\tjsonBytes, err := json.MarshalIndent(record, \"\", \"  \")\n\t\tif err != nil {\n\t\t\tlog.Fatalf(\"Error marshaling to JSON: %v\\n\", err)\n\t\t}\n\t\tfmt.Println(string(jsonBytes))\n\t}\n}\n\n```\n\n### Usage\n- [basketball-reference.com NBA scrape examples](dataprovider/basketballreferencenba/example_test.go) (Deprecated)\n- [baseball-reference.com MLB scrape examples](dataprovider/baseballreferencemlb/example_test.go) (Deprecated)\n- [foxsports.com scraping examples](dataprovider/foxsports/example_test.go)\n- [baseballsavant.mlb.com scraping examples](dataprovider/baseballsavantmlb/example_test.go)\n- [ESPN MMA scraping examples](dataprovider/espn/mma/example_test.go)\n- [nba.com NBA scraping examples](dataprovider/nba/example_test.go)\n\n## Data providers\n\n| Source                           | League   | Feed                                 |      Periods Available       |                                  Data Model                                  |\tDeprecated\t| Point-in-time|\n|----------------------------------|----------|--------------------------------------|:----------------------------:|:----------------------------------------------------------------------------:|:---------------------:|:------------:|\n| https://basketball-reference.com | NBA      | Matchup                              |             Full             |        [model](dataprovider/basketballreferencenba/model/matchup.go)         | 🚩 |✅|\n| https://basketball-reference.com | NBA      | Basic box score stats                | H1, H2, Q1, Q2, Q3, Q4, Full | [model](dataprovider/basketballreferencenba/model/basic_box_score_stats.go)  | 🚩 |✅|\n| https://basketball-reference.com | NBA      | Advanced box score stats             |             Full             |  [model](dataprovider/basketballreferencenba/model/adv_box_score_stats.go)   | 🚩 |✅|\n| https://baseball-reference.com   | MLB      | Matchup                              |             Full             |         [model](dataprovider/baseballreferencemlb/model/matchup.go)          | 🚩 |✅|\n| https://baseball-reference.com   | MLB      | Batting box score stats              |             Full             | [model](dataprovider/baseballreferencemlb/model/batting_box_score_stats.go)  | 🚩 |✅|\n| https://baseball-reference.com   | MLB      | Pitching box score stats             |             Full             | [model](dataprovider/baseballreferencemlb/model/pitching_box_score_stats.go) | 🚩 |✅|\n| https://www.foxsports.com\t\t      | NBA\t     | Matchup\t\t\t\t                          |        Live, Full\t\t\t         |               [model](dataprovider/foxsports/model/matchup.go)               ||✅|\n| https://www.foxsports.com\t\t      | NBA\t     | Box score stats\t\t                    |        Live, Full\t\t\t         |         [model](dataprovider/foxsports/model/nba_box_score_stats.go)         ||✅|\n| https://www.foxsports.com\t\t      | WNBA\t    | Matchup\t\t\t\t                          |        Live, Full\t\t\t         |               [model](dataprovider/foxsports/model/matchup.go)               ||✅|\n| https://www.foxsports.com\t\t      | WNBA\t    | Box score stats\t\t                    |        Live, Full\t\t\t         |         [model](dataprovider/foxsports/model/nba_box_score_stats.go)         ||✅|\n| https://www.foxsports.com\t\t      | MLB\t     | Matchup\t\t\t\t                          |        Live, Full\t\t\t         |               [model](dataprovider/foxsports/model/matchup.go)               ||✅|\n| https://www.foxsports.com\t\t      | MLB\t     | Batting Box score stats              |        Live, Full\t\t\t         |     [model](dataprovider/foxsports/model/mlb_batting_box_score_stats.go)     ||✅|\n| https://www.foxsports.com\t\t      | MLB\t     | Pitching Box score stats             |        Live, Full\t\t\t         |    [model](dataprovider/foxsports/model/mlb_pitching_box_score_stats.goo)    ||✅|\n| https://www.foxsports.com\t\t      | MLB\t     | Probable starting pitcher            |           Full\t\t\t            |    [model](dataprovider/foxsports/model/mlb_probable_starting_pitcher.go)    ||✅|\n| https://www.foxsports.com\t\t      | MLB\t     | Betting Odds Money line              |           Full\t\t\t            |         [model](dataprovider/foxsports/model/mlb_odds_money_line.go)         ||✅|\n| https://www.foxsports.com\t\t      | MLB\t     | Betting Odds Total                   |           Full\t\t\t            |           [model](dataprovider/foxsports/model/mlb_odds_total.go)            ||✅|\n| https://www.foxsports.com\t\t      | NCAAB\t   | Matchup\t\t\t\t                          |        Live, Full\t\t\t         |               [model](dataprovider/foxsports/model/matchup.go)               ||✅|\n| https://www.foxsports.com\t\t      | NFL\t     | Matchup\t\t\t\t                          |        Live, Full\t\t\t         |               [model](dataprovider/foxsports/model/matchup.go)               ||✅|\n| https://baseballsavant.mlb.com\t\t | MLB\t     | Matchup\t\t\t\t                          |        Live, Full\t\t\t         |           [model](dataprovider/baseballsavantmlb/model/matchup.go)           ||✅|\n| https://baseballsavant.mlb.com\t\t | MLB\t     | Batting box score stats              |        Live, Full\t\t\t         |      [model](dataprovider/baseballsavantmlb/model/batting_box_score.go)      ||✅|\n| https://baseballsavant.mlb.com\t\t | MLB\t     | Pitching box score stats             |        Live, Full\t\t\t         |     [model](dataprovider/baseballsavantmlb/model/pitching_box_score.go)      ||✅|\n| https://baseballsavant.mlb.com\t\t | MLB\t     | Fielding box score stats             |        Live, Full\t\t\t         |     [model](dataprovider/baseballsavantmlb/model/fielding_box_score.go)      ||✅|\n| https://baseballsavant.mlb.com\t\t | MLB\t     | Play by play                         |        Live, Full\t\t\t         |        [model](dataprovider/baseballsavantmlb/model/play_by_play.go)         ||✅|\n| https://www.espn.com/mma/     \t\t | UFC\t | Matchups (Event Details)             |           Full\t\t\t            |               [model](dataprovider/espn/mma/model/matchup.go)                ||✅|\n| https://www.espn.com/mma/     \t\t | PFL\t | Matchups (Event Details)             |           Full\t\t\t            |               [model](dataprovider/espn/mma/model/matchup.go)                | 🚩 |✅|\n| https://www.espn.com/mma/     \t\t | UFC\t | Fight details (Stats, Odds, Results) |           Full\t\t\t            |             [model](dataprovider/espn/mma/model/fightdetails.go)             ||✅|\n| https://www.espn.com/mma/     \t\t | PFL\t | Fight details (Stats, Odds, Results) |           Full\t\t\t            |             [model](dataprovider/espn/mma/model/fightdetails.go)             | 🚩 |✅|\n| https://www.nba.com     \t\t | NBA\t | Matchup |           Live, Full\t\t\t            |             [model](dataprovider/nba/model/matchup.go)             ||✅|\n| https://www.nba.com     \t\t | NBA\t | Matchup periods |           Live, Full\t\t\t            |             [model](dataprovider/nba/model/matchup_periods.go)             ||✅|\n| https://www.nba.com     \t\t | NBA\t | Traditional box score stats | Q1, Q2, Q3, Q4, H1, H2, All OT, Full\t\t\t            |             [model](dataprovider/nba/model/box_score_traditional.go)             ||✅|\n| https://www.nba.com     \t\t | NBA\t | Advanced box score stats | Q1, Q2, Q3, Q4, H1, H2, All OT, Full |             [model](dataprovider/nba/model/box_score_advanced.go)             ||✅|\n| https://www.nba.com     \t\t | NBA\t | Scoring box score stats | Q1, Q2, Q3, Q4, H1, H2, All OT, Full |             [model](dataprovider/nba/model/box_score_scoring.go)             ||✅|\n| https://www.nba.com     \t\t | NBA\t | Four factors box score stats | Q1, Q2, Q3, Q4, H1, H2, All OT, Full |             [model](dataprovider/nba/model/box_score_four_factors.go)             ||✅|\n| https://www.nba.com     \t\t | NBA\t | Misc box score stats | Q1, Q2, Q3, Q4, H1, H2, All OT, Full |             [model](dataprovider/nba/model/box_score_misc.go)             ||✅|\n| https://www.nba.com     \t\t | NBA\t | Usage box score stats | Q1, Q2, Q3, Q4, H1, H2, All OT, Full |             [model](dataprovider/nba/model/box_score_usage.go)             ||✅|\n| https://www.nba.com     \t\t | NBA\t | Defense box score stats | Full\t\t\t            |             [model](dataprovider/nba/model/box_score_defense.go)             ||✅|\n| https://www.nba.com     \t\t | NBA\t | Tracking box score stats | Full\t\t\t            |             [model](dataprovider/nba/model/box_score_tracking.go)             ||✅|\n| https://www.nba.com     \t\t | NBA\t | Hustle box score stats | Full\t\t\t            |             [model](dataprovider/nba/model/box_score_hustle.go)             ||✅|\n| https://www.nba.com     \t\t | NBA\t | Matchups box score stats | Full\t\t\t            |             [model](dataprovider/nba/model/box_score_matchups.go)             ||✅|\n| https://www.nba.com     \t\t | NBA\t | Live box score stats | Live\t\t\t            |             [model](dataprovider/nba/model/box_score_live.go)             ||✅|\n| https://www.nba.com     \t\t | NBA\t | play by play | Live, Full\t\t\t            |             [model](dataprovider/nba/model/play_by_play.go)             ||✅|\n\n## Supported Formats\nFile formats the constructed data models support on export and import.\n|Format|Export|Import|Go Package|\n|:------|:----:|:-----:|:-----|\n|Parquet|✅|✅|[xitongsys/parquet-go](https://pkg.go.dev/github.com/xitongsys/parquet-go)|\n|JSON|✅|✅|[encoding/json](https://pkg.go.dev/encoding/json)|\n\n## Development\n### Prerequisites\nGo 1.24 or higher\n\n### Building the CLI\n```console\nmake build-sportscrape\n```\n\n### Testing\nThis project is using [mockery](https://github.com/vektra/mockery) v3.5.0 to mock interfaces.\n\nTo run unit tests:\n```console\nmake unit-tests\n```\n\nTo run unit and integration tests:\n```console\nmake all-tests\n```\n\nTests are also being ran as CI workflows on Github Actions.\n\n## License\nMIT\n\n[sportscrape-ci]: https://github.com/lightning-dabbler/sportscrape/actions/workflows/deploy.yml (Deploy CI)\n[sportscrape-ci-status]: https://github.com/lightning-dabbler/sportscrape/actions/workflows/deploy.yml/badge.svg (Deploy CI)\n[goref-sportscrape]: https://pkg.go.dev/github.com/lightning-dabbler/sportscrape\n[goref-sportscrape-status]: https://pkg.go.dev/badge/github.com/lightning-dabbler/sportscrape.svg\n[release-status]: https://img.shields.io/github/v/release/lightning-dabbler/sportscrape?display_name=tag\u0026sort=semver (Latest Release)\n[releases]: https://github.com/lightning-dabbler/sportscrape/releases (Releases)\n[go-report]: https://goreportcard.com/report/github.com/lightning-dabbler/sportscrape (Go report)\n[go-report-status]: https://goreportcard.com/badge/github.com/lightning-dabbler/sportscrape (Go report Badge)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flightning-dabbler%2Fsportscrape","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flightning-dabbler%2Fsportscrape","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flightning-dabbler%2Fsportscrape/lists"}