{"id":39823357,"url":"https://github.com/danwiseman/scryfall-kafka","last_synced_at":"2026-01-18T13:01:21.874Z","repository":{"id":42240168,"uuid":"475915650","full_name":"danwiseman/scryfall-kafka","owner":"danwiseman","description":"Kafka Producers, Consumers, and Streams Apps Related to the Scryfall API","archived":false,"fork":false,"pushed_at":"2022-04-12T15:06:42.000Z","size":175,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-06-30T11:29:54.439Z","etag":null,"topics":["kafka","kafka-consumer","mtg","mtg-cards","scryfall-api"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/danwiseman.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-30T14:27:11.000Z","updated_at":"2024-06-30T11:29:54.439Z","dependencies_parsed_at":"2022-09-18T22:32:48.044Z","dependency_job_id":null,"html_url":"https://github.com/danwiseman/scryfall-kafka","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/danwiseman/scryfall-kafka","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danwiseman%2Fscryfall-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danwiseman%2Fscryfall-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danwiseman%2Fscryfall-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danwiseman%2Fscryfall-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/danwiseman","download_url":"https://codeload.github.com/danwiseman/scryfall-kafka/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danwiseman%2Fscryfall-kafka/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28536686,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T10:13:46.436Z","status":"ssl_error","status_checked_at":"2026-01-18T10:13:11.045Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-consumer","mtg","mtg-cards","scryfall-api"],"created_at":"2026-01-18T13:01:17.643Z","updated_at":"2026-01-18T13:01:21.861Z","avatar_url":"https://github.com/danwiseman.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Scryfall Kafka \n\nThis project is now known as **scryfall-kafka**. There will be producers for pulling scryfall data from the\nexcellent [Scryfall API](https://scryfall.com/docs/api).\n\n# Scryfall Kafka Producers\n\n## Scryfall Card Producer\n\nThis is a very basic Kafka producer that grabs ALL of the English prints in the Scryfall API. It has \nhard coded rate limits; but will keep going until they are all produced into the kafka topic.\n\n:hammer: soon an AVRO producer...\n\n## Scryfall Card Tagger Producer\n\nThis producer takes a JSON file of the [Scryfall tagger](https://tagger.scryfall.com/) tags, and produces \na JSON into Kafka with the card's name, the tag, and the type of tag. The ID chosen is the oracle_id if the \ntag is an oracle tag, otherwise the tag will be the actual card's id. The JSON used is currently in the \nresources directory. \n\n```json\n{ \n  \"name\": \"Divine Smite\", \n  \"id\": \"3b06b242-caed-4c8f-b5ab-30e86061286e\", \n  \"tag_type\": \"art\", \n  \"tags\": [\"2-people\"]\n}\n```\n# Scryfall Kafka Consumers\n\n## Scryfall Card Consumer\n\nThis is a very basic scryfall card consumer for putting cards into a MongoDB streamed from my \n[Scryfall Kafka Connect](https://github.com/danwiseman/scryfallconnect) application or the above Scyrfall \nKafka Producer. It is set up to use the default settings for a MongoDB docker. Change the settings for \nyour database.\n\n:hammer: soon an AVRO consumer...\n\n## Scryfall Card Tagger Consumer\n\nThis application will take JSONs that have Tagger tags from Scryfall, and it will then\nupdate the corresponding card in the database. This works best in conjunction with the\nabove scryfall card consumer and the Scryfall Card Tagger Producer.\n\n```json\n     {\n        \"name\": \"Abuna Acolyte\",\n        \"id\": \"9e17bbf7-00c0-46f2-9718-2762fd7388d3\",\n        \"tags\": [\"2-people\"],\n        \"tag_type\": \"art\"\n      }\n```\n\nThis could also be combined into a Kafka Streams application for fun.\n\n# Scryfall Kafka Streams\n\n## Oracle Card Streamer\n\nThis Kafka Streams application joins the individual card prints together into a smaller JSON object with some of the card\ninformation. \n\nThe JSON looks similar to below (not finalized):\n\n```json\n{\n  \"oracle_id\": \"fae37e28-e137-4177-b973-fa8b4dd8f409\",\n  \"oracle_name\": \"Generous Gift\",\n  \"colors\": [\n    \"W\"\n  ],\n  \"color_identity\": [\n    \"W\"\n  ],\n  \"color_indicator\": [\n    \"W\"\n  ],\n  \"mana_cost\": \"string\",\n  \"oracle_text\": \"string\",\n  \"power\": \"string\",\n  \"toughness\": \"string\",\n  \"layout\": \"string\",\n  \"cheapest_price\": 2.1,\n  \"highest_price\": 2.5,\n  \"cmc\": 3,\n  \"type_line\": \"Instant\",\n  \"legalities\": {\n    \"standard\": \"legal\",\n    \"future\": \"legal\",\n    \"historic\": \"legal\",\n    \"gladiator\": \"legal\",\n    \"pioneer\": \"legal\",\n    \"modern\": \"legal\",\n    \"legacy\": \"legal\",\n    \"pauper\": \"legal\",\n    \"vintage\": \"legal\",\n    \"penny\": \"legal\",\n    \"commander\": \"legal\",\n    \"brawl\": \"legal\",\n    \"historicbrawl\": \"legal\",\n    \"alchemy\": \"legal\",\n    \"paupercommander\": \"legal\",\n    \"duel\": \"legal\",\n    \"oldschool\": \"legal\",\n    \"premodern\": \"legal\"\n  },\n  \"card_prints\": [\n    \"f2bb6b0c-45e8-43ff-8fd6-78fbf6979b68\",\n    \"96d4e224-e630-4696-85ef-cf42f9d03ca5\"\n  ],\n  \"card_images\": {\n    \"small\": \"string\",\n    \"normal\": \"string\",\n    \"large\": \"string\",\n    \"png\": \"string\",\n    \"art_crop\": \"string\",\n    \"border_crop\": \"string\"\n  }\n}\n```\n\n# :whale: Docker\n\nFirst build each image or run docker compose.\n\n```shell\ndocker build -t scryfall_cards_consumer --target scryfall_cards_consumer .\ndocker build -t scryfall_tagged_cards_consumer --target scryfall_tagged_cards_consumer .\n#----- OR -----\ndocker-compose -f docker-compose.yml up -d\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanwiseman%2Fscryfall-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanwiseman%2Fscryfall-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanwiseman%2Fscryfall-kafka/lists"}