{"id":23396448,"url":"https://github.com/mneedham/pinot-deep-dive","last_synced_at":"2025-09-08T05:47:12.931Z","repository":{"id":66523833,"uuid":"585505863","full_name":"mneedham/pinot-deep-dive","owner":"mneedham","description":null,"archived":false,"fork":false,"pushed_at":"2023-01-12T13:49:10.000Z","size":16,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-14T13:50:20.987Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mneedham.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-05T10:47:31.000Z","updated_at":"2023-01-12T16:55:46.000Z","dependencies_parsed_at":"2023-03-11T00:01:10.993Z","dependency_job_id":null,"html_url":"https://github.com/mneedham/pinot-deep-dive","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mneedham%2Fpinot-deep-dive","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mneedham%2Fpinot-deep-dive/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mneedham%2Fpinot-deep-dive/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mneedham%2Fpinot-deep-dive/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mneedham","download_url":"https://codeload.github.com/mneedham/pinot-deep-dive/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247892416,"owners_count":21013700,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-22T07:35:44.681Z","updated_at":"2025-04-08T17:32:37.450Z","avatar_url":"https://github.com/mneedham.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pinot Deep Dive\n\nThe instructions below assume that you're using the [tea.xyz](https://github.com/teaxyz/cli) package manager.\n\n## Install dependencies\n\n```bash\ntea +python.org pip install -r requirements.txt\n```\n\n## Generate IP addresses / Lat Longs\n\nIf you want to generate your own, you'll need to first get a token from https://ipinfo.io.\nThen run the following:\n\n```bash\nwhile true; do tea +python.org python ips.py \u003e\u003e data/ips.json; done\n```\n\n## Generate events\n\n```bash\ntea +python.org python loop.py --users 10000\n```\n\nor with Docker:\n\n```bash\ndocker build -t event-loop:0.0.1 .\ndocker run -it -v $PWD/data:/usr/src/app/data event-loop:0.0.1 python loop.py\n```\n\n```bash\ndocker run -it -v $PWD/data:/usr/src/app/data event-loop:0.0.1 python loop.py --users 1000 --events 100 | \njq -cr --arg sep 😊 '[.eventId, tostring] | join($sep)' | \nkcat -P -b localhost:9092 -t events -K😊\n```\n\nQuery Kafka stream:\n\n```bash\nkcat -C -b localhost:9092 -t events -f 'Key: %k, payload: %s\\n'\n```\n\nAdd Pinot table:\n\n```bash\ndocker run \\\n   --network deep-dive \\\n   -v $PWD/pinot/config:/config \\\n   apachepinot/pinot:0.11.0-arm64 AddTable \\\n     -schemaFile /config/schema.json \\\n     -tableConfigFile /config/table.json \\\n     -controllerHost \"pinot-controller-deep-dive\" \\\n    -exec  \n```\n\nNavigate to http://localhost:9000:\n\n```sql\nselect eventId, count(*) \nfrom events \ngroup by eventId \nlimit 10\n```\n\n## Actions\n\nWe'll have the following actions: Join, Leave, Like, Dislike\n\nhttps://schema.org/Action\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmneedham%2Fpinot-deep-dive","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmneedham%2Fpinot-deep-dive","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmneedham%2Fpinot-deep-dive/lists"}