{"id":15324404,"url":"https://github.com/spaghettifunk/norman","last_synced_at":"2025-04-14T23:35:21.259Z","repository":{"id":164754297,"uuid":"631232911","full_name":"spaghettifunk/norman","owner":"spaghettifunk","description":"Realtime distributed OLAP datastore, designed to answer OLAP queries with low latency written in Go. In Active development","archived":false,"fork":false,"pushed_at":"2025-03-25T11:51:23.000Z","size":287,"stargazers_count":5,"open_issues_count":6,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-25T12:38:30.975Z","etag":null,"topics":["apache-arrow","apache-parquet","golang","olap","realtime","streaming"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spaghettifunk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-22T11:13:25.000Z","updated_at":"2024-05-02T01:23:39.000Z","dependencies_parsed_at":null,"dependency_job_id":"8e2dd94a-2182-4dfd-859b-e6338b7e6f2d","html_url":"https://github.com/spaghettifunk/norman","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spaghettifunk%2Fnorman","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spaghettifunk%2Fnorman/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spaghettifunk%2Fnorman/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spaghettifunk%2Fnorman/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spaghettifunk","download_url":"https://codeload.github.com/spaghettifunk/norman/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248980612,"owners_count":21193135,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-arrow","apache-parquet","golang","olap","realtime","streaming"],"created_at":"2024-10-01T09:26:36.772Z","updated_at":"2025-04-14T23:35:21.246Z","avatar_url":"https://github.com/spaghettifunk.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# norman\n\nRealtime distributed OLAP datastore, designed to answer OLAP queries with low latency written in Go\n\n## cURLs requests for testing\n\nTo create a new Schema\n\n```bash\ncurl -X POST -H \"Content-Type: application/json\" -d @test/commander/create_schema_flights.json http://localhost:8080/commander/v1/tenants/default/schemas\n```\n\n## Kafka initialization\n\nCreate a topic for testing the ingestion job\n\n```bash\n$ docker exec -it kafka bash\n[appuser@kafka]# cd /\n[appuser@kafka]# cd bin\n[appuser@kafka]# ./kafka-topics --create --topic test-events --bootstrap-server localhost:9092\n[appuser@kafka]# ./kafka-topics --describe --topic test-events --bootstrap-server localhost:9092\n```\n\nNow, let's write some events into the topic\n\n```bash\n[appuser@kafka]# ./kafka-console-producer --topic test-events --bootstrap-server localhost:9092\n```\n\nCopy the following JSON lines\n\n```JSON\n{\"studentID\":205,\"firstName\":\"Natalie\",\"lastName\":\"Jones\",\"gender\":\"Female\",\"subject\":\"Maths\",\"score\":3.8,\"timestampInEpoch\":1571900400000}\n{\"studentID\":205,\"firstName\":\"Natalie\",\"lastName\":\"Jones\",\"gender\":\"Female\",\"subject\":\"History\",\"score\":3.5,\"timestampInEpoch\":1571900400000}\n{\"studentID\":207,\"firstName\":\"Bob\",\"lastName\":\"Lewis\",\"gender\":\"Male\",\"subject\":\"Maths\",\"score\":3.2,\"timestampInEpoch\":1571900400000}\n{\"studentID\":207,\"firstName\":\"Bob\",\"lastName\":\"Lewis\",\"gender\":\"Male\",\"subject\":\"Chemistry\",\"score\":3.6,\"timestampInEpoch\":1572418800000}\n{\"studentID\":209,\"firstName\":\"Jane\",\"lastName\":\"Doe\",\"gender\":\"Female\",\"subject\":\"Geography\",\"score\":3.8,\"timestampInEpoch\":1572505200000}\n{\"studentID\":209,\"firstName\":\"Jane\",\"lastName\":\"Doe\",\"gender\":\"Female\",\"subject\":\"English\",\"score\":3.5,\"timestampInEpoch\":1572505200000}\n{\"studentID\":209,\"firstName\":\"Jane\",\"lastName\":\"Doe\",\"gender\":\"Female\",\"subject\":\"Maths\",\"score\":3.2,\"timestampInEpoch\":1572678000000}\n{\"studentID\":209,\"firstName\":\"Jane\",\"lastName\":\"Doe\",\"gender\":\"Female\",\"subject\":\"Physics\",\"score\":3.6,\"timestampInEpoch\":1572678000000}\n{\"studentID\":211,\"firstName\":\"John\",\"lastName\":\"Doe\",\"gender\":\"Male\",\"subject\":\"Maths\",\"score\":3.8,\"timestampInEpoch\":1572678000000}\n{\"studentID\":211,\"firstName\":\"John\",\"lastName\":\"Doe\",\"gender\":\"Male\",\"subject\":\"English\",\"score\":3.5,\"timestampInEpoch\":1572678000000}\n{\"studentID\":211,\"firstName\":\"John\",\"lastName\":\"Doe\",\"gender\":\"Male\",\"subject\":\"History\",\"score\":3.2,\"timestampInEpoch\":1572854400000}\n{\"studentID\":212,\"firstName\":\"Nick\",\"lastName\":\"Young\",\"gender\":\"Male\",\"subject\":\"History\",\"score\":3.6,\"timestampInEpoch\":1572854400000}\n```\n\nRead the events for testing\n\n```bash\n[appuser@kafka]# ./kafka-console-consumer --topic test-events --from-beginning --bootstrap-server localhost:9092\n```\n\nIf you see the events, you are done!\n\n### Ingest few thousands lines\n\nCopy the file to the kafka running container\n\n```bash\ndocker cp test/data/transcript.jsonl kafka:/home/appuser/transcript.json\n```\n\nExecute the ingestion job to create the Kafka consumer and prepare Norman for ingesting events (see the steps below) and then push these messages to the kafka topic with the following command\n\n```bash\n[appuser@kafka]# ./kafka-console-producer --topic test-events --bootstrap-server localhost:9092 \u003c /home/appuser/transcript.json\n```\n\n### Run the ingestion job\n\nCreate the schema in Norman\n\n```bash\ncurl -X POST -H \"Content-Type: application/json\" -d @test/commander/create_schema_transcript.json http://localhost:8080/commander/v1/tenants/default/tables\n```\n\nCreate the ingestion job\n\n```bash\ncurl -X POST -H \"Content-Type: application/json\" -d @test/commander/create_job_transcript.json http://localhost:8080/commander/v1/tenants/default/jobs\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspaghettifunk%2Fnorman","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspaghettifunk%2Fnorman","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspaghettifunk%2Fnorman/lists"}