{"id":43903262,"url":"https://github.com/singlestore-labs/singlestore-workshop-data-intensive-app","last_synced_at":"2026-02-06T19:20:36.018Z","repository":{"id":37652351,"uuid":"387955989","full_name":"singlestore-labs/singlestore-workshop-data-intensive-app","owner":"singlestore-labs","description":"This repo provides a starting point for building applications using SingleStore, Redpanda (by Vectorized), and the Go language. SingleStore is a scale-out relational database built for data-intensive workloads. Redpanda is a Kafka API compatible streaming platform for mission-critical workloads created by the team at Vectorized.","archived":false,"fork":false,"pushed_at":"2024-03-08T16:46:50.000Z","size":679,"stargazers_count":23,"open_issues_count":2,"forks_count":5,"subscribers_count":29,"default_branch":"main","last_synced_at":"2026-01-18T02:14:37.335Z","etag":null,"topics":["database","docker","getting-started","go","golang","memsql","quickstart","redpanda","singlestore","sql","tutorial","vectorized","workshop"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/singlestore-labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-07-21T01:19:43.000Z","updated_at":"2024-12-15T13:15:01.000Z","dependencies_parsed_at":"2024-03-08T18:02:43.949Z","dependency_job_id":null,"html_url":"https://github.com/singlestore-labs/singlestore-workshop-data-intensive-app","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/singlestore-labs/singlestore-workshop-data-intensive-app","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/singlestore-labs%2Fsinglestore-workshop-data-intensive-app","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/singlestore-labs%2Fsinglestore-workshop-data-intensive-app/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/singlestore-labs%2Fsinglestore-workshop-data-intensive-app/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/singlestore-labs%2Fsinglestore-workshop-data-intensive-app/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/singlestore-labs","download_url":"https://codeload.github.com/singlestore-labs/singlestore-workshop-data-intensive-app/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/singlestore-labs%2Fsinglestore-workshop-data-intensive-app/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29173490,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-06T16:33:35.550Z","status":"ssl_error","status_checked_at":"2026-02-06T16:33:30.716Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","docker","getting-started","go","golang","memsql","quickstart","redpanda","singlestore","sql","tutorial","vectorized","workshop"],"created_at":"2026-02-06T19:20:34.662Z","updated_at":"2026-02-06T19:20:35.998Z","avatar_url":"https://github.com/singlestore-labs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Workshop: Building data-intensive apps with SingleStore, Redpanda, and Golang\n\n**Attention**: The code in this repository is intended for experimental use only and is not fully tested, documented, or supported by SingleStore. Visit the [SingleStore Forums](https://www.singlestore.com/forum/) to ask questions about this repository.\n\n\u003e 👋 Hello! I'm [@carlsverre][gh-carlsverre], and I'll be helping you out today\n\u003e while we build a data-intensive application. Since everyone tends to work at\n\u003e different speeds, this workshop is designed to be self-guided with assistance\n\u003e as needed. If you are taking this workshop on your own and get stuck, feel\n\u003e free to ask for help in the [SingleStore forums][s2-forums] via a\n\u003e [Github Issue][gh-issue]\n\nThis repo provides a starting point for building applications using SingleStore,\nRedpanda (by [Vectorized][vectorized]), and the Go language. SingleStore is a\nscale-out relational database built for data-intensive workloads. Redpanda is a\nKafka API compatible streaming platform for mission-critical workloads created\nby the team at Vectorized.\n\nWhen you finish this workshop, you will have built a simple data-intensive\napplication. You might be wondering, what is a data-intensive application. **An\napplication is data-intensive when data defines its constraints.** This can\nmanifest itself in many ways:\n\n- your application runs a complex query workload against multiple systems\n- you depend on data from many sources\n- you serve complex analytics to customers\n- as your customer base grows your data increases exponentially in volume or\n  velocity\n\nBecause of the inherent complexity of building a data-intensive application,\nthey tend to grow out of control into sprawling systems which look something\nlike this:\n\n![data-intensive-app](data/images/data-intensive-app.png)\n\nSingleStore has always been focused on simplifying the diagram above which is\nwhy our customer's tend to have systems that look a bit more like this:\n\n![singlestore-easy](data/images/singlestore-easy.png)\n\nBy following the workshop documented in this file, you will build a\ndata-intensive application. During the tutorial, you will accomplish the\nfollowing tasks:\n\n1. Prepare your environment\n2. Write a digital-twin\n3. Define a schema and load the data using Pipelines\n4. Expose business logic via an HTTP API\n5. Visualize your data\n\nOnce complete, you will have an application architecture which looks something\nlike this:\n\n![workshop-result](data/images/what-are-we-building.png)\n\nLet's get started!\n\n## 1. Prepare your environment\n\nBefore we can start writing code, we need to make sure that your environment is\nsetup and ready to go.\n\n1. Make sure you clone this git repository to your machine\n\n   ```bash\n   git clone https://github.com/singlestore-labs/singlestore-workshop-data-intensive-app.git\n   ```\n\n2. Make sure you have docker \u0026 docker-compose installed on your machine\n   - [docker][docker]\n   - [docker-compose][docker-compose]\n\n\u003e ❗ **Note:** If you are following this tutorial on Windows or Mac OSX you may\n\u003e need to increase the amount of RAM and CPU made available to docker. You can\n\u003e do this from the docker configuration on your machine. More information:\n\u003e [Mac OSX documentation][docker-ramcpu-osx],\n\u003e [Windows documentation][docker-ramcpu-win]\n\n\u003e ❗ **Note 2:** If you are following this tutorial on a Mac with the new M1\n\u003e chipset, it is unlikely to work. SingleStore does not yet support running on\n\u003e M1. We are actively fixing this, but in the meantime please follow along using\n\u003e a different machine.\n\n3. A code editor that you are comfortable using, if you aren't sure pick one of\n   the options below:\n   - [Visual Studio Code][vscode]\n\n     \u003e _Recommended:_ this repository is setup with VSCode launch configurations\n     \u003e as well as [SQL Tools][sqltools] support.\n\n   - [Jetbrains Goland][jetbrains-go]\n\n4. [Sign up][singlestore-signup] for a free SingleStore license. This allows you\n   to run up to 4 nodes up to 32 gigs each for free. Grab your license key from\n   [SingleStore portal][singlestore-portal] and set it as an environment\n   variable.\n\n   ```bash\n   export SINGLESTORE_LICENSE=\"singlestore license\"\n   ```\n\n### Test that your environment is working\n\n\u003e ℹ️ **Note:** This repository uses a bash script `./tasks` to run common tasks.\n\u003e If you run the script with no arguments it will print out some information\n\u003e about how to use it.\n\nBefore proceeding, please execute `./tasks up` in the root of this repository to\nboot up all of the services we need and check that your environment is working\nas expected.\n\n```bash\n$ ./tasks up\n(... lots of docker output here ...)\n   Name                  Command               State                                                                     Ports\n-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\ngrafana       /run.sh                          Up      0.0.0.0:3000-\u003e3000/tcp,:::3000-\u003e3000/tcp\nprometheus    /bin/prometheus --config.f ...   Up      0.0.0.0:9090-\u003e9090/tcp,:::9090-\u003e9090/tcp\nredpanda      /bin/bash -c rpk config se ...   Up      0.0.0.0:29092-\u003e29092/tcp,:::29092-\u003e29092/tcp, 0.0.0.0:9092-\u003e9092/tcp,:::9092-\u003e9092/tcp, 0.0.0.0:9093-\u003e9093/tcp,:::9093-\u003e9093/tcp, 9644/tcp\nsimulator     ./simulator --config confi ...   Up\nsinglestore   /startup                         Up      0.0.0.0:3306-\u003e3306/tcp,:::3306-\u003e3306/tcp, 3307/tcp, 0.0.0.0:8080-\u003e8080/tcp,:::8080-\u003e8080/tcp\n```\n\nIf the command fails or doesn't end with the above status report, you may need\nto start debugging your environment. If you are completely stumped please file\nan issue against this repository and someone will try to help you out.\n\nIf the command succeeds, then you are good to go! Lets check out the various\nservices before we continue:\n\n| service            | url                   | port | username | password |\n| ------------------ | --------------------- | ---- | -------- | -------- |\n| SingleStore Studio | http://localhost:8080 | 8080 | root     | root     |\n| Grafana Dashboards | http://localhost:3000 | 3000 | root     | root     |\n| Prometheus         | http://localhost:9090 | 9090 |          |          |\n\nYou can open the three urls directly in your browser and login using the\nprovided username \u0026 password.\n\nIn addition, you can also connect directly to SingleStore DB using any MySQL\ncompatible client or [VSCode SQL Tools][sqltools]. For example:\n\n```\n$ mariadb -u root -h 127.0.0.1 -proot\nWelcome to the MariaDB monitor.  Commands end with ; or \\g.\nYour MySQL connection id is 28\nServer version: 5.5.58 MemSQL source distribution (compatible; MySQL Enterprise \u0026 MySQL Commercial)\n\nCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.\n\nType 'help;' or '\\h' for help. Type '\\c' to clear the current input statement.\n\nMySQL [(none)]\u003e select \"hello world\";\n+-------------+\n| hello world |\n+-------------+\n| hello world |\n+-------------+\n1 row in set (0.001 sec)\n```\n\nTo make sure that Redpanda is receiving data use the following command to read\nfrom a test topic and see what the hello simulator is saying:\n\n```bash\n$ ./tasks rpk topic consume --offset latest test\n{\n \"message\": \"{\\\"message\\\":\\\"hello world\\\",\\\"time\\\":\\\"2021-07-21T21:25:50.708768659Z\\\",\\\"worker\\\":\\\"sim-0\\\"}\\n\",\n \"partition\": 3,\n \"offset\": 2,\n \"timestamp\": \"2021-07-21T21:25:50.708Z\"\n}\n{\n \"message\": \"{\\\"message\\\":\\\"hello world\\\",\\\"time\\\":\\\"2021-07-21T21:25:51.70910547Z\\\",\\\"worker\\\":\\\"sim-0\\\"}\\n\",\n \"partition\": 4,\n \"offset\": 2,\n \"timestamp\": \"2021-07-21T21:25:51.709Z\"\n}\n```\n\n## 2. Write a digital-twin\n\nNow that we have our environment setup it's time to start writing some code. Our\nfirst task is to build a digital-twin, which is basically a data simulator. We\nwill use this digital-twin to generate large volumes of real-time data we need\nto make our application truly _data-intensive_.\n\nIn this workshop, our digital-twin will be simulating simple page events for the\nSingleStore website. Lets define it's behavior:\n\n- The simulation will maintain a set of users browsing the website\n- New users will be randomly created\n- Users will \"enter\" the website at a random point in the sitemap\n- Users will navigate through the tree via adjacent nodes (up, down, sibling) at\n  random\n- Users will spend a random amount of time on each page\n- While users remain on the page we will periodically record how long they have\n  been there\n- No event will be generated when a user decides to leave the website\n\n\u003e _Note:_ This simulation is far from realistic and will result in extremely\n\u003e noisy data. A more sophisticated solution would be to take real page view data\n\u003e and use it to generate more accurate distributions (or build an AI replicating\n\u003e user behavior, left as an exercise to the reader). But for the purpose of this\n\u003e workshop, we won't worry too much about the quality of the data.\n\n\u003e _Note 2:_ I have already provided a **ton** of code and helpers to make this\n\u003e workshop run smoothly, so if it feels like something is a bit magic - I\n\u003e probably did that on purpose to keep the tutorial running smoothly. Please\n\u003e take a look through some of the provided helpers if you want to learn more!\n\n### Define our data structures\n\nWe will be working in [simulator.go](src/simulator.go) for this section of the\nworkshop, so please open that file. It should look like this:\n\n```golang\n// +build active_file\n\npackage src\n\nimport (\n  \"time\"\n)\n\nfunc (s *Simulator) Run() error {\n  testTopic := s.producer.TopicEncoder(\"test\")\n\n  for s.Running() {\n    time.Sleep(JitterDuration(time.Second, 200*time.Millisecond))\n\n    err := testTopic.Encode(map[string]interface{}{\n      \"message\": \"hello world\",\n      \"time\":    time.Now(),\n      \"worker\":  s.id,\n    })\n    if err != nil {\n      return err\n    }\n  }\n\n  return nil\n}\n```\n\n\u003e **Note:** A finished version of this file is also included at\n\u003e [simulator_finished.go](src/simulator_finished.go) in case you get stuck. You\n\u003e can switch between which one is used by changing the comment at the top of\n\u003e both files. Go will build the one with the comment `// +build active_file` and\n\u003e will ignore the one with the comment `// +build !active_file`. We will use\n\u003e this technique later on in this workshop as well.\n\nFirst, we need to define an object to track our \"users\". Modify\n[simulator.go](src/simulator.go) as you follow along.\n\n```golang\ntype User struct {\n  UserID      string\n  CurrentPage *Page\n  LastChange  time.Time\n}\n```\n\nWe will also need an object which will help us write JSON to the events topic in\nRedpanda. Go provides a feature called `struct tags` to help configure what the\nresulting JSON object should look like.\n\n```golang\ntype Event struct {\n  Timestamp int64   `json:\"unix_timestamp\"`\n  PageTime  float64 `json:\"page_time_seconds,omitempty\"`\n  Referrer  string  `json:\"referrer,omitempty\"`\n  UserID    string  `json:\"user_id\"`\n  Path      string  `json:\"path\"`\n}\n```\n\n### Data Structures\n\nTime to modify the `Run()` method which you will also find in\n[simulator.go](src/simulator.go). To start, we need a place to store our users.\nWe will use a linked list data structure since we will be adding and removing\nusers often.\n\n```golang\nfunc (s *Simulator) Run() error {\n  users := list.New()\n```\n\nWe also need a way to write events to a Redpanda topic. I have already hooked up\nthe code to Redpanda and provided a way to create what I call `TopicEncoders`. A\nTopicEncoder is a simple wrapper around a Redpanda producer which encodes each\nmessage using JSON.\n\nAs you can see in the code, there is already a `TopicEncoder` for the `test`\ntopic. Let's modify that line to instead target the events topic:\n\n```golang\nevents := s.producer.TopicEncoder(\"events\")\n```\n\n### Creating users\n\nThe `Run()` method has a main loop which starts with the line `for s.Running()`.\n`s.Running()` will return `true` until the program starts to exit at which point\nit will return `false`.\n\nWithin this loop the code \"ticks\" roughly once per second. Each time the loop\nticks we need to run a bit of simulation code.\n\nThe first thing we should do is create some users. Modify the simulation loop to\nlook something like this:\n\n```golang\nfor s.Running() {\n  time.Sleep(JitterDuration(time.Second, 200*time.Millisecond))\n  unixNow := time.Now().Unix()\n\n  // create a random number of new users\n  if users.Len() \u003c s.config.MaxUsersPerThread {\n\n    // figure out the max number of users we can create\n    maxNewUsers := s.config.MaxUsersPerThread - users.Len()\n\n    // calculate a random number between 1 and maxNewUsers\n    numNewUsers := RandomIntInRange(0, maxNewUsers)\n\n    // create the new users\n    for i := 0; i \u003c numNewUsers; i++ {\n      // define a new user\n      user := \u0026User{\n        UserID:      NextUserId(),\n        CurrentPage: s.sitemap.RandomLeaf(),\n        LastChange:  time.Now(),\n      }\n\n      // add the user to the list\n      users.PushBack(user)\n\n      // write an event to the topic\n      err := events.Encode(Event{\n        Timestamp: unixNow,\n        UserID:    user.UserID,\n        Path:      user.CurrentPage.Path,\n\n        // pick a random referrer to use\n        Referrer: RandomReferrer(),\n      })\n\n      if err != nil {\n        return err\n      }\n    }\n  }\n}\n```\n\nThen update imports at the top of the file\n\n```golang\nimport (\n  \"container/list\"\n  \"time\"\n)\n```\n\n\u003e ℹ️ **Note:** You can test your code as you go by running `./tasks simulator`.\n\u003e This command will recompile the code and run the simulator. If you have any\n\u003e errors they will show up in the output.\n\n### Simulating browsing activity\n\nNow that we have users, let's simulate them browsing the site. The basic idea is\nthat each time the simulation loop ticks, all of the users will decide to either\nstay on the current page, leave the site, or go to another page. Once again, we\nwill roll virtual dice to make this happen.\n\nAdd the following code within the `for s.Running() {` loop right after the code\nyou added in the last section.\n\n```golang\n// we will be removing elements from the list while we iterate, so we\n// need to keep track of next outside of the loop\nvar next *list.Element\n\n// iterate through the users list and simulate each users behavior\nfor el := users.Front(); el != nil; el = next {\n  // loop bookkeeping\n  next = el.Next()\n  user := el.Value.(*User)\n  pageTime := time.Since(user.LastChange)\n\n  // users only consider leaving a page after at least 5 seconds\n  if pageTime \u003e time.Second*5 {\n\n      // eventProb is a random value from 0 to 1 but is weighted\n      // to be closer to 0 most of the time\n      eventProb := math.Pow(rand.Float64(), 2)\n\n      if eventProb \u003e 0.98 {\n          // user has left the site\n          users.Remove(el)\n          continue\n      } else if eventProb \u003e 0.9 {\n          // user jumps to a random page\n          user.CurrentPage = s.sitemap.RandomLeaf()\n          user.LastChange = time.Now()\n      } else if eventProb \u003e 0.8 {\n          // user goes to the \"next\" page\n          user.CurrentPage = user.CurrentPage.RandomNext()\n          user.LastChange = time.Now()\n      }\n  }\n\n  // write an event to the topic recording the time on the current page\n  // note that if the user has changed pages above, that fact will be reflected here\n  err := events.Encode(Event{\n    Timestamp: unixNow,\n    UserID:    user.UserID,\n    Path:      user.CurrentPage.Path,\n    PageTime:  pageTime.Seconds(),\n  })\n  if err != nil {\n    return err\n  }\n}\n```\n\nThen update imports at the top of the file:\n\n```golang\nimport (\n  \"container/list\"\n  \"math\"\n  \"math/rand\"\n  \"time\"\n)\n```\n\nSweet! You have built your first digital twin! You can test it by running the\nfollowing commands:\n\n```bash\n$ ./tasks simulator\n...output of building and running the simulator...\n\n$ ./tasks rpk topic consume --offset latest events\n{\n \"message\": \"{\\\"unix_timestamp\\\":1626925973,\\\"page_time_seconds\\\":8.305101859,\\\"user_id\\\":\\\"a1e684e0-2a8f-48f3-8af3-26e55aadb86b\\\",\\\"path\\\":\\\"/blog/case-study-true-digital-group-helps-to-flatten-the-curve-with-memsql\\\"}\",\n \"partition\": 1,\n \"offset\": 3290169,\n \"timestamp\": \"2021-07-22T03:52:53.9Z\"\n}\n{\n \"message\": \"{\\\"unix_timestamp\\\":1626925973,\\\"page_time_seconds\\\":1.023932243,\\\"user_id\\\":\\\"a2e684e0-2a8f-48f3-8af3-26e55aadb86b\\\",\\\"path\\\":\\\"/media-hub/releases/memsql67\\\"}\",\n \"partition\": 1,\n \"offset\": 3290170,\n \"timestamp\": \"2021-07-22T03:52:53.9Z\"\n}\n...tons of messages, ctrl-C to cancel...\n```\n\n## 3. Define a schema and load the data using Pipelines\n\nNext on our TODO list is getting the data into SingleStore! You can put away\nyour Go skills for a moment, this step is all about writing SQL.\n\nWe will be using SingleStore Studio to work on our schema and then saving the\nfinal result in [schema.sql](schema.sql). Open http://localhost:8080 and login\nwith username: `root`, password: `root`. Once you are inside Studio, click **SQL\nEditor** on the left side.\n\nTo start, we need a table for our events. Something like this should do the\ntrick:\n\n```sql\nDROP DATABASE IF EXISTS app;\nCREATE DATABASE app;\nUSE app;\n\nCREATE TABLE events (\n    ts DATETIME NOT NULL,\n    path TEXT NOT NULL COLLATE \"utf8_bin\",\n    user_id TEXT NOT NULL COLLATE \"utf8_bin\",\n\n    referrer TEXT,\n    page_time_s DOUBLE NOT NULL DEFAULT 0,\n\n    SORT KEY (ts),\n    SHARD KEY (user_id)\n);\n```\n\nAt the top of the code you will see `DROP DATABASE...`. This makes it easy to\niterate, just select-all (`ctrl/cmd-a`) and run (`ctrl/cmd-enter`) to rebuild\nthe whole schema in SingleStore. Obviously, don't use this technique in\nproduction 😉.\n\nWe will use SingleStore Pipelines to consume the events topic we created\nearlier. This is surprisingly easy, check it out:\n\n```sql\nCREATE PIPELINE events\nAS LOAD DATA KAFKA 'redpanda/events'\nSKIP DUPLICATE KEY ERRORS\nINTO TABLE events\nFORMAT JSON (\n    @unix_timestamp \u003c- unix_timestamp,\n    path \u003c- path,\n    user_id \u003c- user_id,\n    referrer \u003c- referrer DEFAULT NULL,\n    page_time_s \u003c- page_time_seconds DEFAULT 0\n)\nSET ts = FROM_UNIXTIME(@unix_timestamp);\n\nSTART PIPELINE events;\n```\n\nThe pipeline defined above connects to Redpanda and starts loading the events\ntopic in batches. Each message is processed by the `FORMAT JSON` clause which\nmaps the event's fields to the columns in the `events` table. You can read more\nabout the powerful `CREATE PIPELINE` command [in our docs][create-pipeline].\n\nOnce `START PIPELINE` completes, SingleStore will start loading events from the\nsimulator. Within a couple seconds you should start seeing rows show up in the\nevents table. Try running `SELECT COUNT(*) FROM events`.\n\nIf you don't see any rows check `SHOW PIPELINES` to see if your pipeline is\nrunning or has an error. You can also look for error messages using the query\n`SELECT * FROM INFORMATION_SCHEMA.PIPELINES_ERRORS`.\n\n\u003e **Remember!** Once your schema works, copy the DDL statements\n\u003e (`CREATE TABLE, CREATE PIPELINE, START PIPELINE`) into\n\u003e [schema.sql](schema.sql) to make sure you don't loose it.\n\nYou can find a finished version of [schema.sql](schema.sql) in\n[schema_finished.sql](schema_finished.sql).\n\n## 4. Expose business logic via an HTTP API\n\nNow that we have successfully generated and loaded data into SingleStore, we can\neasily expose that data via a simple HTTP API. We will be working in\n[api.go](src/api.go) for most of this section.\n\nThe first query I suggest writing is a simple leaderboard. It looks something\nlike this:\n\n```sql\nSELECT path, COUNT(DISTINCT user_id) AS count\nFROM events\nGROUP BY 1\nORDER BY 2 DESC\nLIMIT 10\n```\n\nThis query counts the number of distinct users per page and returns the top 10.\n\nWe can expose the results of this query via the following function added to\n[api.go](src/api.go):\n\n```golang\nfunc (a *Api) Leaderboard(c *gin.Context) {\n  limit, err := strconv.Atoi(c.DefaultQuery(\"limit\", \"10\"))\n  if err != nil {\n    c.JSON(http.StatusBadRequest, gin.H{\"error\": \"limit must be an int\"})\n    return\n  }\n\n  out := []struct {\n    Path  string `json:\"path\"`\n    Count int    `json:\"count\"`\n  }{}\n\n  err = a.db.SelectContext(c.Request.Context(), \u0026out, `\n    SELECT path, COUNT(DISTINCT user_id) AS count\n    FROM events\n    GROUP BY 1\n    ORDER BY 2 DESC\n    LIMIT ?\n  `, limit)\n  if err != nil {\n    c.JSON(http.StatusInternalServerError, gin.H{\"error\": err.Error()})\n    return\n  }\n\n  c.JSON(http.StatusOK, out)\n}\n```\n\nMake sure you update the imports at the top of the file to:\n\n```golang\nimport (\n  \"net/http\"\n  \"strconv\"\n\n  \"github.com/gin-gonic/gin\"\n)\n```\n\nThen register the function in the `RegisterRoutes` function like so:\n\n```golang\nfunc (a *Api) RegisterRoutes(r *gin.Engine) {\n  r.GET(\"/ping\", a.Ping)\n  r.GET(\"/leaderboard\", a.Leaderboard)\n}\n```\n\nYou can run the api and test your new endpoint using the following commands:\n\n```bash\n$ ./tasks api\n...output of building and running the api...\n\n$ ./tasks logs api\nAttaching to api\napi            | [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.\napi            |\napi            | [GIN-debug] [WARNING] Running in \"debug\" mode. Switch to \"release\" mode in production.\napi            |  - using env:  export GIN_MODE=release\napi            |  - using code: gin.SetMode(gin.ReleaseMode)\napi            |\napi            | [GIN-debug] GET    /ping                     --\u003e src.(*Api).Ping-fm (3 handlers)\napi            | [GIN-debug] GET    /leaderboard              --\u003e src.(*Api).Leaderboard-fm (3 handlers)\napi            | [GIN-debug] Listening and serving HTTP on :8000\n\n$ curl -s \"localhost:8000/leaderboard?limit=2\" | jq\n[\n  {\n    \"path\": \"/blog/memsql-spark-connector\",\n    \"count\": 524\n  },\n  {\n    \"path\": \"/blog\",\n    \"count\": 507\n  }\n]\n```\n\nBefore moving forward consider creating one or two additional endpoints or\nmodifying the leaderboard. Here are some ideas:\n\n- (_easy_) change the leaderboard to accept a filter for a specific path prefix;\n  i.e. `/leaderboard?prefix=/blog`\n- (_medium_) add a new endpoint which returns a referrer leaderboard (you only\n  care about rows where referrer is `NOT NULL`)\n- (_hard_) add a new endpoint which returns the number of page loads over time\n  bucketed by minute - keep in mind that each row in the events table represents\n  a user viewing a page for 1 second\n\n## 5. Visualize your data\n\nWhew! We are almost to the end of the workshop! As a final piece of the puzzle,\nwe will visualize our data using Grafana. I have already setup grafana for you,\nso just head over to http://localhost:3000 to get started. Username and password\nare both `root`.\n\nAssuming your simulation and schema matches what is in this README, you can\ncheck out the pre-created dashboard I have already created here:\nhttp://localhost:3000/d/_TsB4vZ7k/user-analytics?orgId=1\u0026refresh=5s\n\n![grafana-dashboard](data/images/grafana_dashboard.png)\n\nI highly recommend playing around with Grafana and experimenting with its many\nfeatures. I have setup SingleStore as a datasource called \"SingleStore\" so it\nshould be pretty easy to create new dashboards and panels.\n\nFor further Grafana education, I recommend checking out their docs\n[starting here][grafana-getting-started]. Good luck!\n\n# Wrapping up\n\nCongrats! You have now learned the basics of building a data-intensive app with\nSingleStore! As a quick recap, during this workshop you have:\n\n- created a user simulator to generate random browsing data\n- ingested that data into SingleStore using Pipelines\n- exposed business analytics via an HTTP API\n- created dashboards to help you run your business\n\nWith these skills, you have learned the foundation of building data-intensive\napplications. For example, using this exact structure I built a\n[logistics simulator][logistics-sim] which simulated global package logistics at\nmassive scale. You can read more about that project on the\n[SingleStore Blog][logistics-blog].\n\nI hope you enjoyed the workshop! Please feel free to provide feedback in the\n[SingleStore forums][s2-forums] or via a [Github Issue][gh-issue].\n\nCheers!\n\n[@carlsverre][gh-carlsverre]\n\n# Contributors\n\nThis project wouldn't be complete without saying thank you to these amazing\ncontributors!\n\n- [Bailey Hayes (@ricochet)](https://github.com/ricochet)\n\n\u003c!-- Link index --\u003e\n\n[docker-compose]: https://docs.docker.com/compose/install/\n[docker-ramcpu-osx]: https://docs.docker.com/docker-for-mac/#resources\n[docker-ramcpu-win]: https://docs.docker.com/docker-for-windows/#resources\n[docker]: https://docs.docker.com/get-docker/\n[jetbrains-go]: https://www.jetbrains.com/go/\n[sqltools]: https://marketplace.visualstudio.com/items?itemName=mtxr.sqltools\n[vscode]: https://code.visualstudio.com/\n[singlestore-signup]: https://www.singlestore.com/try-free/\n[singlestore-portal]: https://portal.singlestore.com/?utm_medium=osm\u0026utm_source=github\n[create-pipeline]: https://docs.singlestore.com/db/v7.3/en/reference/sql-reference/pipelines-commands/create-pipeline.html\n[grafana-getting-started]: https://grafana.com/docs/grafana/latest/getting-started/getting-started/\n[s2-forums]: https://www.singlestore.com/forum\n[gh-issue]: https://github.com/singlestore-labs/singlestore-workshop-data-intensive-app/issues/new\n[logistics-sim]: https://github.com/singlestore-labs/singlestore-logistics-sim\n[logistics-blog]: https://www.singlestore.com/blog/scaling-worldwide-parcel-logistics-with-singlestore-and-vectorized/\n[vectorized]: https://vectorized.io/\n[gh-carlsverre]: https://github.com/carlsverre\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsinglestore-labs%2Fsinglestore-workshop-data-intensive-app","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsinglestore-labs%2Fsinglestore-workshop-data-intensive-app","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsinglestore-labs%2Fsinglestore-workshop-data-intensive-app/lists"}