{"id":19186215,"url":"https://github.com/mg52/redis-mnist-vector-search","last_synced_at":"2026-02-13T06:17:42.188Z","repository":{"id":261775488,"uuid":"857900399","full_name":"mg52/redis-mnist-vector-search","owner":"mg52","description":null,"archived":false,"fork":false,"pushed_at":"2024-11-08T10:46:16.000Z","size":13,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-20T05:32:18.910Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mg52.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-15T22:26:03.000Z","updated_at":"2024-11-08T10:46:19.000Z","dependencies_parsed_at":"2024-11-08T11:44:51.331Z","dependency_job_id":null,"html_url":"https://github.com/mg52/redis-mnist-vector-search","commit_stats":null,"previous_names":["mg52/redis-mnist-vector-search"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mg52%2Fredis-mnist-vector-search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mg52%2Fredis-mnist-vector-search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mg52%2Fredis-mnist-vector-search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mg52%2Fredis-mnist-vector-search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mg52","download_url":"https://codeload.github.com/mg52/redis-mnist-vector-search/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252979312,"owners_count":21835028,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T11:13:48.295Z","updated_at":"2026-02-13T06:17:42.113Z","avatar_url":"https://github.com/mg52.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Redis Vector Search with MNIST Dataset in Go\n\nThis repository demonstrates how to use Redis for vector similarity search on the [MNIST dataset](https://yann.lecun.com/exdb/mnist/) ([csv can be found here](https://git-disl.github.io/GTDLBench/datasets/mnist_datasets/)), leveraging Redis' RediSearch and RedisJSON modules with a Go implementation. The project achieves 96% accuracy, correctly predicting 9691 out of 10000 test images, with an efficient average search duration of 29 ms, making Redis a powerful solution for high-dimensional data search.\n \n## Setup\n\n### Step 1: Run Redis in Docker\n\nFirst, start Redis with `RediSearch` and `RedisJSON` modules using Docker. The command below also sets up persistent data storage and a password.\n\n```bash\nmkdir $HOME/redisdata\ndocker run -d --name redis-stack-server -p 6379:6379 -v $HOME/redisdata:/data -e REDIS_ARGS=\"--requirepass thepassword --appendonly yes --appendfsync always\" redis/redis-stack-server:latest\n```\n\n### Step 2: Clone the Repository\nClone this GitHub repository to your local machine:\n```bash\ngit clone https://github.com/mg52/redis-mnist-vector-search.git\ncd redis-mnist-vector-search\n```\n\n### Step 3: Clone the Repository\nInstall the required Go packages:\n```bash\ngo mod tidy\n```\n\n### Step 4: Download MNIST CSV\nDownload MNIST CSV file and replace the file paths (os.Open paths) in the code.\n\n### Step 5: Run the Code\nRun the Go application:\n```bash\ngo run .\n```\n\n## Code Explanation\n\n### 1. Creating Index\nWe are indexing vectors with 784 dimensions for MNIST data pixels in float32 using a FLAT vector index with 6 initial vectors with L2(Euclidean) distance metric.\n```bash\ncreateIndex := []interface{}{\n  \"FT.CREATE\", \"mnist_index\", \"ON\", \"JSON\",\n  \"PREFIX\", \"1\", \"number:\",\n  \"SCHEMA\", \"$.embedding\", \"AS\", \"embedding\",\n  \"VECTOR\", \"FLAT\", \"6\", \"DIM\", \"784\",\n  \"DISTANCE_METRIC\", \"L2\", \"TYPE\", \"FLOAT32\",\n}\n```\n\n### 2. Storing MNIST data as JSON in Redis\nMNIST dataset consists 70000 grayscale images of handwritten digits (0-9), each of size 28x28 pixels. We are converting those numbers in `mnist_train.csv` file to float32, dividing the pixel value with 255 and then storing as a JSON object in embedding field.\n```bash\n// Create JSON data for Redis\njsonData := fmt.Sprintf(`{\"result\": %d, \"embedding\": [%s]}`, result, embedding)\n\n// Execute the JSON.SET command directly in Redis\nkey := fmt.Sprintf(\"number:%d:%d\", i, result)\nerr = rdb.Do(ctx, \"JSON.SET\", key, \"$\", jsonData).Err()\nif err != nil {\n  return err\n}\n```\n\n\n### 3. Performing a Vector Search\nWe are performing a K-Nearest Neighbors (KNN) search on the mnist_index, finding the closest 1 vector to the provided embedding (embeddingBytes), sorts the results by distance (dist), and uses RediSearch's query dialect 2 for parameter handling. This embeddingBytes comes from `mnist_test.csv` file.\n```bash\nsearchQuery := []interface{}{\n  \"FT.SEARCH\",                           \n  \"mnist_index\",                        \n  \"*=\u003e[KNN 1 @embedding $blob AS dist]\",\n  \"SORTBY\", \"dist\",                      \n  \"PARAMS\", \"2\", \"blob\", embeddingBytes, \n  \"DIALECT\", \"2\", \n}\n```\n\n## Output\n\nWhen you run the code it prints below output:\n```bash\n...\n...\nStored JSON for number:59996:3\nStored JSON for number:59997:5\nStored JSON for number:59998:6\nStored JSON for number:59999:8\nAll data has been stored in Redis.\nTest image 0: expected = 7, found = 7 in 81ms\nTest image 1: expected = 2, found = 2 in 31ms\nTest image 2: expected = 1, found = 1 in 30ms\nTest image 3: expected = 0, found = 0 in 30ms\n...\n...\nTest image 9997: expected = 4, found = 4 in 30ms\nTest image 9998: expected = 5, found = 5 in 30ms\nTest image 9999: expected = 6, found = 6 in 29ms\nNumber of Correct guess = 9691\nNumber of Wrong guess = 309\nAccuracy = 96%\nRedis Vector Search Min Duration = 29ms\nRedis Vector Search Max Duration = 99ms\nRedis Vector Search Average Duration = 29ms\n```\n\n## References\n\nhttps://github.com/redis-stack\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmg52%2Fredis-mnist-vector-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmg52%2Fredis-mnist-vector-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmg52%2Fredis-mnist-vector-search/lists"}