{"id":18354749,"url":"https://github.com/siddhantprateek/qdrant","last_synced_at":"2025-08-24T15:08:03.659Z","repository":{"id":184769280,"uuid":"671971248","full_name":"siddhantprateek/qdrant","owner":"siddhantprateek","description":"Inside the repository, you can find the DevOps task that was given to evaluate my skillset.","archived":false,"fork":false,"pushed_at":"2024-02-23T06:48:18.000Z","size":12422,"stargazers_count":6,"open_issues_count":2,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-27T07:11:50.384Z","etag":null,"topics":["aws","go","monitoring","terraform","vector-database"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/siddhantprateek.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-28T15:18:25.000Z","updated_at":"2025-03-11T17:42:10.000Z","dependencies_parsed_at":"2023-11-17T07:50:54.445Z","dependency_job_id":"4c53534a-d0f6-444b-becd-2fa76593bd69","html_url":"https://github.com/siddhantprateek/qdrant","commit_stats":null,"previous_names":["siddhantprateek/qdrant"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/siddhantprateek%2Fqdrant","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/siddhantprateek%2Fqdrant/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/siddhantprateek%2Fqdrant/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/siddhantprateek%2Fqdrant/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/siddhantprateek","download_url":"https://codeload.github.com/siddhantprateek/qdrant/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248741846,"owners_count":21154386,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","go","monitoring","terraform","vector-database"],"created_at":"2024-11-05T22:05:00.807Z","updated_at":"2025-04-13T16:21:49.947Z","avatar_url":"https://github.com/siddhantprateek.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Qdrant \n\n[![Build and Test Qdrant](https://github.com/siddhantprateek/qdrant/actions/workflows/app-test.yml/badge.svg)](https://github.com/siddhantprateek/qdrant/actions/workflows/app-test.yml)\n[![Publish Docker Image workflow](https://github.com/siddhantprateek/qdrant/actions/workflows/docker-publish.yml/badge.svg)](https://github.com/siddhantprateek/qdrant/actions/workflows/docker-publish.yml)\n[![Manifest Validation](https://github.com/siddhantprateek/qdrant/actions/workflows/manifest-validation.yml/badge.svg)](https://github.com/siddhantprateek/qdrant/actions/workflows/manifest-validation.yml)\n\n## Objective\n\nThe objectives of this task are as follows:\n- [x] Create a highly scalable Qdrant vector database hosted on AWS.\n- [x] Have automatic snapshotting and backup options available.\n- [x] Have a recovery mechanism from backup for the database.\n- [x] Develop an efficient mechanism to ingest around 1 million records in the database.\n- [x] Set up observability and performance monitoring with alerts on the system. \n- [x] Use Terraform to spin up the required resources.\n\n## Architecture Design\n\n![](/assets/qdrant-arch.png)\n\n## Tech Stack\n\n- `Go` \n- `Go Fiber` - Go Framework\n- `Prometheus`\n- `Grafana`\n- `Qdrant` - Vector Database\n- `Terraform` - Infra-structure as Code (IaC)\n- `Aws` - Cloud Provider\n- `Kube-Prometheus` deploys the Prometheus Operator and already schedules a Prometheus\ncalled prometheus-k8s with alerts and rules by default.\n- `Apache Kafka` - Distributed event streaming platform, It is used for creating Data Streaming Pipeline\n- `Trivy` - security scanner\n\n## Setting up the Application using `docker-compose`\n\n- To start the application in the local environment, execute the following command:\n\n```bash\n# for local enivronment\ndocker-compose up -d \n```\n- If you want to deploy the application in a production environment, use the production-specific Docker Compose configuration file by executing the following command:\n```bash\n# for production environment\ndocker-compose -f docker-compose.prod.yaml up -d\n```\n\n## Setting up `kube-prometheus`\n\n```bash\ngit clone --recursive https://github.com/prometheus-operator/kube-prometheus\ncd kube-prometheus\n\nkubectl create -f manifests/setup\nuntil kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo \"\"; done\nkubectl create -f manifests/ \n\n# or\n./kube_prom.sh\n```\n\n## K8s Monitoring Pods\n```bash\n$ kubectl get all -n monitoring  \nNAME                                       READY   STATUS    RESTARTS      AGE\npod/alertmanager-main-0                    2/2     Running   4 (17h ago)   29h\npod/alertmanager-main-1                    2/2     Running   4 (17h ago)   29h\npod/alertmanager-main-2                    2/2     Running   4 (17h ago)   29h\npod/blackbox-exporter-7d8c77d7b9-p4txc     3/3     Running   6 (17h ago)   29h\npod/grafana-79f47474f7-tsrpc               1/1     Running   2 (17h ago)   29h\npod/kube-state-metrics-8cc8f7df6-wslgq     3/3     Running   7 (86m ago)   29h\npod/node-exporter-bd97l                    2/2     Running   4 (17h ago)   29h\npod/prometheus-adapter-6b88dfd544-4rr57    1/1     Running   3 (86m ago)   29h\npod/prometheus-adapter-6b88dfd544-vhb98    1/1     Running   2 (17h ago)   29h\npod/prometheus-k8s-0                       2/2     Running   4 (17h ago)   29h\npod/prometheus-k8s-1                       2/2     Running   4 (17h ago)   29h\npod/prometheus-operator-557b4f4977-q76cz   2/2     Running   6 (86m ago)   29h\npod/qdapi-5fdb7df48b-cjfrz                 1/1     Running   0             31m\npod/qdapi-5fdb7df48b-s9l5x                 1/1     Running   0             31m\npod/qdapi-5fdb7df48b-wl82n                 1/1     Running   0             31m\npod/qdrant-db-0                            1/1     Running   0             44m\npod/qdrant-db-1                            1/1     Running   0             44m\npod/qdrant-db-2                            1/1     Running   0             44m\n\nNAME                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE\nservice/alertmanager-main       ClusterIP   10.108.137.87    \u003cnone\u003e        9093/TCP,8080/TCP            29h\nservice/alertmanager-operated   ClusterIP   None             \u003cnone\u003e        9093/TCP,9094/TCP,9094/UDP   29h\nservice/blackbox-exporter       ClusterIP   10.103.243.118   \u003cnone\u003e        9115/TCP,19115/TCP           29h\nservice/grafana                 ClusterIP   10.96.214.152    \u003cnone\u003e        3000/TCP                     29h\nservice/kube-state-metrics      ClusterIP   None             \u003cnone\u003e        8443/TCP,9443/TCP            29h\nservice/node-exporter           ClusterIP   None             \u003cnone\u003e        9100/TCP                     29h\nservice/prometheus-adapter      ClusterIP   10.107.130.104   \u003cnone\u003e        443/TCP                      29h\nservice/prometheus-k8s          ClusterIP   10.106.89.198    \u003cnone\u003e        9090/TCP,8080/TCP            29h\nservice/prometheus-operated     ClusterIP   None             \u003cnone\u003e        9090/TCP                     29h\nservice/prometheus-operator     ClusterIP   None             \u003cnone\u003e        8443/TCP                     29h\nservice/qdapi                   ClusterIP   10.104.190.99    \u003cnone\u003e        80/TCP                       45m\nservice/qdrant-db               ClusterIP   10.99.231.223    \u003cnone\u003e        6333/TCP,6334/TCP            31m\n\nNAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE\ndaemonset.apps/node-exporter   1         1         1       1            1           kubernetes.io/os=linux   29h\n\nNAME                                  READY   UP-TO-DATE   AVAILABLE   AGE\ndeployment.apps/blackbox-exporter     1/1     1            1           29h\ndeployment.apps/grafana               1/1     1            1           29h\ndeployment.apps/kube-state-metrics    1/1     1            1           29h\ndeployment.apps/prometheus-adapter    2/2     2            2           29h\ndeployment.apps/prometheus-operator   1/1     1            1           29h\ndeployment.apps/qdapi                 3/3     3            3           45m\n\nNAME                                             DESIRED   CURRENT   READY   AGE\nreplicaset.apps/blackbox-exporter-7d8c77d7b9     1         1         1       29h\nreplicaset.apps/grafana-79f47474f7               1         1         1       29h\nreplicaset.apps/kube-state-metrics-8cc8f7df6     1         1         1       29h\nreplicaset.apps/prometheus-adapter-6b88dfd544    2         2         2       29h\nreplicaset.apps/prometheus-operator-557b4f4977   1         1         1       29h\nreplicaset.apps/qdapi-5fdb7df48b                 3         3         3       31m\nreplicaset.apps/qdapi-69d5bfcc99                 0         0         0       45m\n\nNAME                                 READY   AGE\nstatefulset.apps/alertmanager-main   3/3     29h\nstatefulset.apps/prometheus-k8s      2/2     29h\nstatefulset.apps/qdrant-db           3/3     44m\n\nNAME                           SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE\ncronjob.batch/qdrant-cronjob   0 0 * * *   False     0        \u003cnone\u003e          4s\n```\n\n\n## For Recovery Mechanism in Database\n\nIn the StatefulSet configuration, I have used `volumeClaimTemplates` section to define the PVC template that will be used by each replica of the StatefulSet. Each replica will have its own PVC `PersistantVolumeClaim` with its unique identity, backed by the requested storage.\n\nWith this configuration, the Qdrant vector database instances will have their data persisted across restarts and rescheduling events, providing data durability and stability for your deployment.\n\n- When using StatefulSets, you can request persistent storage using Persistent Volume Claims (PVCs). Each Pod in the StatefulSet can have its own PVC, which can be backed by a Persistent Volume (PV). The PVs are independent of the Pods and their lifecycle, so if a Pod fails, the PV and the data it holds will remain intact.\n\n- In the case of a StatefulSet with a master-slave configuration, the master Pod is responsible for handling write operations to the data storage, while the slave Pods can read from the storage. This configuration ensures data consistency, as only one Pod is writing to the data at a time.\n\n![](./assets/qdrant-stateful.png)\n\n```yaml\napiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n  name: qdrant-db\nspec:\n  selector:\n    matchLabels:\n      app: qdrant-db\n  serviceName: qdrant-db\n  replicas: 3\n  template:\n    metadata:\n      labels:\n        app: qdrant-db\n    spec:\n      containers:\n      - name: qdrant-db\n        image: qdrant/qdrant\n        ports:\n        - containerPort: 6333\n          name: web\n        - containerPort: 6334\n          name: grpc        \n        volumeMounts:\n        - name: qdrant-data\n          mountPath: /data\n  volumeClaimTemplates:\n  - metadata:\n      name: qdrant-data\n    spec:\n      accessModes: [ \"ReadWriteOnce\" ]\n        requests:\n      resources:\n          storage: 10Gi\n```\n\n## For automatic snapshotting and backup options\n\nKubernetes will generate a Job based on the schedule provided in the CronJob. The Job will run the container with the specified image at the scheduled time and take snapshots of the `qdrant-db`.\n\n```yaml\napiVersion: batch/v1\nkind: CronJob\nmetadata:\n  name: qdrant-cronjob\nspec:\n  schedule: \"0 0 * * *\" # Run once a day at midnight\n  jobTemplate:\n    spec:\n      template:\n        spec:\n          containers:\n          - name: qdrant-db\n            image: qdrant:qdrant\n            imagePullPolicy: IfNotPresent\n            ports:\n            - containerPort: 6333\n              name: web\n            - containerPort: 6334\n              name: grpc \n            volumeMounts:\n            - name: qdrant-dump\n              mountPath: /data\n          restartPolicy: OnFailure\n```\n\n- @yearly (or @annually)\tRun once a year at midnight of 1 January\t`0 0 1 1 *`\n- @monthly\tRun once a month at midnight of the first day of the month\t`0 0 1 * *`\n- @weekly\tRun once a week at midnight on Sunday morning\t`0 0 * * 0`\n- @daily (or @midnight)\tRun once a day at midnight\t`0 0 * * *`\n- @hourly\tRun once an hour at the beginning of the hour\t`0 * * * *`\n\n```bash\ndocker build --tag qdapi .\ndocker run -p 8000:8000 -e PORT=8000 -e QDRANT_ADDR=qdrant:6334 -d qdapi\n```\n\n## To run the Application using docker-compose\n```bash\n# production\ndocker-compose -f docker-compose.prod.yaml up -d \n# development\ndocker-compose up -d \n```\n\n\n## Set up the port forwarding manually\n\n```bash\nkubectl --namespace monitoring port-forward svc/prometheus-k8s 10000:9090 \u003e/dev/null \u0026\nkubectl --namespace monitoring port-forward svc/grafana 20000:3000 \u003e/dev/null \u0026\nkubectl --namespace monitoring port-forward svc/alertmanager-main 30000:9093 \u003e/dev/null \u0026 \nkubectl --namespace monitoring port-forward svc/qdapi 8080:80 \u003e/dev/null \u0026\nkubectl --namespace monitoring port-forward svc/qdrant-db 6334:6334 \u003e/dev/null \u0026\n\n# or \n# Use the provided script to automate port forwarding\n./portforwarding.sh\n```\n\n### Accessing Services\nOnce the port forwarding is set up, you can access the following services on your local machine:\n\nGrafana Dashboard: http://localhost:20000\n\nUse this link to access the Grafana dashboard, where you can view various monitoring and analytics visualizations.\nApplication API: http://localhost:8080\n\nUse this link to access the application's API, allowing you to interact with the application programmatically.\nPrometheus Dashboard: http://localhost:10000\n\nUse this link to access the Prometheus dashboard, where you can explore and monitor various metrics collected by Prometheus.\n\n## To Run Terraform Code\n\n### Setting up AWS Access\n\n1. Create IAM User:\n- Log in to the AWS Management Console using an account with administrative privileges.\n- Navigate to the IAM service.\n- Click on \"Users\" in the left navigation pane and create a new user.\n- Add the user to a group with access to EC2. You can use an existing group with the `AmazonEC2FullAccess` policy attached, or create a custom group with the necessary EC2 permissions.\n- Take note of the Access Key ID and Secret Access Key provided during the user creation process. You will need these to configure AWS CLI access.\n\n2. Configure AWS CLI:\n  - Open a terminal or command prompt on your local machine.\n  - Run the following command and provide the Access Key ID and Secret Access Key when prompted:\n     ```bash\n     aws configure\n     ```\n\n### Running Terraform\n\n1. Clone the Repository:\n   - Clone the repository containing the Terraform code to your local machine using Git or download the code as a ZIP archive and extract it.\n\n2. Navigate to the Terraform Configuration Folder:\n   - Using the terminal or command prompt, navigate to the folder that contains the Terraform configuration files (e.g., `cd ./.terraform`).\n\n3. Initialize Terraform:\n   - Run the following command to initialize Terraform and download the necessary providers:\n     ```bash\n     terraform init\n     ```\n\n4. Plan the Terraform Deployment (Optional):\n   - It's recommended to create a Terraform plan to preview the changes before applying them. Run the following command to generate a plan:\n     ```\n     terraform plan\n     ```\n\n5. Apply the Terraform Configuration:\n   - If the plan looks good, apply the Terraform configuration to create the AWS EC2 instances. Run the following command and confirm the action:\n     ```\n     terraform apply\n     ```\n\n6. Verify the EC2 Instances:\n   - Once the Terraform apply process is complete, log in to your AWS Management Console and navigate to the EC2 service. You should see the newly created EC2 instances.\n\n### Cleaning Up\n\nIf you want to remove the resources created by Terraform, you can use the following command:\n\n```\nterraform destroy\n```\n\n## Grafana Dashboard \n\n- Screenshot of Grafana visualization dashboard.\n- Tha `dashboard.yaml` file present in `grafana` directory \n\n![](./assets/grafana-dashboard.png)\n\n## Minikube Dashboard\n\n- Screenshot of Minikube dashboard visualization.\n\n![](./assets/minikube-dash.png)\n\n## API Documentation\n\n- Get All the Collection Created\n  ```js\n  GET {{ baseURL }}/all\n  ```\n\n- Create Collection\n  ```js\n  POST {{ baseURL }}/all\n  ```\n  ```json\n  {\n      \"collectionName\": \"test_collection\"\n  }\n  ```\n- Create Field Inside Collection\n  ```js\n  POST {{ baseURL }}/field/create\n  ```\n  ```json\n  {\n      \"collectionName\": \"test_collection\",\n      \"fieldName\": \"location\"\n  }\n  ```\n\n- Insert Data into Collection\n  ```js\n  POST {{ baseURL }}/upsert\n  ```\n  ```json\n  {\n      \"id\": 2,\n      \"city\": \"New York\",\n      \"location\": \"Washington DC\",\n      \"collectionName\": \"test_collection\"\n  }\n  ```\n- Get Data by Id\n  ```js\n  POST {{ baseURL }}/data/id\n  ```\n  ```json\n  {\n      \"id\": 2,\n      \"collectionName\": \"test_collection\"\n  }\n  ```\n- DeleteCollection\n  ```js\n  DELETE {{ baseURL }}/collection/delete\n  ```\n  ```json\n  {\n      \"collectionName\": \"test_collection\"\n  }\n  ```\n\n## Setting up the Data Pipeline\n\n\n### For the Consumer\nTo set up the Kafka consumer, follow these steps:\n\n1. Open a terminal or command prompt.\n2. Navigate to the `kafka_consumer` directory using the `cd` command:\n   ```bash\n   cd kafka_consumer\n   ```\n3. Build the consumer using the following command:\n   ```bash\n   go build -o out/consumer utils.go consumer.go\n   ```\n   This command will compile the code and generate the executable file named `consumer` inside the `out` directory.\n\n### For the Producer\nTo set up the Kafka producer, follow these steps:\n\n1. Open a terminal or command prompt.\n2. Navigate to the `kafka_producer` directory using the `cd` command:\n   ```bash\n   cd kafka_producer\n   ```\n3. Build the producer using the following command:\n   ```bash\n   go build -o out/producer utils.go producer.go\n   ```\n   This command will compile the code and generate the executable file named `producer` inside the `out` directory.\n\n### Modifying Data Consumption\n\nBy default, the data consumption value is set to 10,000 (10K) records in the `kafka_producer/producer.go` file. You can modify this value to any desired number, such as 1 million (1M), by following these steps:\n\n1. Open the `kafka_producer/producer.go` file in a text editor or code editor of your choice.\n2. Locate the following section of code within the `producer.go` file:\n\n- Currently the Cosumption value is set to 10K, it can be modified to 1Mil in `kafka_producer/producer.go` file. \n\n```go\nfor n := 0; n \u003c 10000; n++ {\n    key := users[rand.Intn(len(users))]\n    payload := Payload{\n        ID:             n + 1,\n        City:           key,\n        Location:       \"Spain\",\n        CollectionName: \"test_collection\",\n    }\n    .\n    .\n    .\n}\n```\n3. Update the loop condition `10000` to your desired value. For example, to generate 1 million records, change it to `1000000`.\n4. Save the changes to the file.\n\nWith this modification, the Kafka producer will now generate the specified number of records when executed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsiddhantprateek%2Fqdrant","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsiddhantprateek%2Fqdrant","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsiddhantprateek%2Fqdrant/lists"}