{"id":15208079,"url":"https://github.com/skp-1997/bigvideoanalytics","last_synced_at":"2026-01-24T05:02:35.294Z","repository":{"id":253503899,"uuid":"843709098","full_name":"skp-1997/BigVideoAnalytics","owner":"skp-1997","description":"Building video analytics framework for large scale application using Big Data.","archived":false,"fork":false,"pushed_at":"2024-08-17T08:30:14.000Z","size":757,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-29T11:44:56.890Z","etag":null,"topics":["bigdata","docker","docker-compose","kafka","python","spark","videoanalytics"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/skp-1997.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-17T07:13:06.000Z","updated_at":"2024-08-17T08:30:17.000Z","dependencies_parsed_at":"2024-08-17T08:29:56.389Z","dependency_job_id":"418ad2ea-eb30-46dc-a373-15628414817e","html_url":"https://github.com/skp-1997/BigVideoAnalytics","commit_stats":{"total_commits":6,"total_committers":2,"mean_commits":3.0,"dds":"0.16666666666666663","last_synced_commit":"98488ebb3768c1a5129b9b1b8c35bc1e6825800f"},"previous_names":["skp-1997/bigvideoanalytics"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/skp-1997/BigVideoAnalytics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skp-1997%2FBigVideoAnalytics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skp-1997%2FBigVideoAnalytics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skp-1997%2FBigVideoAnalytics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skp-1997%2FBigVideoAnalytics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/skp-1997","download_url":"https://codeload.github.com/skp-1997/BigVideoAnalytics/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skp-1997%2FBigVideoAnalytics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28712841,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-24T05:01:10.984Z","status":"ssl_error","status_checked_at":"2026-01-24T04:59:18.328Z","response_time":89,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigdata","docker","docker-compose","kafka","python","spark","videoanalytics"],"created_at":"2024-09-28T07:00:59.885Z","updated_at":"2026-01-24T05:02:35.276Z","avatar_url":"https://github.com/skp-1997.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Architecture of the project\n\n\n![BigDataVideo drawio](https://github.com/user-attachments/assets/52548c39-2f1e-4e20-82b2-13d24758cb2c)\n\nThe architecture consists of the following components:\n\n- Producer: Reads frames from video files or live streams and publishes them to a Kafka server. Each frame is sent to a topic corresponding to the video file name.\n- Kafka Server: Stores frames in their respective topics.\n- Spark Consumer: Consumes frames from Kafka, applies a user-defined function (UDF), such as a face detector, and pushes processed frames to a second Kafka server.\n- Final Kafka Consumer: Writes frames according to the topic name and saves the processed videos to the output folder.\n\n# Installation\n\n## Using Docker Compose\n\n\nSpin up Kafka containers for two servers (listening on ports 9093 and 9095) and Zookeeper (listening on port 2181) using Docker Compose:\n\n```\ndocker-compose up -d\n```\n## Manual Installation\n\n1. Kafka:\n\n    - Run Kafka using kafka_start.sh.\n    - Ensure you create two different server.properties files in the conf directory and adjust the broker ID and listening port.\n      \n2. Spark:\n\n    - Download and install Spark from Apache Spark Downloads.\n    - Alternatively, use the provided Dockerfile for Spark installation.\n\n3. Python Libraries:\n\n    - Create a Conda environment and install the required libraries from requirements.txt:\n      \n      ```\n      pip install -r requirements.txt\n      ```\n# Running the program\n\n1. Start the Producer:\n      ```\n      python confluentKafkaProducer\n      ```\n2. Start the Spark Consumer:\n\n  - Source the bash profile:\n      ```\n      source ~/.bash_profile\n      ```\n  - Run Spark with the following command:\n    \n      ```\n      spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.1 sparkConsumer.py\n      ```\n\n3. Start the Kafka Consumer:\n\n      ```\n      python kafkaConsumer.py\n      ```\n\n\n# Useful Tips\n\n- To ensure Spark can access Conda environment libraries, set these environment variables:\n\n      \n      export PYSPARK_PYTHON=$(which python)\n      export PYSPARK_DRIVER_PYTHON=$(which python)\n      \n- To list running Kafka topics:\n\n      \n      bin/kafka-topics.sh --list --bootstrap-server localhost:PORT\n      \n\n- To delete a Kafka topic:\n\n      \n      kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic your_topic_name\n\n- Some suggestions:\n\n  * Here, I have used only two brokers with replication factor of 2, you can update it as per the requirements.\n  * I have taken only one partition each topic, you can update it for faster processing.\n  * You can use Kafka streaming API instead of Spark for processing frames.\n  * You can work on tracking objects across the frames. The basic code is there in repo.\n  * I am using .csv file to read camera metadata. You can use other databases for storing camera details.\n      \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskp-1997%2Fbigvideoanalytics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fskp-1997%2Fbigvideoanalytics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskp-1997%2Fbigvideoanalytics/lists"}