{"id":13644085,"url":"https://github.com/fjcu-ee-islab/Spiking_Converted_YOLOv4","last_synced_at":"2025-04-21T06:32:56.722Z","repository":{"id":59515850,"uuid":"401365738","full_name":"fjcu-ee-islab/Spiking_Converted_YOLOv4","owner":"fjcu-ee-islab","description":"Object Detection Based on Dynamic Vision Sensor with Spiking Neural Network","archived":false,"fork":false,"pushed_at":"2023-05-29T06:12:42.000Z","size":788,"stargazers_count":19,"open_issues_count":1,"forks_count":4,"subscribers_count":0,"default_branch":"master","last_synced_at":"2024-11-09T16:43:45.837Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fjcu-ee-islab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-08-30T14:05:56.000Z","updated_at":"2024-07-17T17:41:05.000Z","dependencies_parsed_at":"2024-01-14T12:18:16.232Z","dependency_job_id":"f9178be4-03ab-40ae-9648-c11404fb3f0b","html_url":"https://github.com/fjcu-ee-islab/Spiking_Converted_YOLOv4","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fjcu-ee-islab%2FSpiking_Converted_YOLOv4","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fjcu-ee-islab%2FSpiking_Converted_YOLOv4/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fjcu-ee-islab%2FSpiking_Converted_YOLOv4/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fjcu-ee-islab%2FSpiking_Converted_YOLOv4/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fjcu-ee-islab","download_url":"https://codeload.github.com/fjcu-ee-islab/Spiking_Converted_YOLOv4/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250008286,"owners_count":21359959,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T01:01:57.347Z","updated_at":"2025-04-21T06:32:52.259Z","avatar_url":"https://github.com/fjcu-ee-islab.png","language":"Python","funding_links":[],"categories":["Object Detection Applications","Applications"],"sub_categories":[],"readme":"# Spiking_Converted_YOLOv4\nThis repository contains the official code described in the IEEE Access paper \"[Spike-event Object Detection for Neuromorphic Vision](https://ieeexplore.ieee.org/document/10016699/)\".\n\n## Citation\n```\n@ARTICLE{10016699,\n  author={Wang, Yuan-Kai and Wang, Shao-En and Wu, Ping-Hsien},\n  journal={IEEE Access}, \n  title={Spike-Event Object Detection for Neuromorphic Vision}, \n  year={2023},\n  volume={11},\n  number={},\n  pages={5215-5230},\n  doi={10.1109/ACCESS.2023.3236800}}\n```\n\n## Introduction\nObject Detection Based on Dynamic Vision Sensor with Spiking Neural Network\n\nWe provide three methods: Frequency, SAE, and LIF to convert dynamic vision sensor data into visualization data in ```Frequency、SAE、LIF``` floder\n\nWe provide object detection trained on the MNIST-DVS dataset label by the auto_labeling algorithm in ```MNIST-DVS-Detection``` floder\n\nWe provide the auto_labeling algorithm program in ```Auto_labeling_algorithm``` floder\n\nWe provide object detection trained on the PAFBenchmark dataset label by the auto_labeling algorithm in ```PAFBenchmark``` floder\n\nWe provide object detection trained on the FJU_event_pedestrian_detection dataset label by the auto_labeling algorithm in ```fju_event_pedestrian_detection``` floder\n\nWe provide object detection based on dynamic vision sensor with spiking neural network trained on the FJU_event_pedestrian_detection dataset label by the auto_labeling algorithm in ```Spiking_converted_YOLOv4``` floder\n\n![](https://github.com/fjcu-ee-islab/Spiking_Converted_YOLOv4/blob/master/flowchart/flowchart.png)\n\n## FJUPD Event Dataset\nFJUPD Event Dataset is available at [IEEE DataPort](https://dx.doi.org/10.21227/x8x3-mw77) and [FJUPD Download Link](https://u.pcloud.link/publink/show?code=kZO8ujVZz1BJJUPqzAS7v0h9M1gTJyTnhNW7).\n\n## AEDAT4 files to .jpg or .avi\nYou must clone and install [DV-python](https://gitlab.com/inivation/dv/dv-python)\n\nYou can download [AEDAT4 files](https://drive.google.com/file/d/14MbYG6216m2hCdOdjKVSkYRfqeZZ29Fr/view?usp=sharing)\n\nFirst install PIL and matplotlib\n```\ncd Frequency、SAE、LIF/\npip3 install scipy\npip3 install matplotlib\npip3 install Pillow\n```\nPlease install it if opencv is not installed\n```\npip3 install opencv-python \n```\nYou can use three methods to convert .jpg\n```\npython tool_LIF_aedat4.py\npython tool_frequency_aedat4.py\npython tool_sae_aedat4.py\n```\nOr you can use jupyter notebook for .avi\n```\npip install jupyter notebook\ntool_sae_aedat4_avi.ipynb\n```\n\n## MNIST-DVS-Detection\nYou must clone and install [Darknet](https://github.com/AlexeyAB/darknet)\nCheck the MakeFile and change the following parameters\n```\nGPU=1\t\t\t\t\t\nCUDNN=1\t\t\t\t\t\nCUDNN_HALF=0\nOPENCV=1\t\t\t\t\nAVX=0\nOPENMP=0\nLIBSO=0\nZED_CAMERA=0 # ZED SDK 3.0 and above\nZED_CAMERA_v2_8=0 # ZED SDK 2.X \n```\nEnter the darknet folder and execute\n```\ncd MNIST-DVS-Detection/darknet/\nmake\n```\nDownload and use the trained weights for testing [Weights](https://drive.google.com/file/d/1cewMfusmPjYWbrnuJRuKhPMwRe_b9PaT/view)\nUse the pictures that come with darknet for testing\n```\n./darknet detect /darknet/cfg/yolov4.cfg /yolov4.weights /darknet/data/dog.jpg\n```\nYou can download the training data that we have marked [download](https://drive.google.com/file/d/1X1C-MsoPxtH6S5pBU_F2WlpOiPYIzM2Q/view?usp=sharing)\nEnter the train and dev folders respectively and execute the following programs to generate txt files with absolute paths\n```\ncd train\nls -d \"$PWD\"/*.jpg \u003e train.txt \ncd dev\nls -d \"$PWD\"/*.jpg \u003e dev.txt \n```\nChange the .cfg file\nbatch、subdivisions \n```\nbatch = 64\nsubdivisions = 16           //Can be adjusted according to the memory\n```\nChange max_batches = clsss * 2000\n```\nmax_batches = 20000 \n```\nChange steps = max_batches * 0.8, 0.9\n```\nsteps = 16000, 18000 \n```\nChange width and height (must be a multiple of 32)\n```\nwidth = 416\nheight = 416 \n```\nChange the classes of the three [yolo] blocks to the categories that need to be identified\n```\nclasses=10\n```\nThe filter of the previous [convolution] block of the three [yolo] blocks is changed to (classes + 5) x 3, we have 3 categories so it is changed to 24, remember that there are three places to modify\n```\nfilters = 45\n```\nAdd .name file and .data file\n.name file is the object type to be recognized\n```\n0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n```\n.data file. Store some parameters, the number of object types, and the path (train.txt \u0026 dev.txt in the previous step)\n```\nclasses=Number of object classes\ntrain=data/train.txt (the train.txt path of the previous step)\nvalid=data/dev.txt (dev.txt path in the previous step)\nnames=data/mask.names (.names file path)\nbackup=backup/ (Weight storage path)\n```\nTo start training, first download the pre-training weights trained by others [download](https://drive.google.com/file/d/1JKF-bdIklxOOVy-2Cr5qdvjgGpmGfcbp/view)\n```\n./darknet detector train /mnist.data /yolov4_MNIST_DVS512.cfg /yolov4.conv.137 \n```\nUse a single image for testing\n```\n./darknet detector test /mnist.data /yolov4_MNIST_DVS512.cfg /yolov4_last.weights /images.jpg\n```\nUse our test program\n```\npython test.py\n```\n## Auto_labeling algorithm\nThe environmental requirements we use are\n```\nWindow 10\nVisual studio 2017\nOpenCV - 3.4.11\nOpenCV_contrib - 3.4.11\nCmake 3.10.0\n```\nIf you finish installing OpenCV, you can use the following program to test\n```\n/Auto_labeling_algorithm/test_opencv.cpp\n```\nThe following programs can be used to automatically mark and test OpenCV_contrib, and most of the other programs are used to remove noise\n```\n/Auto_labeling_algorithm/CSRT_大量存圖_存取影像(可存原圖).cpp\n```\n## PAFBenchmark\nParameter adjustment and training methods are roughly the same as MNIST-DVS-Detection\nYou can download relevant training dataset [here](https://drive.google.com/file/d/1rhByl3rk0yGTepXOb9sQAFXmoZ19pghk/view?usp=sharing)\n## fju_event_pedestrian_detection\nParameter adjustment and training methods are roughly the same as MNIST-DVS-Detection\nYou can download relevant training dataset [here](https://drive.google.com/file/d/14MbYG6216m2hCdOdjKVSkYRfqeZZ29Fr/view?usp=sharing)\n## Spiking converted YOLOv4\nYou must clone and install [PyTorch-Spiking-YOLOv3](https://github.com/cwq159/PyTorch-Spiking-YOLOv3)\nPackage version requirements\n```\npytorch 1.3\npython 3.7\n```\nIf you encounter the following error\n```\nFile \"/home/shon/anaconda3/envs/torch1.3/lib/python3.7/site-packages/torch/tensor.py\", line 433, in __array__\n    return self.numpy()\nTypeError: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.\n```\nPlease go to the path listed above to modify the tensor.py\n```\nreturn self.numpy()\n改成\nreturn self.cpu().numpy()\n```\ntrain\n```\npython train.py --batch-size 32 --cfg /spiking_yolov4.cfg --data /fju_YOLOv4.data --weights ''\n```\ntest\n```\npython test.py --cfg /spiking_yolov4.cfg --data /fju_YOLOv4.data --weights weights/best.pt --batch-size 32 --img-size 640\n```\nCNN to SNN\n```\npython ann_to_snn.py --cfg /spiking_yolov4.cfg --data /fju_YOLOv4.data --weights weights/best.pt --timesteps 32 --batch-size 1\n```\n\n\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffjcu-ee-islab%2FSpiking_Converted_YOLOv4","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffjcu-ee-islab%2FSpiking_Converted_YOLOv4","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffjcu-ee-islab%2FSpiking_Converted_YOLOv4/lists"}