{"id":18829809,"url":"https://github.com/gojibjib/jibjib-model","last_synced_at":"2025-07-14T23:06:07.999Z","repository":{"id":198259775,"uuid":"130880793","full_name":"gojibjib/jibjib-model","owner":"gojibjib","description":"Machine learning model for bird songs recognition","archived":false,"fork":false,"pushed_at":"2020-02-01T13:42:21.000Z","size":6295,"stargazers_count":42,"open_issues_count":5,"forks_count":7,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-14T03:52:04.386Z","etag":null,"topics":["audio-analysis","audioset","birdsong","docker","image-classification","machinelearning","python","tensorflow","vggish"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gojibjib.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-04-24T16:06:06.000Z","updated_at":"2025-03-03T23:03:49.000Z","dependencies_parsed_at":"2023-10-05T03:54:39.734Z","dependency_job_id":null,"html_url":"https://github.com/gojibjib/jibjib-model","commit_stats":null,"previous_names":["gojibjib/jibjib-model"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/gojibjib/jibjib-model","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gojibjib%2Fjibjib-model","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gojibjib%2Fjibjib-model/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gojibjib%2Fjibjib-model/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gojibjib%2Fjibjib-model/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gojibjib","download_url":"https://codeload.github.com/gojibjib/jibjib-model/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gojibjib%2Fjibjib-model/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265365576,"owners_count":23753355,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["audio-analysis","audioset","birdsong","docker","image-classification","machinelearning","python","tensorflow","vggish"],"created_at":"2024-11-08T01:46:28.491Z","updated_at":"2025-07-14T23:06:07.975Z","avatar_url":"https://github.com/gojibjib.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# A model for bird sound classification\n\nThe model for training the bird classifier.\n\n## Repo layout\nThe complete list of JibJib repos is:\n\n- [jibjib](https://github.com/gojibjib/jibjib): Our Android app. Records sounds and looks fantastic.\n- [deploy](https://github.com/gojibjib/deploy): Instructions to deploy the JibJib stack.\n- [jibjib-model](https://github.com/gojibjib/jibjib-model): Code for training the machine learning model for bird classification\n- [jibjib-api](https://github.com/gojibjib/jibjib-api): Main API to receive database requests \u0026 audio files.\n- [jibjib-data](https://github.com/gojibjib/jibjib-data): A MongoDB instance holding information about detectable birds.\n- [jibjib-query](https://github.com/gojibjib/jibjib-query): A thin Python Flask API that handles communication with the [TensorFlow Serving](https://www.tensorflow.org/serving/) instance.\n- [gopeana](https://github.com/gojibjib/gopeana): A API client for [Europeana](https://europeana.eu), written in Go.\n- [voice-grabber](https://github.com/gojibjib/voice-grabber): A collection of scripts to construct the dataset required for model training\n\n## Overview\n\n### CNN for Spectrogram-wise Classification\nIn vggish_train.py we are training a convolutional classifier model for an arbitrary number of birds. We take a pretrained [VGGish/ Audioset](https://github.com/tensorflow/models/tree/master/research/audioset) model by Google and finetune it by letting it iterate during training on more than 80,000 audio samples of 10 second length. Please read the following papers for more information:\n\n- Hershey, S. et. al., [CNN Architectures for Large-Scale Audio Classification](https://research.google.com/pubs/pub45611.html), ICASSP 2017\n- Gemmeke, J. et. al., [AudioSet: An ontology and human-labelled dataset for audio events](https://research.google.com/pubs/pub45857.html), ICASSP 2017\n\nBefore you can start, you first need to download a VGGish checkpoint file. You can either use a checkpoint provided by ![Google](https://storage.googleapis.com/audioset/vggish_model.ckpt) or ![our](https://s3-eu-west-1.amazonaws.com/jibjib/model/jibjib_model_raw.tgz) very own model that has been additionally trained for more than 100 hours and 60 epochs on a GPU cluster inside a Docker container.\n\nThe original final layer is cut off and replaced with our own output nodes.\n\nDuring the first training step a directory containing labeled bird songs is iterated over and each .wav file is converted into a spectrogram where the x-axis is the time and the y-axis symbolyzes the frequency. For instance, this is the spectrogram of a golden eagles call:\n\n![mel spectogram](https://github.com/gojibjib/jibjib-model/blob/master/assets/steinadler_50_50.png)\n\nFurthermore, each bird class is one-hot-encoded and then in pairs of features and corresponding labels fed into the model.\nAfter, VGGish's convolutional filters run over each spectrogram and extract meaningful features. The following graphic gives a short overview about how after some convolutions and subpooling the extracted features are then fed into the fully connected layer just like in any other CNN:\n\n![mel spectogram](https://raw.githubusercontent.com/gojibjib/jibjib-model/master/assets/Typical_cnn_spectrogram.png)\n\nAfter every epoch a snapshot of the models weights and biases is saved on disk. In the next step we can restore the model to either do a query or continue with training.\n\nWe are deploying the model by enabling TensorFlow Serving to reduce response time drastically. Check out ![jibjib-query](https://github.com/gojibjib/jibjib-query) to learn more about how we implemented TensorFlow Serving for our model.\n\n### New: Convolutional LSTM for Sequence Classification\nIn train_LSTM.py we provide a Convolutional LSTM for audio event recognition. Similar to vggish_train.py it performs classification tasks on mel spectrograms. In contrast to vggish_train.py, it does not perform a classification for each spectrogram but analyzes an array of matrices and then performs a single classification on the entire sequence. C-LSTMs may outperform traditional CNNs when data only contains sparse specific features or when audio scenes are event-rich with many overlapping signals.\nThe script train_LSTM.py uses the same input function as in vggish_train.py converting .wav files into their audio footprint using mel-frequency cepstral coeeficients, separating each file into 1 second frame where each frame is made up of mel features. Simultaneously, the corresponding labels are extracted, one-hot-encoded and shown to our model further downstream at the fully connected layer. The script uses Keras as a TensorFlow wrapper to build the model and is compatible with Python3.6 or upwards.\n\n\n## Training\n\n### Docker\n\nGet the container:\n\n```\n# GPU, needs nvidia-docker installed\ndocker pull obitech/jibjib-model:latest-gpu\n\n# CPU\ndocker pull obitech/jibjib-model:latest-cpu\n```\n\nCreate folders, if necessary:\n```\nmkdir -p output/logs output/train output/model input/data\n```\n\nGet the [audioset](https://github.com/tensorflow/models/tree/master/research/audioset) checkpoint:\n\n```\ncurl -O input/vggish_model.ckpt https://storage.googleapis.com/audioset/vggish_model.ckpt\n```\n\nCopy all training folders / files into `input/data/`\n\n\nGet the [`bird_id_map.pickle`](github.com/gojibjib/voice-grabber):\n\n```\ncurl -O input/bird_id_map.pickle https://github.com/gojibjib/voice-grabber/raw/master/meta/bird_id_map.pickle\n```\n\nRun the container:\n\n```\ndocker container run --rm -d \\\n    --runtime=nvidia \\\n    -v $(pwd)/input:/model/input \\\n    -v $(pwd)/output:/model/output \\\n    obitech/jibjib-model:latest-gpu\n```\n\nFor quickly starting training run:\n\n```\n# GPU\n./train_docker.sh\n\n# CPU\n./train_docker.sh\n```\n\n### Locally\n\nClone the repo:\n\n```\ngit clone https://github.com/gojibjib/jibjib-model\n```\n\nInstall dependencies, **use python3.6 or upwards**:\n\n```\n# CPU training\npip install -r requirements.txt\n\n# GPU training\npip install -r requirements-gpu.txt\n```\n\nCopy all training folders / files into `input/data/`\n\nGet the [audioset](https://github.com/tensorflow/models/tree/master/research/audioset) checkpoint:\n\n```\ncurl -O input/vggish_model.ckpt https://storage.googleapis.com/audioset/vggish_model.ckpt\n```\n\nGet the [`bird_id_map.pickle`](github.com/gojibjib/voice-grabber):\n\n```\ncurl -O input/bird_id_map.pickle https://github.com/gojibjib/voice-grabber/raw/master/meta/bird_id_map.pickle\n```\n\nStart training:\n\n```\n# Make sure to start the script from the code/ directory !\ncd code\npython ./vggish_train.py\n```\n\nYou can then use `modelbuilder.py` to convert the model to protocol buffer.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgojibjib%2Fjibjib-model","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgojibjib%2Fjibjib-model","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgojibjib%2Fjibjib-model/lists"}