{"id":23662961,"url":"https://github.com/di37/multiclass-image-classification-using-multimodal-llms","last_synced_at":"2025-04-10T18:54:31.936Z","repository":{"id":267329665,"uuid":"900447617","full_name":"di37/multiclass-image-classification-using-multimodal-llms","owner":"di37","description":"A comprehensive comparison of multimodal models - llama3.2-vision, minicpm-v, llava-llama3, llava, llava13:b and closed source models for animal classification tasks. This project evaluates various models' performance in classifying 10 different animal species, ranging from common to rare animals.","archived":false,"fork":false,"pushed_at":"2024-12-10T21:17:10.000Z","size":1905,"stargazers_count":8,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T16:41:51.152Z","etag":null,"topics":["artificial-intelligence","computer-vision","gemini","google-generative-ai","large-language-models","machine-learning","natural-language-processing","ollama","openai","python"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/di37.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-08T19:47:43.000Z","updated_at":"2025-02-23T11:28:40.000Z","dependencies_parsed_at":"2024-12-10T22:22:24.797Z","dependency_job_id":null,"html_url":"https://github.com/di37/multiclass-image-classification-using-multimodal-llms","commit_stats":null,"previous_names":["di37/multimodal-image-classification","di37/image-classification-using-multimodal-llms","di37/multiclass-image-classification-using-multimodal-llms"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/di37%2Fmulticlass-image-classification-using-multimodal-llms","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/di37%2Fmulticlass-image-classification-using-multimodal-llms/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/di37%2Fmulticlass-image-classification-using-multimodal-llms/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/di37%2Fmulticlass-image-classification-using-multimodal-llms/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/di37","download_url":"https://codeload.github.com/di37/multiclass-image-classification-using-multimodal-llms/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248275559,"owners_count":21076625,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","computer-vision","gemini","google-generative-ai","large-language-models","machine-learning","natural-language-processing","ollama","openai","python"],"created_at":"2024-12-29T05:15:38.028Z","updated_at":"2025-04-10T18:54:31.906Z","avatar_url":"https://github.com/di37.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Evaluating Multimodal LLMs on Image Classification: A Comparative Analysis of Open-Source and Proprietary Models\n\nThis project evaluates and compares the performance of various multimodal Large Language Models (LLMs)—both open-source and closed-source—on an animal image classification task. The repository demonstrates data sampling, model inference, output normalization, and comprehensive evaluation of accuracy, precision, recall, F1 scores. It also explores trade-offs in inference time, data handling, and output formatting, ultimately providing insights into how different models fare in visual classification.\n\n---\n\n## Key Features\n\n- **Multimodal Image Classification:**  \n  Leverages LLMs that can process visual input to classify a curated set of animal images.\n  \n- **Variety of Models:**  \n  Tests both open-source models (e.g., LLaMA variants, minicpm-v) and closed-source models (e.g., Gemini, GPT-4o) to highlight differences in performance, format consistency, and inference speed.\n\n- **Normalization of Outputs:**  \n  Implements post-processing steps to correct misspellings, verbose labels, or truncated predictions, ensuring fair and accurate metric comparisons.\n\n- **Metrics \u0026 Visualizations:**  \n  Provides accuracy, precision, recall, and F1 scores, alongside confusion matrices and inference time statistics to offer a complete performance profile.\n\n---\n\n## Repository Structure\n\n- **`custom_logger/`**  \n  Contains custom logging utilities that provide consistent, structured logs throughout the codebase.\n  \n- **`data/`**  \n  Holds input data files and model-generated results.  \n  - **`results_*.csv`:** Classification outputs for each model family (ollama, gemini, openai).  \n  - **`sampled_animals.csv`:** Lists the subset of animals and images selected for evaluation.\n\n- **`image_classification/`**  \n  Contains core logic for performing image classification using various models, handling prompts, and evaluating outputs.\n\n- **`notebooks/`**  \n  Jupyter notebooks outlining each stage of the workflow:  \n  1. **Data Gathering \u0026 Sampling:** Selecting a subset of animal images.  \n  2. **Image Classification:** Running images through all models.  \n  3. **Data Normalization:** Cleaning and standardizing outputs.  \n  4. **Model Evaluation:** Computing metrics, plotting confusion matrices, and analyzing results.\n\n- **`utilities/`**  \n  Provides helper scripts, constants, and command-line utilities (that simplify repetitive tasks and support the main codebase.\n\n- **`classify.py`**  \n  A script to run classification across various models, generating CSV results.\n\n- **`.env` \u0026 Configuration Files:**  \n  May store environment variables or keys required to access closed-source models.\n\n- **`README.md`** (this file):  \n  A high-level overview of the entire project, guiding users through setup and usage.\n\n---\n\n## Getting Started\n\n**Prerequisites:**\n\n- Python 3.10+  \n- A virtual environment (recommended)\n- Required packages listed in `requirements.txt` (if provided).\n\n**Installation Steps:**\n\n1. Clone the repository:\n   ```bash\n   git clone https://github.com/di37/multimodal-image-classification.git\n   ```\n2. Change into the project directory:\n   ```bash\n   cd multimodal-image-classification\n   ```\n3. Set up a virtual environment and install dependencies:\n   ```bash\n   conda create -n image_classification python=3.10\n   conda activate image_classification\n   pip install -r requirements.txt\n   ```\n4. Add any necessary API keys or model credentials to your `.env` file.\n\n---\n\n## Usage\n\n1. **Data Preparation:**  \n   Use `01_Data_Gathering_And_Sampling.ipynb` in `notebooks/` to generate `sampled_animals.csv`.\n\n2. **Classification:**  \n   Run the classification script to process all images:\n   ```bash\n   python classify.py\n   ```\n   This will invoke all models (open-source and closed-source) and store results in the `data/` directory.\n\n3. **Normalization:**  \n   Use `03_Data_Normalization_of_Outputs.ipynb` to clean and standardize outputs from models that require it (e.g., Ollama models).\n\n4. **Evaluation:**  \n   Finally, run `04_Models_Evaluation.ipynb` to compute metrics, generate confusion matrices, compare inference times, and produce a comprehensive report of each model’s performance.\n\n---\n\n## Results and Interpretation\n\n- **Performance Metrics:**  \n  The evaluation notebooks summarize accuracy, precision, recall, and F1. Additional charts (e.g., confusion matrices, bar plots) are generated to visualize each model’s strengths and weaknesses.\n\n- **Impact of Normalization:**  \n  By comparing pre- and post-normalization results for open source models, users can see how minor formatting issues influenced initial metrics, uncovering the true capability of open-source models.\n\n- **Trade-Offs:**  \n  Closed-source models may yield perfect results but at higher latency and possibly less flexibility. Open-source models run locally and faster but may need some refinement and prompt tuning.\n\n[Please read the article for in-depth analysis.](https://medium.com/@d.isham.ai93/evaluating-multimodal-llms-on-image-classification-a-comparative-analysis-of-open-source-and-077c5fc8a9d3)\n\n---\n\n## Contributing\n\nContributions are welcome. Please open an issue or submit a pull request if you have improvements, bug fixes, or new features to propose.\n\n---\n\n## Acknowledgments\n\n- **Data:** Sourced from [Kaggle’s animal images dataset](https://www.kaggle.com/datasets/iamsouravbanerjee/animal-image-dataset-90-different-animals).\n- **Models:**  \n  - Open-Source: `LlaVa` variants, `Llama` models and `minicpm-v`  \n  - Closed-Source: Gemini, GPT-4o, etc.\n- **Community:** Thanks to the open-source community and model developers for providing the tools and resources enabling this project.\n\n---\n\n*This README provides a roadmap for anyone looking to understand, reproduce, or build upon the project.*","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdi37%2Fmulticlass-image-classification-using-multimodal-llms","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdi37%2Fmulticlass-image-classification-using-multimodal-llms","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdi37%2Fmulticlass-image-classification-using-multimodal-llms/lists"}