{"id":25905674,"url":"https://github.com/jeffasante/nimble","last_synced_at":"2026-06-09T10:31:48.954Z","repository":{"id":279094105,"uuid":"937704638","full_name":"jeffasante/nimble","owner":"jeffasante","description":"A diffusion-based model for converting images into vector sketches, inspired by human drawing processes. Transforms bitmap images into point-based vector representations using JAX and Flax.","archived":false,"fork":false,"pushed_at":"2025-02-23T18:02:12.000Z","size":549,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-03T05:16:31.048Z","etag":null,"topics":["computer-vision","diffusion-models","jax","sketch-generation"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jeffasante.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-23T17:51:11.000Z","updated_at":"2025-02-23T18:02:15.000Z","dependencies_parsed_at":"2025-02-23T18:47:24.398Z","dependency_job_id":null,"html_url":"https://github.com/jeffasante/nimble","commit_stats":null,"previous_names":["jeffasante/nimble"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jeffasante/nimble","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffasante%2Fnimble","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffasante%2Fnimble/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffasante%2Fnimble/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffasante%2Fnimble/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jeffasante","download_url":"https://codeload.github.com/jeffasante/nimble/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffasante%2Fnimble/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34103355,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-09T02:00:06.510Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","diffusion-models","jax","sketch-generation"],"created_at":"2025-03-03T05:16:32.871Z","updated_at":"2026-06-09T10:31:48.925Z","avatar_url":"https://github.com/jeffasante.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Nimble: Image-to-Vector Sketch Generation\n\nA diffusion-based model that converts images into vector sketches, inspired by human drawing processes.\n\n## Architecture\n\n```mermaid\nflowchart TB\n    %% Input subgraph\n    subgraph Input[\" \"]\n        bitmap[Bitmap Image]\n        coords[Target Coordinates]\n    end\n    \n    %% Diffusion Process subgraph\n    subgraph DiffusionProcess[\" \"]\n        timestep[Sample Timestep t]\n        noise[Add Noise]\n        noised_coords[Noised Coordinates]\n    end\n    \n    %% Encoder Module subgraph\n    subgraph EncoderModule[\"Bitmap Encoder\"]\n        conv1[Initial Conv Layer]\n        ir1[Inverted Residual Block 1]\n        ir2[Inverted Residual Block 2]\n        se[SE Block]\n        pool[Global Average Pooling]\n        embedding[Image Embedding]\n    end\n    \n    %% Decoder Module subgraph\n    subgraph DecoderModule[\"Transformer Decoder\"]\n        pos[Positional Encoding]\n        time[Timestep Embedding]\n        subgraph DecoderLayer[\"Decoder Layer x6\"]\n            sa[Self Attention]\n            ca[Cross Attention]\n            ff[Feed Forward]\n            ln[Layer Norm]\n        end\n        proj[Output Projection]\n    end\n    \n    %% Loss subgraph\n    subgraph Loss[\" \"]\n        coord_loss[Coordinate Loss]\n        raster_loss[Rasterization Loss]\n        total[Total Loss]\n    end\n    \n    %% Connections\n    bitmap --\u003e conv1\n    conv1 --\u003e ir1\n    ir1 --\u003e ir2\n    ir2 --\u003e se\n    se --\u003e pool\n    pool --\u003e embedding\n    coords --\u003e noise\n    timestep --\u003e noise\n    noise --\u003e noised_coords\n    noised_coords --\u003e pos\n    timestep --\u003e time\n    embedding --\u003e ca\n    pos --\u003e sa\n    sa --\u003e ca\n    ca --\u003e ff\n    ff --\u003e ln\n    ln --\u003e proj\n    proj --\u003e coord_loss\n    proj --\u003e raster_loss\n    coord_loss --\u003e total\n    raster_loss --\u003e total\n```\n\n## Overview\nNimble takes an input image and generates a vector sketch representation using points and strokes. It employs a diffusion process to gradually denoise random points into a coherent sketch, guided by the input image.\n\n### Key Features\n- Converts images to vector sketches (sequences of 2D coordinates)\n- Works with any input image size\n- Controllable generation via temperature parameter\n- Visualizes the generation process step by step\n\n## Installation\n\n```bash\n# Clone the repository\ngit clone https://github.com/jeffasante/nimble.git\ncd nimble\n\n# Install package\npip install -e .\n```\n\n## Getting the Data\n\n1. Download the Quick, Draw! dataset:\n```bash\n# Create data directory\nmkdir -p data\n\n# Download bird sketches (simplified .ndjson format)\nwget https://storage.googleapis.com/quickdraw_dataset/full/simplified/bird.ndjson -O data/bird.ndjson\n\n# Download bird sketches binary format (optional)\nwget https://storage.googleapis.com/quickdraw_dataset/full/binary/bird.bin -O data/bird.bin\n```\n\n2. Process the data using the provided script:\n```python\nfrom dataset import PhotoSketchDataset, save_processed_data\n\n# Create and process dataset\ndataset = PhotoSketchDataset('data/bird.ndjson', num_points=100, image_size=28)\nsave_processed_data(dataset, 'data/processed')\n```\n\n## Training\n\n1. Create a training configuration file `train_config.yaml`:\n```yaml\ndata:\n  train_path: \"data/processed\"\n  image_size: 28\n  num_points: 100\n  batch_size: 32\n\nmodel:\n  embed_dim: 256\n  diffusion_steps: 1000\n\ntraining:\n  learning_rate: 3e-5\n  num_epochs: 100\n  validate_every: 1\n```\n\n2. Start training:\n```python\nfrom train import train_model\n\ntrain_model(\n    data_path='data/processed',\n    batch_size=32,\n    num_epochs=100\n)\n```\n\nThe training script will:\n- Save checkpoints to the `checkpoints` directory\n- Display progress with tqdm\n- Log training and validation metrics\n- Save the best model based on validation loss\n\n## Inference\n\n```python\nfrom nimble.inference import NimbleInference\n\n# Initialize model\ninferencer = NimbleInference(\n    checkpoint_path=\"checkpoints/best_model/checkpoint_0\",\n    image_size=(28, 28),\n    num_points=100\n)\n\n# Generate single sketch\ncoords = inferencer.generate(\n    \"test_images/bird_0.jpg\",\n    temperature=0.8  # Controls randomness (lower = more precise)\n)\n\n# Visualize generation process\ninferencer.visualize_generation(\n    \"test_images/bird_0.jpg\",\n    temperature=0.8,\n    num_steps=5\n)\n```\n\n## Project Structure\n```\ndeploy-jax/\n├── architecture-diagram.mermaid   # Architecture visualization\n├── checkpoints/                   # Model checkpoints\n├── test_images/                   # Sample test images\n├── dataset.py                     # Data loading and processing\n├── decoder.py                     # Transformer decoder implementation\n├── diffusion.py                   # Diffusion process implementation\n├── encoder.py                     # MobileNetV3-based image encoder\n├── inference.py                   # Inference and visualization\n├── losses.py                      # Training loss functions\n├── train.py                       # Training loop and validation\n├── play.ipynb                     # Interactive examples notebook\n├── readme.md                      # Project documentation\n├── setup.py                       # Package installation configuration\n```\n\n## Technical Details\n\n### Architecture Components\n- **Bitmap Encoder**: MobileNetV3-based network for processing input images\n- **Transformer Decoder**: Converts noised coordinates to sketch points\n- **Diffusion Process**: Manages the denoising sequence\n- **Loss Functions**: \n  - Coordinate loss (L2)\n  - Rasterization loss (Binary Cross-Entropy)\n\n### Model Parameters\n- Input Resolution: 28×28 pixels\n- Output: 100 2D coordinates\n- Embedding Dimension: 256\n- Diffusion Steps: 1000\n\n### Requirements\n```\njax\u003e=0.4.27\nflax\u003e=0.1.0.3\noptax\u003e=0.2.4\nnumpy\u003e=1.23.2\nPillow\u003e=10.0.0\nmatplotlib\u003e=3.7.0\ntqdm\u003e=4.65.0\n```\n\n## Citation\n\nIf you use this code in your research, please cite:\n\n```bibtex\n@software{nimble2025,\n  author = {Jeffrey Oduro Asante},\n  title = {Nimble: Image-to-Vector Sketch Generation},\n  year = {2025},\n  url = {https://github.com/jeffasante/nimble}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjeffasante%2Fnimble","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjeffasante%2Fnimble","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjeffasante%2Fnimble/lists"}