Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/geekodour/chonky

img dump
https://github.com/geekodour/chonky

Last synced: 8 days ago
JSON representation

img dump

Awesome Lists containing this project

README 

        
* Chonky

** Table of contents

- [[#chonky][Chonky]]

  - [[#requirements][Requirements]]

    - [[#goal][Goal]]

      - [[#concept][Concept]]

      - [[#design-todo][Design (TODO)]]

      - [[#aspects][Aspects]]

        - [[#mood-boarding-and-idea-generation][Mood-boarding and Idea generation]]

        - [[#abstraction-on-collections][Abstraction on Collections]]

        - [[#collection][Collection]]

        - [[#search][Search]]

    - [[#not-doing][Not doing]]

    - [[#background-and-strategic-fit][Background and strategic fit]]

    - [[#assumptions-and-dependencies][Assumptions and Dependencies]]

    - [[#user-stories-use-cases][User Stories/ Use cases]]

    - [[#system-features-and-requirements][System Features and Requirements]]

      - [[#functional-requirements][Functional Requirements]]

        - [[#supported-media-types][Supported media types]]

        - [[#storage][Storage]]

      - [[#non-functional-requirements][Non-functional Requirements]]

      - [[#external-interface-requirements][External Interface Requirements]]

        - [[#web][Web]]

        - [[#pwa][PWA]]

        - [[#cli][CLI]]

        - [[#bot][Bot]]

    - [[#questions][Questions]]

  - [[#readings][Readings]]

    - [[#tagging-systems][Tagging systems]]

    - [[#overall-design-and-idea][Overall design and Idea]]

    - [[#clip][CLIP]]

  - [[#other-links][Other links]]

    - [[#photo-management][Photo management]]

  - [[#action-items][Action Items]]

    - [[#decide-on-the-tech][Decide on the tech]]

      - [[#background-jobs][Background jobs]]



** Requirements

*** Goal

**** Concept

- Tagging means nothing if at the time when I need to actually need it I can't find it

- Additionally, if your collection becomes very large, it's very easy to forget about things that you've previously seen.

- Very much resonates with this tweet that I just found: https://twitter.com/spencerc99/status/1787915562041098636

  - "excited to share Gather, an offline first app for cultivating, curating, and revisiting multimedia collections that sync with AREdotNA"

**** TODO Design (TODO)

- Chonk

- Media <> Collections

  - Boards

**** Aspects

***** Mood-boarding and Idea generation

- Discussions around entities (boards/collection/images)

- Moodboards are just views on underlying chonks

- https://kinopio.club | https://same.energy/ | https://www.are.na/about | Pinterest

***** Abstraction on Collections

Abstractions can have different operations on them.

- Eg. Cluster all photos inside the tag meme with the other tags.

- Eg. Mini Sculptures examples.

- Eg. Create a view of all the images and videos by time of entry.

- Eg. Based on the content of these selection generate a nice moodboard(the format of the moodboard needs to be thought of)

- Eg. Based on the content of these selection generate a topic of study. Fetch wikipedia pages related to the specific design methods used etc. Maybe fetch other relevant info and dump it for me.



Basically the UI should be fluid and based on the view. View is combination of the specific abstraction+operation on it. The operation can be filtering, enhancing, [[https://github.com/ivandokov/phockup][mutation]], etc etc.



***** Collection

There is 3 parts to the collection, one is ~sourcing~ the collection, second is managing the ~storage of the collection~ items, 3rd is taking care of the ~abstraction~ of the collection itself.



- Photos

  - Picture and media dump. (For media other than image, needs to support twitter, insta, and YT links.)

  - Source:

    - https://github.com/mikf/gallery-dl

- Video

  - Also my YT video manager

    - https://github.com/jmbannon/ytdl-sub

    - https://github.com/horahoradev/PrometheusTube

    - https://github.com/tubearchivist/tubearchivist

    - Other ways to enrich the video for consumption etc.

  - Videos from instagram etc

    - https://github.com/iawia002/lux also ~yt-dlp~

***** Search

All search only happens within your chonk

- Tagging

  - Automatic tagging

  - Rule based tagging (See https://github.com/tfeldmann/organize)

- Semantic search

  - Reference: https://github.com/deepfates/memery

    - I had a conversation with the dev, I forgot what the conversation was, revisit.

- Search by image (same.energy)

*** Not doing

- Not supposed to be a collaborative software

- Not a google photos alternative (This is not mean to contain sensitive stuff, it'll probably have public/private scoping later but this is more for random inspiration and for managing other pictures that you find interesting, it could be a picture that you clicked obviously but not a icloud or google photos alternative)

- Not meant to a community board but more of a "personal" dump

  - No need of moderation

- But need public/private scoping of entities (Can be implemented later)

*** Background and strategic fit

How do your actions affect product development and align with the company’s goals

*** Assumptions and Dependencies

Create a list of technical or business assumptions that the team might have.

- There might be processing heavy background jobs for each operation

*** User Stories/ Use cases

My primary usecase is *FAST* meme search, and idea dump that should be easily searchable later, there are other things also eg. I see instagram reels etc which are like "oh it'll be fun to do this with friends" and later that never happens because I don't remember it only. Just an example but many such cases. Have to think better on the "tagging"(complicated than it seems) and the interfacing aspect of the moodboard specific to the usecases at hand.

- Storing memes

- Storing screenshots

- Mood boarding

  - [[https://resources.arc.net/hc/en-us/articles/19228534606743-Share-Spaces-Folders-Splits-with-Anyone][Share Spaces, Folders & Splits with Anyone – Arc Help Center]]

  - Eg. one usecase could be, I want to build this furniture, I want to do this DIY etc.

- DIY ideas capturing

  - Follow up on some idea that was captured via some media

  - Architecture ideas capture. etc.

- Create collection of images of things

  - Eg. Gates of Guwahati

  - Eg. People wearing Red Tshirt

- What else?

*** System Features and Requirements

**** Functional Requirements

- Needs to be a binary (./chonky)

- Archival, Sourcing and classification

  - Media once cataloged should be accessible forever

  - Regex based classification.

- Sync media across devices

***** Supported media types

- Image

  - Direct upload

  - Public URL

- Video

  - Direct upload

  - Public URL/Social media links

***** Storage

- Files could be uploaded via CLI or via web/mobile

- For daemon: support getting media from local filesystems

  - local fs (Initially this)

  - cloud drives

  - object stores

  - The local daemon access fs thing will remain as is, if it encounters new files it'll upload them to chonky, and do any futher pre-processing. Otherwise it'll not do anything.

**** Non-functional Requirements

- Offline first (Local first software)

- Automatic compression of the image for various sizes (Good for later)

- Automatic captioning of things so that can be search later

- Backup of everything

  - https://github.com/rclone/rclone

- Automatic detection of duplicate images etc

**** External Interface Requirements

| Platform | Description         |

|----------+---------------------|

| Web      |                     |

| CLI      | Daemon & Client     |

| PWA      | Maybe use [[https://capacitorjs.com/][capacitor]] |

| Bot      | Zulip bot           |

- All different interfaces should be in sync

- Different interfaces can have interface specific features that are only accessible in that interface. Eg. Instead of trying to make the web application access your filesystem, maybe the cli daemon can do that for you.

***** Web

***** PWA

- More of a web client

- Mostly to be used for search

- It's not expected to run as a background application.

- If we want it to access my screenshots, the flow would be to use something like [[https://syncthing.net/][syncthing]] and get that into the machine which would be running the CLI daemon which can then sync the content of the directory to chonky catalog.

***** CLI

- Daemon & Client

- Tagging, rules management etc. (See https://github.com/2mol/pboy)

- Search from CLI

- Mostly management realted things for chonky

- I think for how we're planning to structure our media, I don't think it'll fit in nicely into the typical linux FSH. It'll be more around tags. So If someone decides to do a local export of the whole thing and wants to interact with the media files directly, we'd need a middleman to get us from the tag/graph to the actual file on disk. Think the CLI will be useful here.

***** Bot

- Bot is especially useful when we want to quickly share when using mobile. Either telegram or zulip works for me. Eg. It can be a sole thing running for one channel or it can be called upon some command, word mention etc.

*** Questions

A good practice is to record all these questions and track them.



** Readings

*** Tagging systems

- https://github.com/metabrainz/picard

- [[https://news.ycombinator.com/item?id=33248391][I am endlessly fascinated with content tagging systems | Hacker News]]

- [[https://amoffat.github.io/supertag/architecture.html][Architecture — Supertag 0.1.0 documentation]]

*** Overall design and Idea

- InkandSwitch task specific papers

- Breadboard (KilltheApp): Build context in which the different data can interoperate ([[https://www.youtube.com/watch?v=yzudS3ABHcA][Breadboard Proto Demo 1 - YouTube]])

*** CLIP

Unsure what's the state of the art is, what would be useful to us. What about multimodal?

- [[https://arxiv.org/abs/2309.16671][[2309.16671] Demystifying CLIP Data]]

  - [[https://mazzzystar.github.io/2022/12/29/Run-CLIP-on-iPhone-to-Search-Photos/][Run CLIP on iPhone to Search Photos | TL;DR]]

- SigLIP: [[https://huggingface.co/docs/transformers/en/model_doc/siglip][SigLIP]]

- [[https://news.ycombinator.com/item?id=39067615][Indexing iCloud Photos with AI Using LLaVA and Pgvector | Hacker News]]

  - https://github.com/eagledot/hachi

- [[https://llava-vl.github.io/][LLaVA]]

  - [[https://github.com/unum-cloud/uform][unum-cloud/uform]]

    - Claims to be up to 5x faster than OpenAI CLIP and LLaVA

- [[https://huggingface.co/Salesforce/blip-image-captioning-large][Salesforce/blip-image-captioning-large · Hugging Face]]

  - Running BLIP in browser: [[https://huggingface.co/spaces/radames/Candle-BLIP-Image-Captioning][Candle BLIP Image Captioning - a Hugging Face Space by radames]]

- [[https://github.com/monatis/clip.cpp][GitHub - monatis/clip.cpp: CLIP inference in plain C/C++ with no extra dependencies]]

- [[https://news.ycombinator.com/item?id=38023544][MetaCLIP – Meta AI Research | Hacker News]]

- Embeddings

  - https://montyanderson.net/writing/embeddings

** Other links

*** Photo management

Chonky is not exactly a personal photo management app, while it can be used as such. That's not the primary usecase and I'll not be focusing on it. Here are some good alternatives to pick from

- https://damselfly.info/

- https://github.com/immich-app/immich

  - Has a nice table of features which we can consider when checking what features we want to support

- https://github.com/LibrePhotos/librephotos

- https://www.photoprism.app/

- https://photoview.github.io/

- https://github.com/gbbirkisson/spis

- Ref image moodboard maker

  - https://www.reddit.com/r/macapps/comments/ruc8mf/affordable_moodboard_app_for_macios/

    - https://en.eagle.cool/

    - https://milanote.com/

    - https://www.pureref.com

** Action Items

*** Decide on the tech

**** Background jobs

- [[https://www.reddit.com/r/golang/comments/1b80zsn/goqite_is_a_persistent_message_queue_go_library/][Reddit - Dive into anything]]

- https://github.com/pocketbase/pocketbase/discussions/2101

- https://github.com/pocketbase/pocketbase/discussions/4547

** Meeting notes from Spencer's session

- I want to remember folder

  - Is there any recall algorithm?

- digital and physical world

  - different databases and datastore

  - so now we build an unified layer of top of it for search

- My problem with zulip

  - Too many channels

  - Would be nice for it to learn from my data curation and then in my downtime the plucking becomes easier

  - The meta problem that charles mentioned that deciding which technology/medium to use in the first place

- communal computing

  - software made by community and brings the community together

  - explorantion of how does data ownership look like something like thta?

  - https://www.are.na/spencer-chang/communal-computing-infrastructure

** Ideas/Doubts

*** Metadata

**** Tools that use metadata for search

Media metadata tags is interesting, but when i tried searching for tools that do this, there is not much interest apparently and the tooling is scattered/manual and not that great.

- [[https://karl-voit.at/2019/11/26/Tagging-Files-With-Windows-10/][Tagging Files With Windows 10]]

- [[https://unix.stackexchange.com/questions/705727/indexing-files-metadata][filesystems - Indexing files' metadata - Unix & Linux Stack Exchange]]

- [[https://stackoverflow.com/questions/55957673/how-to-add-metadata-to-a-folder-on-mac-linux/55958866#55958866][node.js - How to add metadata to a folder on mac/linux - Stack Overflow]]

- [[https://unix.stackexchange.com/questions/31063/is-there-a-file-search-engine-like-everything-in-linux][filesystems - Is there a file search engine like "Everything" in Linux? - Unix & Linux Stack Exchange]]

- [[https://stackoverflow.com/questions/74687084/how-to-search-for-keywords-in-metadata-across-all-files-in-a-folder-recursively][bash - How to search for keywords in metadata across all files in a folder recursively? - Stack Overflow]]

- [[https://superuser.com/questions/1734380/can-you-tag-a-file-with-metadata-and-later-search-for-files-by-metadata][linux - Can you tag a file with metadata and later search for files by metadata? - Super User]]

- [[https://askubuntu.com/questions/1295602/how-to-find-and-return-the-names-of-all-images-containing-specific-metadata][command line - How to find and return the names of all images containing specific metadata? - Ask Ubuntu]]

- linux and win tools

  - https://github.com/ozeidan/gosearch

  - https://github.com/ngirard/lolcate-rs

  - https://www.voidtools.com/

  - https://www.recoll.org/index.html

  - https://api.kde.org/frameworks/baloo/html/searching.html

  - [[https://openimageio.readthedocs.io/en/latest/igrep.html][Searching Image Metadata With igrep - OpenImageIO 2.6.2 documentation]]

**** Should we use metadata tags?

- [[https://www.reddit.com/r/googlephotos/comments/1aiathr/in_which_metadata_field_exif_or_iptc_do_you_store/][There are some attempts]] wherer ppl try to use metadata in the file. Options are IPTC, XMP and EXIF but tooling is bad

  - [[https://www.reddit.com/r/AskPhotography/comments/w0g5n1/exif_iptc_xmp_which_one_should_i_keep/][EXIF, IPTC, XMP: Which one should I keep? : AskPhotography]]

  - [[https://en.wikipedia.org/wiki/Digital_Negative][Digital Negative - Wikipedia]]

  - https://savemetadata.org/ (WG are almost dead)

**** Search interface

- https://github.com/kashav/fsql : SQL for search?

*** Media browser

- So metadata in the file seems like a not so great idea for a tool of this kind.

  - And people need to do what we call [[https://en.wikipedia.org/wiki/Folksonomy][Folksonomy - Wikipedia]]

- So the other way is to maintain a separate metata and search management thing. I like how [[https://tmsu.org/][TMSU]] works by mounting a virtual FS. but i don't think this will work as is in windows. so we'll think.

  #+begin_quote

TMSU is a command line tool to tag files. You define a set of tags and with the tools you assign any number of tags to a file. Tags can have different values, like year=1992, or year=2002. It can also make query based on those tags.



My workflow with documents is as follows:



- Identify the file or the url and get as much information as possible, iso standard number, isbn, doi or rfc number. I tried automated extraction but usually is not worth the time.

- Hash the file or the url contents and rename the file with the hash name

- Obtain metadata from the file using the iso number, isbn, etc... for as many sources as possible

- Store each metadata and associate with the file

- Obtain relevant tags from the metadata (XQuery, JSON query)...

- Tag the files with tmsu.



After all this steps I can make complex queries over files and links like:

- I want all the rfcs about mime formats written before 2010 that are not superseded.

- I want all the books from Oxford University Press that have the topic France



The workflow warranties that the classification can be regenerated from the raw data and that no information is lost. I don't use most of the metadata I query but in case I need more I can regenerate everything just changing the xml queries.



  #+end_quote

- Examples

  - https://github.com/stashapp/stash

  - [[https://www.reddit.com/r/StableDiffusion/comments/zsqpdq/i_made_a_tool_that_lets_you_easily_browse_search/][I made a tool that lets you easily browse, search, and manage all Stablediffusion images on your machine (Mac, Windows, Linux) : StableDiffusion]]

  - [[https://www.reddit.com/r/StableDiffusion/comments/102ca1u/breadboard_a_stablediffusion_browser_version_010/][Breadboard - A Stablediffusion Browser, version 0.1.0 release : StableDiffusion]]