{"id":15724133,"url":"https://github.com/v4lli/aiecho","last_synced_at":"2026-02-25T13:39:03.954Z","repository":{"id":251842533,"uuid":"838043863","full_name":"v4lli/AIecho","owner":"v4lli","description":"AI-based visual environment description service (\"Be My Eyes\")","archived":false,"fork":false,"pushed_at":"2024-09-30T00:00:47.000Z","size":241,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-07T07:53:00.705Z","etag":null,"topics":["accessibility","assistant","go","gpt","react","streaming","webrtc","whip"],"latest_commit_sha":null,"homepage":"https://aiecho.unimplemented.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/v4lli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-04T19:17:55.000Z","updated_at":"2024-09-30T00:00:51.000Z","dependencies_parsed_at":"2024-10-24T17:53:55.874Z","dependency_job_id":"9b0a2033-8071-4e65-90d6-0ff15f6b7eaa","html_url":"https://github.com/v4lli/AIecho","commit_stats":null,"previous_names":["v4lli/aiecho"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/v4lli%2FAIecho","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/v4lli%2FAIecho/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/v4lli%2FAIecho/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/v4lli%2FAIecho/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/v4lli","download_url":"https://codeload.github.com/v4lli/AIecho/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247615480,"owners_count":20967183,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["accessibility","assistant","go","gpt","react","streaming","webrtc","whip"],"created_at":"2024-10-03T22:14:57.701Z","updated_at":"2025-10-30T11:48:35.983Z","avatar_url":"https://github.com/v4lli.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AI Echo - \u003csmall\u003eAI \u0026 Cloud-Based Environment Description Service\u003c/small\u003e\n\n\u003cimg align=\"left\" src=\"frontend/public/logo.png\"\u003e\n\n[AI Echo](https://aiecho.unimplemented.org/) is a service that\nprovides visually impaired people with a tool to describe their\nsurroundings in real time. The service uses a combination of image\nrecognition and language models to provide a detailed description\nof the environment. The service is open source and aims to enhance\nthe independence and quality of life of visually impaired individuals.\n\nClick [here](https://aiecho.unimplemented.org/) to try it out. 👀\n\n# Repository Overview\n- [Frontend](frontend) react-based frontend, for more information go to the [README](frontend/README.md)\n- [Pipeline](pipeline) python-based sequential pipeline for processing images, for more information go to the [README](pipeline/README.md)\n- [Whipcapture](whipcapture) adapted version of whipcapture, for more information go to the [README](whipcapture/README.md)\n\n\n## Development\n\nAll components can be built \u0026 deployed using `docker compose`:\n\n```bash\ndocker compose build \u0026\u0026 docker compose up\n```\n\nThis will bring up all functional and infrastructure components,\nexcept for the public facing reverse proxy (not required for\ndevelopment):\n\n![Architecture diagram](resources/architecture.png)\n\n* whipcapture: can be built independently (see [whipcapture](whipcapture/README.md) or use the Dockerfile: `cd whipcapture \u0026\u0026 docker build -t whipcapture .`)\n* pipeline: can be developed locally  (see [pipeline](pipeline/README.md), or use the Dockerfile: `cd pipeline \u0026\u0026 docker build -t pipeline .`)\n* frontend: use `npm install \u0026\u0026 npm run dev` in the frontend directory\n\nFor deployment, take care of the adequate TCP and UDP forwardings\n(see docker-compose.yml), configure the reverese proxy accordingly,\nand adapt the API URLs in the frontend. Additionally, a pipeline.env\nfile is required in the pipeline directory (see\n[pipeline](pipeline/README.md) for details). Also adapt the (currently\nhardcoded) server IP in the whipcapture Dockerfile. This is necessary\nfor the streaming SDP exchange to work correctly.\n\n## Background\n\nWe aim to provide a service that is aimed to give visually impaired people a tool to gather the most important parts of\ntheir surroundings, in real time. This should enhance their ability to solve problems such as finding their glasses.\nThough there are similar projects, such as [\"be my eyes\"](https://www.bemyeyes.com/), which crowdsources descriptions for\nimages, or its collaboration with OpenAI\n[\"Be My Eyes Accessibility with GPT-4o\"](https://openai.com/index/be-my-eyes/), they are closed source, which our\nproject is not. Furthermore, there are projects that aim to provide local only services, such\nas [Whishper](https://whishper.net/), which allows for transcription of videos or subtitling. Our differentiation is\nthat we aim to provide the service in real time, with an open source approach. It is interesting to apply the use of\nLLMs and Image Recognition to Accessibility,as they allow for a fully automated and scalable aid, compared to services\nsuch as \"be my eyes\".\n\n## Design\n\nThe client web app will send their video feed to the Backend, which will analyse the images and use 1 frame per second (fps)\nto perform image recognition of the objects. It will then return this output to the client and use the cumulative output\nof 5 stills to generate a description that will be sent to the client, where the built-in TTS of the browser will\ndescribe it to the client.\n\n### Frontend\n\nThe frontend uses React and Next.js, and incorporates accessibility features, such as descriptive labels, as well as a\nhigh contrast option. It allows for adjustment of the output volume and the selection of the video input as well as\naudio output device. The client will hear the description from the built-in browser API Speech Synthesis, alongside a\nmore detailed fully text-based history that shows the output of the individual frames. There is a Video Feed that shows\nthe upstreamed video for control.\n\n### Backend\n\n![](resources/pipeline.svg)\n\nThe Backend is written in Go, and it's pipeline can be seen in the figure above.\nFirst the 1 fps limit is enforced, and then similarity detection is performed on the image, comparing to the previous\nimages. This is used as metadata, as the output needs to differ if there is constant change/movement compared to a\nstatic observer. After the similarity detection is performed, the image is fed into a image to text model. The output is\nsent back to the client, and to an LLM, which provides a short summary. Here the metadata that was gathered about the\nsimilarity of frames is relevant in the prompt used. The description is then sent to the client in text form. The prompt\nshould also generate additional metadata that indicates if a message is urgent, such as tripping hazards.\n\n\n## Credits\n- [Tim Kruse](https://github.com/krusetim)\n- [Valentin Dornauer](https://github.com/v4lli)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fv4lli%2Faiecho","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fv4lli%2Faiecho","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fv4lli%2Faiecho/lists"}