{"id":30605596,"url":"https://github.com/jjmachan/aeye","last_synced_at":"2025-09-09T07:47:14.993Z","repository":{"id":43237269,"uuid":"260424921","full_name":"jjmachan/aeye","owner":"jjmachan","description":"AI for the blind","archived":false,"fork":false,"pushed_at":"2023-10-03T21:57:45.000Z","size":88,"stargazers_count":10,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-30T02:29:35.490Z","etag":null,"topics":["ai-as-a-service","face-recognition","image-captioning","object-detection","pytorch"],"latest_commit_sha":null,"homepage":"https://aeye.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jjmachan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/contributing.rst","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-01T09:45:31.000Z","updated_at":"2023-07-11T21:29:53.000Z","dependencies_parsed_at":"2023-01-22T20:15:39.127Z","dependency_job_id":null,"html_url":"https://github.com/jjmachan/aeye","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jjmachan/aeye","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jjmachan%2Faeye","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jjmachan%2Faeye/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jjmachan%2Faeye/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jjmachan%2Faeye/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jjmachan","download_url":"https://codeload.github.com/jjmachan/aeye/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jjmachan%2Faeye/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274261723,"owners_count":25251951,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-09T02:00:10.223Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-as-a-service","face-recognition","image-captioning","object-detection","pytorch"],"created_at":"2025-08-30T02:21:25.652Z","updated_at":"2025-09-09T07:47:14.984Z","avatar_url":"https://github.com/jjmachan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# A-eye\n\n[![Documentation Status](https://readthedocs.org/projects/aeye/badge/?version=latest)](https://aeye.readthedocs.io/en/latest/?badge=latest)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"500\" height=\"500\" src=\"https://github.com/jjmachan/aeye/blob/master/docs/imgs/poster.png\"\u003e\n\u003c/p\u003e\n\nAeye aspires to be an open-source AI-as-a-Service platform for hobbyists and\ntinkerers to develops apps and services for the visually challenged. We want to\nprovide the best tools and services for you to go out there and build cool stuff\nfor our less privileged brothers and sisters, to empower them to prevail over\ntheir shortcomings so that they too can come forth and contribute and fully be a\npart of our society. \n\nThe latest advancements in Computer vision thanks to technologies like Deep\nLearning we are now able to augment human vision with computer vision models\nthat help detect the objects around us, detect faces and identify who they are,\ndescribe the environment you are in and much more. This can be of great use to\nvisually challenged people, to help them perform their day to day tasks more\nefficiently. \n\nThe service offers simple to use API end points that help deliver basic deep\nlearning modules like Object Detection and Image Captioning.\n\n## Modules\n\nCurrently we have the following modules planned. Each module is an individual\nPython module that is has functionality to perform at least inferencing when\nprovided with the pretrained models and other artifacts. These modules are\nimported into Aeye and the necessary endpoints are created.\n\nThe module we support are:\n\n1. [Image Captioning](https://github.com/jjmachan/imagecaptioning-aeye) - This generates a description of the image that is passed\n   to it. The current implementation is based on the [Show and\n   Tell](https://arxiv.org/abs/1411.4555) paper. This can be used to generate a\n   description of the surrounding and is able to give some idea to the user\n   about his/her surroundings. Please refer the Module Repo or the docs for more\n   information.\n   \n   \u003cp align=\"center\"\u003e\n    \u003cimg width=\"684\" height=\"465\" src=\"https://github.com/jjmachan/aeye-demo/blob/master/demo/football_captioned.jpeg\"\u003e\n    \u003cp align=\"center\"\u003e \"a group of young men playing a game of soccer\" - text generated by the API \u003c/p\u003e\n   \u003c/p\u003e\n\n\n2. [Object Detection](https://github.com/jjmachan/objectdetection-aye) - This generates the bounding boxes for a range of objects\n   and is based on the [Single Shot MuiltiBox\n   Detector](https://arxiv.org/abs/1512.02325). This takes an image and returns the objects that are \n   in the image and their bounding boxes. \n   \n   \u003cp align=\"center\"\u003e\n    \u003cimg width=\"684\" height=\"465\" src=\"https://github.com/jjmachan/aeye-demo/blob/master/demo/livingroom_objects_marked.jpg\"\u003e\n    \u003cp align=\"center\"\u003e The objects in the living room. As you can see there is some small issues with this. \u003c/p\u003e\n   \u003c/p\u003e\n\n\n3. [Face Detection and Recognition](https://github.com/jjmachan/facedetection-aeye) - We can both detect and recognise the people in photos using this API. Face detection is implemented using the [Multitask Cascaded Convolutional Network\n(MTCNN)](https://kpzhang93.github.io/MTCNN_face_detection_alignment/paper/spl.pdf) which gives very reliable results.\n\n   \u003cp align=\"center\"\u003e\n    \u003cimg width=\"700\" height=\"400\" src=\"https://github.com/jjmachan/aeye-demo/blob/master/demo/people_faces_marked.jpg\"\u003e\n    \u003cp align=\"center\"\u003e This has accuratly marked the different faces with very little compute. \u003c/p\u003e\n   \u003c/p\u003e\n   \n   We can also use this to recognise people. This is basically just comparing the faces that are extracted by the detection API and\n   running and running it through a pretrained InceptionNet to compute the similarities and output the scores. This only works \n   for a few people currently.\n   \u003cp align=\"center\"\u003e\n    \u003cimg width=\"602\" height=\"820\" src=\"https://github.com/jjmachan/aeye-demo/blob/master/demo/disha_face_identified.jpg\"\u003e\n    \u003cp align=\"center\"\u003e As you can see the API has successfully recognised Disha Patani and labeled her. \u003c/p\u003e\n   \u003c/p\u003e\n   \n   More work on this end is required like recognition based on the user and is planned in the upcomming version.\n## Usage\n\nThe quickest way to use the API is our hosted solution. Its hosted on GCP but using the free tier, so reliablitly is \nan issue. If the project gets some traction we can upgrade to a more stable solution. Head over to `http://35.225.149.37:5000` on your browser to access the Swagger UI of the API. Alternatively if you want to add it \nto you scripts try the cURL or requests method\n\n#### curl\n```\n  $ curl -X POST http://35.225.149.37:5000/detection -F \"image=@path/to/image.jpg\"\n```\n\n#### Requests\n```\nimport requests\nurl = 'http://35.225.149.37:5000/detection'\ndata = open('path/to/image.jpg', 'rb')\nr = requests.post(url, data=data)\nprint(r.json())\n```\nBut if the service is down please feel free to run it locally. This was build using `PyTorch=1.5` and `bentoml=0.7.7`\n([bentoml](docs.bentoml.org) is used to create the server and serve the trained models). To setup locally follow the steps.\n\n1. Clone the repo to you machine\n\n2. Install all the dependencies\n```\n$ pip install -r requirements.txt\n```\n\n3. Download the pretrained weights and keep it in the `artifacts` folder.\n\n    - [Image Captioning](https://storage.cloud.google.com/aeye-artifacts/imgcap_checkpoint.pth.tar?authuser=1)\n    - [Image Captioning WordMap](https://storage.cloud.google.com/aeye-artifacts/imgcap_wordmap.json?authuser=1)\n    - [Object Detection](https://storage.cloud.google.com/aeye-artifacts/objdet_checkpoint.pth.tar?authuser=1)\n    - [Face Recognition](https://storage.cloud.google.com/aeye-artifacts/vggface2.pt?authuser=1)\n    - [Face Recogniion db](https://storage.cloud.google.com/aeye-artifacts/faces_db.fdb?authuser=1)\n\n4. Run \n```\n$ python saveToBento.py\n```\nThis will pack the dependencies and models into a docker container which is now ready to be hosted on any system.\n\n5. Serve using bentoML.\n```\n$ bentoml serve AeyeService:latest\n```\nThis will launch the server and the service is running on your system.\n\n## Contributing\n\nContributions would be awesome!\nThis is just an experiment so there is a lot of places were improvements are actually required. If you like the idea \nand would like to contribute sent a mail to jamesjithin97@gmail.com.\n## Sponsorship\n\nWe would like to thank [Future Technologies\nLab](https://futuretechnologieslab.com/) under\n[KSUM](https://startupmission.kerala.gov.in/) for providing us with the GPU to\nbuild the prototype. \n\nI know this sounds cliche but without your support none of this will would have been possible.\n\n\n\n--------\n\n\u003cp\u003e\u003csmall\u003eProject based on the \u003ca target=\"_blank\" href=\"https://drivendata.github.io/cookiecutter-data-science/\"\u003ecookiecutter data science project template\u003c/a\u003e. #cookiecutterdatascience\u003c/small\u003e\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjjmachan%2Faeye","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjjmachan%2Faeye","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjjmachan%2Faeye/lists"}