{"id":16509669,"url":"https://github.com/joonb14/gazel","last_synced_at":"2025-08-16T12:20:50.542Z","repository":{"id":111832727,"uuid":"286186601","full_name":"joonb14/GAZEL","owner":"joonb14","description":"Gaze Estimation Framework with Android Firebase","archived":false,"fork":false,"pushed_at":"2021-11-08T13:33:07.000Z","size":401191,"stargazers_count":26,"open_issues_count":2,"forks_count":7,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-01T11:04:01.231Z","etag":null,"topics":["android","android-application","calibration","eye-detection","eye-tracking","firebase","firebase-custom-model","firebase-mlkit","firebase-mlkit-vision","gaze-estimation","gaze-tracking","gazel","gazel-framework","mobile","mobile-app","smartphones","tflite-model"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joonb14.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-08-09T07:13:29.000Z","updated_at":"2024-12-23T15:13:03.000Z","dependencies_parsed_at":"2023-06-04T09:02:05.713Z","dependency_job_id":null,"html_url":"https://github.com/joonb14/GAZEL","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joonb14%2FGAZEL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joonb14%2FGAZEL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joonb14%2FGAZEL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joonb14%2FGAZEL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joonb14","download_url":"https://codeload.github.com/joonb14/GAZEL/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238590595,"owners_count":19497350,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["android","android-application","calibration","eye-detection","eye-tracking","firebase","firebase-custom-model","firebase-mlkit","firebase-mlkit-vision","gaze-estimation","gaze-tracking","gazel","gazel-framework","mobile","mobile-app","smartphones","tflite-model"],"created_at":"2024-10-11T15:51:38.550Z","updated_at":"2025-02-13T03:32:54.410Z","avatar_url":"https://github.com/joonb14.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GAZEL\nThis work is official implementation of GAZEL framework which is published in \u003ca href=\"http://www.percom.org/\"\u003ePerCom 2021(GAZEL: Runtime Gaze Tracking for Smartphones)\u003c/a\u003e .\n![overview](https://user-images.githubusercontent.com/30307587/109145148-71f31f80-77a5-11eb-94a7-34a47a1f19d5.png)\nThis work is heavily based on Google's Firebase ML kit Sample(2020, June Version)\u003cbr\u003e\nhttps://github.com/googlesamples/mlkit/tree/master/android/vision-quickstart \u003cbr\u003e\ninspired by: \u003ca href=\"https://gazecapture.csail.mit.edu/\"\u003eEye Tracking for Everyone(2016 CVPR)\u003c/a\u003e\u003cbr\u003e\u003cbr\u003e\nCollaborators: \u003cbr\u003e\n\u003ca href=\"https://github.com/oleeyoung520?tab=repositories\"\u003eoleeyoung520\u003c/a\u003e Email: 2015147520@yonsei.ac.kr \u003cbr\u003e\n\u003ca href=\"https://github.com/Yeeun55\"\u003eYeeun55\u003c/a\u003e Email: joyce9559@naver.com \u003cbr\u003e\n\u003ca href=\"https://github.com/yeokyeong46\"\u003eyeokyeong46\u003c/a\u003e Email: yeokyeong46@gmail.com \u003cbr\u003e\n\n## Sample Videos\nThis work is based on Galaxy Tab S6. \u003cbr\u003e\nWe trained model with data collected using  \u003ca href=\"https://github.com/joonb14/MLKitGazeDataCollectingButton.git\"\u003e MLKitGazeDataCollectingButton\u003c/a\u003e\u003cbr\u003e\nWe also provide a method to utilize our Tablet model to Smartphones by calibration.\u003cbr\u003e\nRed dot represent \"Raw Gaze Estimation Output\"\u003cbr\u003e\nBlue dot represent \"Moving Averaged Output\" (if Calibration is done, moving average on calibrated output, else on raw output)\u003cbr\u003e\nGreen dot represent \"Calibrated Raw Output\"\u003cbr\u003e\nTrained with Galaxy Tab S6 data ,tested on Galaxy Tab S6\u003cbr\u003e\n[![TabS6](https://img.youtube.com/vi/rhO5kjTn0Ts/0.jpg)](https://www.youtube.com/watch?v=rhO5kjTn0Ts)\u003cbr\u003e\nTrained with Galaxy Tab S6 data, tested and calibrated on Galaxy S9+\u003cbr\u003e\n[![S9+](https://img.youtube.com/vi/iwPjHlGeRpw/0.jpg)](https://youtu.be/iwPjHlGeRpw)\u003cbr\u003e\n### Summary\nI mainly changed \u003cb\u003eFaceDetectorProcessor.java, LivePreviewActivity.java\u003c/b\u003e and \u003cb\u003eFaceGraphic.java\u003c/b\u003e \u003cbr\u003e\nAlso deleted most of the source code that is not needed\u003cbr\u003e\nAdded a guide to load and use custom TensorFlow Lite model which is used for Gaze Estimation\u003cbr\u003e\n\n## Details\n#### Gaze Estimation Model\n\u003cimg src=\"https://user-images.githubusercontent.com/30307587/109145286-a6ff7200-77a5-11eb-86ff-41925981af10.png\" alt=\"model\" style=\"width:800px;\"/\u003e\n\nModel should be stored in asset folder. I recommend to create model with Keras, then converted it to the TFlite model.\u003cbr\u003e\nYou can check the output also on Logcat. \u003cb\u003eTAG is \"MOBED_GazePoint\"\u003c/b\u003e\u003cbr\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/30307587/109152350-b8994780-77ae-11eb-882b-0ef26e723d86.png\" alt=\"logcat\"/\u003e\n\n#### Before we start...\nGAZEL uses \u003cb\u003ePersonalized model\u003c/b\u003e. This model is heavily dependent on my facial appearances (Wearing glasses \u0026 in Lab environment). So would not work well on other person.(Have already tested...)\u003cbr\u003e\nSo, I decided to exclude the .tflite model and provide training source codes and data collecting application for you to follow.\u003cbr\u003e\nThe Keras model training \u0026 TensorFlow Lite conversion Code is provided in \u003ca href=\"https://github.com/joonb14/MLKitGazeDataCollectingButton.git\"\u003e MLKitGazeDataCollectingButton\u003c/a\u003e.\u003cbr\u003e\n\n#### For the Calibration\nWe used  5 points calibration with translation, and rescaling.\u003cbr\u003e\n5 points are TopLeft, TopRight, BottomLeft, BottomRight, and Center\u003cbr\u003e\nWe also tried to provide SVR calibration. However, multi output SVR doesn't exist in android. So we are using 2 regressors(with android \u003ca href=\"https://github.com/yctung/AndroidLibSVM\"\u003elibsvm\u003c/a\u003e) for each x and y coordinate (The calibration experiments on the paper are conducted on the \"server\" with scikit-learn, not on the \"smartphones\"). This does not work as well as the linear calibration, so we recommend to use linear calibration.\u003cbr\u003e\n\n\n## TFLite Configuration\u003ca id=\"tflite_config\"\u003e\u003c/a\u003e\nIf you want to use custom TFLite model with our GAZEL Framework. First check  configuration options below(in \u003cb\u003eFaceDetectorProcessor.java\u003c/b\u003e ). We provide Face bitmap, Left/Right Eye Grids, Face Grid.\nWe used 1-channel bitmap for enhancing gaze estimation accuracy, but like other papers which use 3-channel RGB images as input, we provide 3-channel image mode. You can change the mode with THREE-CHANNEL flag. We also provide various options for you to test your model with various model inputs, so try to create gaze estimation model with various inputs!\n\n```java\nprivate final boolean USE_EULER = true; // true: use euler x,y,z as input\nprivate final boolean USE_FACE = false; // true: use face x,y,z as input\nprivate final boolean USE_EYEGRID = false; // true: use eye_grid as input\nprivate final boolean USE_FACEGRID = true; // true: use face_grid as input\nprivate final boolean THREE_CHANNEL = false; // false for Black and White image, true for RGB image\nprivate final boolean calibration_mode_SVR = false; // false for translation \u0026 rescale. true for SVR\nprivate final boolean CORNER_CALIBRATION = false; // false for translation \u0026 rescale with center, true for only 4 corners\n```\nAbove configuration flags are about  switching modes, now below configuration values are specific values for initializing modes.\n```java\nprivate final double SACCADE_THRESHOLD = 300; // distance for classifying FIXATION and SACCADE\nprivate final int resolution = 64; // for eye and face\nprivate final int grid_size = 50; // for eye_grids\nprivate final int face_grid_size = 25; // for face_grid\nprivate final int FPS = 30; // for calibration count\nprivate final int SKIP_FRAME = 10; // for calibration count\nprivate final int COST = 40; // for SVR\nprivate final int GAMMA = 1; // for SVR\nprivate final int QUEUE_SIZE = 20; // for moving average\nprivate final float EYE_OPEN_PROB = 0.0f; //empirical value\n```\n\nIn case you put your TFLite model in the \u003cb\u003e\"GAZEL/GazeTracker/app/src/main/assets/custom_models/eval/\"\u003c/b\u003e directory, you must change the below line in \u003cb\u003eLivePreviewActivity.java\u003c/b\u003e, change\n\n```java\nInputStream inputStream = getAssets().open(\"custom_models/eval/[your_model_name]].tflite\");\n```\n\nthen follow the [issues](#issues)\n#### Issues\nTensorFlow Lite Conversion. Before you load your tflite model, you must check the input details to make sure input order is correct.\u003cbr\u003e\nIn case you are using python interpreter,\n\n```python\nimport tensorflow as tf\ntflite = tf.lite.Interpreter(model_path=\"path/to/model.tflite\")\ntflite.get_input_details()\n```\nexample output will be\n\n```\n[{'name': 'left_eye',\n  'index': 4,\n  'shape': array([ 1, 64, 64,  1], dtype=int32),\n  'dtype': numpy.float32,\n  'quantization': (0.0, 0)},\n {'name': 'right_eye',\n  'index': 56,\n  'shape': array([ 1, 64, 64,  1], dtype=int32),\n  'dtype': numpy.float32,\n  'quantization': (0.0, 0)},\n {'name': 'euler',\n  'index': 1,\n  'shape': array([1, 1, 1, 3], dtype=int32),\n  'dtype': numpy.float32,\n  'quantization': (0.0, 0)},\n {'name': 'facepos',\n  'index': 3,\n  'shape': array([1, 1, 1, 2], dtype=int32),\n  'dtype': numpy.float32,\n  'quantization': (0.0, 0)},\n {'name': 'face_grid',\n  'index': 2,\n  'shape': array([ 1, 25, 25,  1], dtype=int32),\n  'dtype': numpy.float32,\n  'quantization': (0.0, 0)}]\n```\nThen reorder your inputs in \u003cb\u003eFaceDetectorProcessor.java\u003c/b\u003e \u003ca id=\"issues\"\u003e\u003c/a\u003e\n```java\ninputs = new float[][][][][]{left_4d, right_4d, euler, facepos, face_grid}; // make sure the order is correct\n```\n## Custom Device Configuration\nThis work is based on Tablet devices. So if you want to use this framework on Smartphones, you need to follow some instructions.\u003cbr\u003e\n\n* First, you need Tablet device for training base Gaze Estimation CNN Model.\n* Second, you need to collect \u003cb\u003e\"Ground Truth Gaze Data\"\u003c/b\u003e  with \u003ca href=\"https://github.com/joonb14/MLKitGazeDataCollectingButton.git\"\u003e MLKitGazeDataCollectingButton\u003c/a\u003e.\n* Third, you need to train your Gaze Estimation CNN Model with \u003ca href=\"https://github.com/joonb14/MLKitGazeDataCollectingButton/blob/master/GAZEL.ipynb\"\u003eprovided python code\u003ca/\u003e.\n* Fourth, you need to follow [TFLite Configuration](#tflite_config)\n* Fifth, follow the instructions below\nChange the configuration options below(in \u003cb\u003eFaceDetectorProcessor.java\u003c/b\u003e ) with your Target device spec.\n```java\nprivate final boolean isCustomDevice = true;\n//custom device\nprivate final float customDeviceWidthPixel = 1440.0f;\nprivate final float customDeviceWidthCm = 7.0f;\nprivate final float customDeviceHeightPixel = 2960.0f;\nprivate final float customDeviceHeightCm = 13.8f;\nprivate final float customDeviceCameraXPos = 4.8f; // in cm | at Android coordinate system where use top left corner as (0,0)\nprivate final float customDeviceCameraYPos = -0.3f; // in cm | at Android coordinate system where use top left corner as (0,0)\n//original device\nprivate final float originalDeviceWidthPixel = 1600.0f;\nprivate final float originalDeviceWidthCm = 14.2f;\nprivate final float originalDeviceHeightPixel = 2560.0f;\nprivate final float originalDeviceHeightCm = 22.5f;\nprivate final float originalDeviceCameraXPos = 7.1f; // in cm | at Android coordinate system where use top left corner as (0,0)\nprivate final float originalDeviceCameraYPos = -0.5f; // in cm | at Android coordinate system where use top left corner as (0,0)\n```\n\nset the \u003cb\u003eisCustomDevice\u003c/b\u003e flag to true, then change all of the \u003cb\u003ecustomDevice[option]\u003c/b\u003e values\n\n* Lastly, run the GAZEL application, and you must click \u003cb\u003e\"START CALIB\"\u003c/b\u003e button to start calibration and use it as Gaze Tracker.\n\n#### Tips\nCollect data as much as you can before training your model. Recommend you to use different head position with different light conditions. \n\n## Discussion\nIt is well known knowledge to use massive image dataset to train feature extraction layers then only train fully connected layers for targeted environment. I as a mobile system developer however wanted to try using mobile embedded sensors to improve gaze estimation accuracy. That's why I tried to build up new application using collected sensor outputs as well as front camera frames. (And that's why I didn't use the massive image dataset for training).\nAlso I believe our work is the first open source smartphone gaze tracking framework. I hope this little proof of concept framework would help your research. Thank you.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoonb14%2Fgazel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoonb14%2Fgazel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoonb14%2Fgazel/lists"}