{"id":39510152,"url":"https://github.com/rcdilorenzo/inspector-lestrade","last_synced_at":"2026-01-18T06:01:22.765Z","repository":{"id":146270183,"uuid":"147511977","full_name":"rcdilorenzo/inspector-lestrade","owner":"rcdilorenzo","description":"Neural network to predict eye gaze location from front-facing camera","archived":false,"fork":false,"pushed_at":"2018-10-21T02:04:51.000Z","size":38574,"stargazers_count":17,"open_issues_count":0,"forks_count":2,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-04-15T22:26:26.308Z","etag":null,"topics":["deep-learning","exploratory-data-analysis","facial-landmarks","open-images","regis-university","yolov3"],"latest_commit_sha":null,"homepage":"https://github.com/CSAILVision/GazeCapture","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rcdilorenzo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-09-05T12:10:00.000Z","updated_at":"2023-09-27T16:01:12.000Z","dependencies_parsed_at":null,"dependency_job_id":"2d44a403-4646-4518-b446-6f6b73bddcd5","html_url":"https://github.com/rcdilorenzo/inspector-lestrade","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/rcdilorenzo/inspector-lestrade","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rcdilorenzo%2Finspector-lestrade","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rcdilorenzo%2Finspector-lestrade/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rcdilorenzo%2Finspector-lestrade/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rcdilorenzo%2Finspector-lestrade/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rcdilorenzo","download_url":"https://codeload.github.com/rcdilorenzo/inspector-lestrade/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rcdilorenzo%2Finspector-lestrade/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28531991,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T00:39:45.795Z","status":"online","status_checked_at":"2026-01-18T02:00:07.578Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","exploratory-data-analysis","facial-landmarks","open-images","regis-university","yolov3"],"created_at":"2026-01-18T06:01:22.161Z","updated_at":"2026-01-18T06:01:22.757Z","avatar_url":"https://github.com/rcdilorenzo.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Inspector Lestrade\nA Deep Learning project for predicting gaze location from a front-facing iOS camera\n\n\u003ca href=\"discussion/InspectorLestrade-RChristianDiLorenzo-2018Oct.pdf\"\u003e\u003cimg src=\"discussion/paper-thumbnail.png\" width=\"500px\" /\u003e\u003c/a\u003e\n\n\u003ca href=\"discussion/InspectorLestrade-RChristianDiLorenzo-2018Oct.pdf\"\u003eInspectorLestrade: An 8-Week Foray into Predicting Eye Gaze\u003c/a\u003e\n\n**What's with the name?**\n\n\u003e Inspector Lestrade: “I find it hard enough to tackle facts, Holmes, without flying away after theories and fancies.\"\n\u003e\n\u003e \"You are right,\" said Holmes demurely: \"you do find it very hard to tackle the facts.”\n\u003e\n\u003e ― Arthur Conan Doyle\n\nHe's always confident, but not always right.\n\n## Table of Contents\n\n- [Background](#background)\n- [Writings](#writings)\n  - [Week 1: GazeCapture download, DataCamp tutorials, custom computer, and prep work](writings/week-01-dated-09-01-2018.md)\n  - [Week 2: Exploratory data analysis w/ cool visualizations, quick blog article, and reading](writings/week-02-dated-09-08-2018.md)\n  - [Week 3: 3D Facial landmarks, Open Images v4, Yolo v3, and 2TB RAID upgrade](writings/week-03-dated-09-15-2018.md)\n  - [Visualization Quick Tip: Relative Heatmaps, Medium.com](https://medium.com/@rcdilorenzo/visualization-quick-tip-relative-heatmaps-86a52a0c5a0c)\n  - [Week 4: 🎥 Face Alignment, Yolo v3 Training, and 2nd 1080 Ti](https://youtu.be/QvKLHZyOniE)\n  - [Week 5: A Running Multi-Input DNN, Tensorflow Image Preprocess, TIL Yolo](writings/week-05-dated-09-29-2018.md)\n  - [Week 6: 📝 Neural Network Architecture and Loss Functions](https://rcd.ai/eye-tracking-nn-and-loss/)\n  - [Week 7: 📝 Activation Functions](https://rcd.ai/eye-tracking-nn-activation-functions/)\n- [Project Discussion](#project-discussion)\n\n## Background\n\nAs a part of my graduate studies at Regis University, I have had the opportunity to begin exploring my own project during my Deep Learning class.\n\n- Dataset Source: https://github.com/CSAILVision/GazeCapture\n\n- Regis University (M.S. in Data Science): https://www.regis.edu/CCIS/Academics/Degrees-Programs/Graduate-Programs/MS-Data-Science.aspx\n\nNote that in order to use any of the files in this repository for training purposes, the GazeCapture data set will be needed. Visit http://gazecapture.csail.mit.edu/download.php for details on requesting approval.\n\nIf you just want to use the pretrained models and explore the various architecture iterations, just clone the repository and checkout the [results.ipynb notebook](results.ipynb). I can't guarantee that everything will run the first time, but I've made an effort to make the code well-written and easily discoverable through the notebooks throughout the repository. However, I have had limited time so proceed at your own risk! :-)\n\n## Writings\n\nAs I progress through this 8-week class (and this project), I've given weekly updates on my progress or on specific topics I am working through.\n\n- [Week 1: GazeCapture download, DataCamp tutorials, custom computer, and prep work](writings/week-01-dated-09-01-2018.md)\n- [Week 2: Exploratory data analysis w/ cool visualizations, quick blog article, and reading](writings/week-02-dated-09-08-2018.md)\n- [Week 3: 3D Facial landmarks, Open Images v4, Yolo v3, and 2TB RAID upgrade](writings/week-03-dated-09-15-2018.md)\n- [Visualization Quick Tip: Relative Heatmaps, Medium.com](https://medium.com/@rcdilorenzo/visualization-quick-tip-relative-heatmaps-86a52a0c5a0c)\n- [Week 4: 🎥 Face Alignment, Yolo v3 Training, and 2nd 1080 Ti](https://youtu.be/QvKLHZyOniE)\n- [Week 5: A Running Multi-Input DNN, Tensorflow Image Preprocess, TIL Yolo](writings/week-05-dated-09-29-2018.md)\n- [Week 6: 📝 Neural Network Architecture and Loss Functions](https://rcd.ai/eye-tracking-nn-and-loss/)\n- [Week 7: 📝 Activation Functions](https://rcd.ai/eye-tracking-nn-activation-functions/)\n\n## Project Discussion\n\n\u003ca href=\"discussion/InspectorLestrade-RChristianDiLorenzo-2018Oct.pdf\"\u003e\u003cimg src=\"discussion/paper-thumbnail.png\" width=\"400px\" /\u003e\u003c/a\u003e\n\n\u003ca href=\"discussion/InspectorLestrade-RChristianDiLorenzo-2018Oct.pdf\"\u003eInspectorLestrade: An 8-Week Foray into Predicting Eye Gaze\u003c/a\u003e\n\n\u003e All of the following material is available in the PDF document and was specifically written for that layout. Check it out [there](discussion/InspectorLestrade-RChristianDiLorenzo-2018Oct.pdf) or continue reading here.\n\n### Introduction\n\nMost applications of eye tracking firmly lie within medicine or user-based research and are dominated by specialized devices with external sensors and wearables. With the advent of computationally feasible Deep Learning, what was previously outside the reach of the everyday user may well be possible now.\n\nFor example, a common measure of eye strength involves saccade latency and accuracy, that is, the ability to jump from one visual focus to another. Currently, both the exercises and testing must be done with specialized equipment. If the predictive power of neural networks were applied to this problem, patients would be able to perform exercises on their own and thereby have the tools to measure and perhaps even overcome eye disabilities.\n\n\u003e \u003cimg src=\"discussion/faces-blurred-grayscale-2.png\" width=\"300px\" /\u003e\n\u003e\n\u003e *FIG. 1. A sample of the +1400 participants with bounding boxes of eyes and faces determined by the built-in, real-time detection system of iOS.*\n\nIn this work, I present my modeling process to predict from a front-facing iOS camera the gaze location of person’s eyes relative to the device’s lens. Using the data set GazeCapture from Krafka et al. (2016) containing almost 2.5M frames, I trained a deep convolutional neural network called InspectorLestrade\u003csup\u003e[1](#footnote1)\u003c/sup\u003e to predict a gaze coordinate relative to the lens as well as the probability that both eyes are within the frame.\n\n#### Background\n\nAs a part of my Deep Learning class for my masters in Data Science, I have been privileged to not only self-determine a project but also have the entire 8-week semester to produce a decent model. With some knowledge of machine learning but essentially no experience with Deep Learning, the current results of this project are very much a work in progress and further effort would be needed before its utility could be realized in any real situation. The detailed limitations will be discussed further in a later section.\n\nDue to previous interest in eye tracking from my work at a software consultancy, I did a bit of searching and stumbled upon the work of Krafka et al. (2016). This data set, named GazeCapture\u003csup\u003e[2](#footnote2)\u003c/sup\u003e, contains data of nearly 2.5 million images of over 1400 participants gathered via crowdsourcing. Because my goal was to understand Deep Learning over the course of this project, I only did a preliminary read of their paper and entirely skipped the section on their model iTracker in order to keep my bias free.\u003csup\u003e[3](#footnote3)\u003c/sup\u003e Throughout the entire process, I wrote weekly updates either directly on the status of the project or individual bits of learning. The writings, pretrained models, and all code are available from the repository: https://github.com/rcdilorenzo/inspector-lestrade.\n\n### Exploratory Data Analysis\n\nBefore considering model architectures, an exploratory data analysis\u003csup\u003e[4](#footnote4)\u003c/sup\u003e (EDA) revealed some important details about GazeCapture. As seen in Fig. 2, a variety of devices were employed through the crowdsourced outlet which produces a good range of screen sizes from the iPhone 4S to the 12.9” iPad Pro. Likewise, Table I shows the maximum extents of the data which will later be important if the model is specialized to iOS devices.\n\n\u003e \u003cimg src=\"discussion/records-by-device-type.png\" width=\"400px\" /\u003e\n\u003e\n\u003e *FIG. 2. Frequency of frames taken by iOS device type.*\n\n\n\u003e |      | dotInfo.XPts | dotInfo.YPts | dotInfo.XCam | dotInfo.YCam |\n\u003e |------|--------------|--------------|--------------|--------------|\n\u003e | mean | 294.22       | 260.50       | 0.12         | -1.85        |\n\u003e | min  | 40.00        | 40.00        | -26.52       | -26.52       |\n\u003e | max  | 1326.00      | 1326.00      | 26.52        | 26.52        |\n\u003e\n\u003e *TABLE I. Extent of dot information. “Pts” represent screen pixels while “Cam” represents centimeters from camera lens (normalized by the lens location on each device type).*\n\nBeyond these basic numbers, a scaled representation as seen in Fig. 3 illustrates the distribution of dots that participants are asked to gaze at while their picture is repeatedly taken from the front-facing camera. This visualization clearly indicates that all of the primary sectors of the device were covered in addition to some uniformly random points across the entire screen.\n\n\u003e \u003cimg src=\"discussion/dot-values-relative.png\" width=\"400px\" /\u003e\n\u003e\n\u003e *FIG. 3. Frequency and distribution of generated dots for each device.*\n\nFor the purposes of this project, however, the most important distribution is the absolute position of the dots in centimeters from the camera lens. Fig. 4 clearly shows not only the outline of multiple device sizes–5.7”, 9.7”, and 12.9”–but also the various device orientations–portrait, upside-down portrait, landscape left, and landscape right. Note how the iPhone is never used in the upside-down orientation as well as the left offset of the camera seen in the mismatched landscape iPhone frequencies.\n\n\u003e \u003cimg src=\"discussion/cam-points-absolute.png\" width=\"450px\" /\u003e\n\u003e\n\u003e *FIG. 4. Frequency and distribution of dots in centimeters\nfrom the camera.*\n\n### Data Preprocessing\n\nBoth before and during the modeling stage, data preprocessing was an essential component. To explore these details, the initial two approaches must be briefly outlined.\n\n#### Approach 1: Object Detection of Eyes\n\nAlthough this approach was eventually dropped due to time constraints, it was certainly sound in concept. Although Krafka et al. (2016) included the real-time iOS predictions of eyes and faces, my initial exploration revealed that these bounding boxes were seriously wanting. Finding the freely available sample\u003csup\u003e[5](#footnote5)\u003c/sup\u003e of Open Images v4 by Krasin et al. (2017), I employed YOLOv3 by Redmon and Farhadi (2018) to more accurately find bounding boxes labeled “Human eye” from Open Images.\n\nTo accomplish this task, however, the entire sample needed to be downloaded (∼560 GB) in order to have available images. From multiple metadata files, image paths had to be cross-referenced with bounding boxes after being filtered by tag name (i.e. “Human eye”).\n\nMost of the data preprocessing\u003csup\u003e[6](#footnote6)\u003c/sup\u003e consisted of transforming the Open Image metadata into the text file format readily accepted by YOLOv3 (Redmon and Farhadi, 2018). For example, each bounding box was listed as a separate row even though it referenced the same image. These had to be grouped and then assigned a numerical class identifier (in this case just 0 for a single class). Below is a sample of the produced text file.\n\n```\nopenimagesv4/train_08/000004f4400f6ec5.jpg 489,509,274,303,0 618,645,216,232,0 675,696,219,229,0\nopenimagesv4/train_08/0000271195f2c007.jpg 424,522,429,486,0 573,650,440,494,0\n...\n```\n\n#### Approach 2: Face Landmarks\n\nThe second approach begins with an external deep neural network that identifies facial landmark features in 3D-space. Using the face-alignment network of Bulat and Tzimiropoulos (2017), points of the jaw, mouth, nose, eyebrows, and eyes are projected into 3D space even when certain portions of the face do not entirely appear in the frame.\n\n\u003e \u003cimg src=\"discussion/landmarks-to-eyes.png\" width=\"450px\" /\u003e\n\u003e\n\u003e *FIG. 5. Transforming landmarks to 128x128 eye images.*\n\u003e\n\u003e *(face-landmarks-modeling-preprocessing.ipynb: https://git.io/fxoSN)*\n\nFrom these facial features, a bit of data preprocessing was performed to produce the necessary inputs for InspectorLestrade–128x128 images of the left and right eyes as well as the scaled landmark points (Fig. 5). Several hurdles had to be overcome to produce a successful slice for every set of facial landmarks.\u003csup\u003e[7](#footnote7)\u003c/sup\u003e For example, sometimes the landmarks fall outside of the image such as when only half of a face is visible. Because InspectorLestrade always expects the same size inputs, it was necessary to create a 1px vertical slice closest to the “invisible” eye and then expand it to the expected 128x128 size. As will be seen later, this became a hurdle for the modeling process.\n\nFor the training process, these eye images needed be sliced either on the fly or beforehand. Due to the longer computational complexity of predicting these landmarks over slicing the images, I opted to run the face-alignment network across all the frames in preparation for training InspectorLestrade. Although the preprocessing of the landmarks took nearly an entire week to save the results in batches, 8 it enabled slicing images on the fly and thereby more concentrated focus on modeling InspectorLestrade.\n\nAdditionally, the raw points from the face-alignment library (Bulat and Tzimiropoulos, 2017) needed to be scaled. Using the distribution of the x, y, and z values, ‘StandardScaler’ from ‘sklearn’ could be used to create smaller values for better neural network performance. This distribution information is then saved for later retrieval during prediction.\n\n\u003e \u003cimg src=\"discussion/inspector-lestrade-overview.jpg\" width=\"500px\" /\u003e\n\u003e\n\u003e *FIG. 6. Visual overview of the inputs and outputs of InspectorLestrade.*\n\nAll of these preprocessing elements are brought together into the ‘InspectorLestradeGenerator’\u003csup\u003e[9](#footnote9)\u003c/sup\u003e that takes the original GazeCapture data frame plus the pregenerated landmarks. In turn, it renders the inputs and outputs for training. The inputs include the three elements seen in Fig. 6 while the outputs include an (x, y) coordinate as well as a boolean output of whether both eyes were visible (roughly corresponding to whether the person is actually looking at the screen at all).\n\n### Modeling\n\n#### Approach 1: Object Detection of Eyes\n\nWith Python and keras chosen as the primary technologies for the class, I employed a keras version\u003csup\u003e[10](#footnote10)\u003c/sup\u003e of the YOLOv3 architecture that provided a decent example of retraining the existing weights for a new object classification problem. In a transfer learning problem such as this first approach, freezing most of the layers for the first phase of training is common. As seen in Fig. 7, I experimented with the learning rate, number of frozen layers, and number of images.\u003csup\u003e[11](#foonote11)\u003c/sup\u003e\n\n\u003e \u003cimg src=\"discussion/yolo-loss-initial.png\" width=\"400px\" /\u003e\n\u003e\n\u003e *FIG. 7. TensorBoard snapshot of first 35 steps of YOLOv3 training with various data sizes, learning rates, and frozen layers.*\n\nFor the initial phase, it proved that the fourth version (named “v4-12k-adam1e3-train10”) proved to be the most successful of the various iterations as it was more stable even than the second full version. Decoding the name of this version, v4 included 12000 images, a learning rate of 0.001, and 10 layers of unfrozen layers (i.e. layers whose weights and biases were updated). Continuing the long training process for v4, the final validation loss ended at 7.847 after almost 24 hours of training on a 1080 Ti (Fig. 8). Although bounding box detection of eyes does not actually predict coordinates, this approached was dropped due to the better performance of the second approach at the time and due to time constraints.\n\n\u003e \u003cimg src=\"discussion/yolo-final-val-loss.png\" width=\"300\" /\u003e\n\u003e\n\u003e *FIG. 8. Final steps of the v4 model after unfreezing all of the layers and allowing the Adam optimizer to shrink the learning rate.*\n\n#### Approach 2: Face Landmarks\n\nAs previously described, the primary model InspectorLestrade needs to predict coordinates based on the landmarks from the face-alignment library as well as the left and right eye images. Although I produced twelve separate architectures over the course of 3 weeks, the sixth iteration ended up with the best results. Through many failures, a little bit of success, and a lot of learning, I produced a decent 30-layer architecture with a trained weight size of 477KB.\n\nFor the train, test, and validation data sets, the original data was filtered by the frames that had detected facial landmarks and then split based on the researchers’ original split. As seen in Fig. 9, the train data set consists of nearly 1.6M frames.\n\n\u003e \u003cimg src=\"discussion/train-val-test-split.png\" width=\"400px\" /\u003e\n\u003e\n\u003e *FIG. 9. Split of train/test/validation by frames.*\n\n##### Basic Intuition\n\nIn this image-based problem, perhaps the most basic intuition is that convolutional layers with pooling operations will be required. Although current research suggests that other techniques such as capsule layers may provide better results, all of the top-performing image-based architectures since 2014 have employed some combination of these two layer types. This pattern of convolutional layers should be symmetrical for the left and right eyes with repeated blocks of Conv2D layers and MaxPooling2D keras layers.\n\nIn a regression problem and boolean problem such as this one, the chosen activation function for each layer in the network is critical for reasonable success. For the coordinate output, the activation function should be able to produce a semi-unconstrained output at least within the range of [−26.52, 26.52] (Table I). For the gaze likelihood output, its function should only produce values of [0, 1] since it represents a numerically-encoded boolean. Backtracking from these function requirements, hidden layers only present for that particular output should be tailored to that range.\n\n##### Loss Function\n\nWith the realization of the “invisible eye” problem that produces an eye with essentially “garbage” image data (stretched from a 1px strip), the network now has three outputs. Whereas the mean-squared error (MSE) of simple Euclidean distance in 2D was previously acceptable to measure accuracy, the third boolean value complicates the loss function. While keras does allow for performing a metric separately on different outputs, it does not seem to support using different metrics for different outputs. If a network had two loss functions, it would be quite difficult if not impossible to take the partial derivative with respect to each simultaneously for the backpropagation phase. Therefore, some combination function is needed that contains a gradient for each variable.\n\nConceptually, the goal is to penalize the distance by a factor that describes the delta of predicted versus actual probability that both eyes are visible. If the network predicts that both eyes are visible but one is actually not, the loss should be described as the Euclidean distance times some factor larger than 1. Additionally, the Euclidean distance should be returned if the probability output is perfectly correct. Assembling these requisites, a satisfying equation combines a delta between 1 and 2 (the factor) and the original loss function.\n\n\u003cimg src=\"discussion/latex-equations.png\" width=\"400px\" /\u003e\n\n\nThis equation is quite easily translated into a tensorflow operation.\n\n```python\ndef loss_func(actual, pred):\n    x_diff = tf.square(pred[:, 0] - actual[:, 0])\n    y_diff = tf.square(pred[:, 1] - actual[:, 1])\n    g_diff = tf.square(pred[:, 2] - actual[:, 2])\n    return K.mean(tf.square(g_diff + 1) * (y_diff + x_diff))\n```\n\n##### Activation Functions\n\nAlthough the sixth iteration of the InspectorLestrade architecture did not include a custom activation (but instead used linear), I did experiment quite a bit\u003csup\u003e[12](#footnote12)\u003c/sup\u003e with custom activation functions especially when the loss values were in the range of [10\u003csup\u003e9\u003c/sup\u003e, 10\u003csup\u003e24\u003c/sup\u003e]. Most of these functions (Fig. 10) are a variation of the sigmoid function in order to allow for a finer output even when the input to the function is more granular.\n\n\u003e \u003cimg src=\"discussion/activation-functions.png\" width=\"600px\" /\u003e\n\u003e\n\u003e *FIG. 10. Possible activation functions that constrain the output of the neural network.*\n\n##### Neural Network Architecture\n\nPutting all of these considerations into practice, the sixth model iteration (Fig. 11) employs three groups of convolutional and pooling operations–two groups of eight 3x3 Conv2D and 3x3-stride MaxPooling2D layers as well as a group of four 2x2 Conv2D and 2x2-stride MaxPooling2D layers. These groups, employed for each eye, were capped off with a BatchNormalization layer before concatenation with the landmark pathway. The landmark path consisted of three layers before being flattened: 16 fully-connected neurons, BatchNormalization, and another 8 fully-connected neurons. In addition to the common pattern of Conv2D followed by MaxPooling2D, the BatchNormalization layers added throughout the network assist in “centering” the values around zero to keep the neurons from wildly flailing with large weights and biases.\n\n\u003e \u003cimg src=\"discussion/v6-lestrade.svg\" width=\"600px\" /\u003e\n\u003e\n\u003e *FIG. 11. InspectorLestrade architecture. Code available from train.py: (https://git.io/fxojD)*\n\nFor the remaining two pathways leading to the coordinate and likelihood values, a combination of diminishingly smaller Dense layers as well as BatchNormalization are employed. The final coordinate layer uses a linear activation function while the likelihood layer applies a sigmoid function to represent the boolean value.\n\n### Results \u0026 Evaluation\n\n#### Approach 1: Object Detection of Eyes\n\nAlthough the validation set of the retrained YOLOv3-\n320 network achieved a loss of 8.318, an easy test on\na simple image kept indicating a significant problem\n(Fig. 12). Because YOLOv3 was designed to distinguish\nbetween multiple classes as well as perform bounding box\ndetection, its loss function is artificially lowered since its\nclass prediction is always right (“Human eye”).\n\n\u003e \u003cimg src=\"discussion/yolo-issue.png\" width=\"400px\" /\u003e\n\u003e\n\u003e *FIG. 12. Sample image of retrained YOLOv3 architecture. The extremely low class confidence (\u003c 0.5%) indicates a major issue. (eye-detection-yolo-test.ipynb: https://git.io/fxKJm)*\n\nDespite this approach being abandoned due to time constraints, I believe that it holds promise. By reworking the data preprocessing step to include faces as a different class, the architecture should be able to properly train. From there, slicing the image as well as including the face and eye locations should give a second neural network plenty of information in order to achieve a good result. If this second network did achieve convergence, it could easily be appended as an extension to the YOLOv3 network to output a coordinate prediction and confidence level.\n\n#### Approach 2: Face Landmarks\n\nTurning attention to the primary approach, InspectorLestrade had three iterations that were turned out to be moderately decent. However, the original loss function is not a good measurement of accuracy since it is not an intuitive measurement and cannot be compared to Krafka et al. (2016). Instead of incorporating likelihood as the loss function originally did, the test function collapses the measurement to only include coordinates where both eyes of the participant were visible and detected by the face-alignment library. The following tensorflow operations achieve this metric.\n\n```python\n# From model.py (https://git.io/fxKTC)\n\ndef scaled_distance_loss(actual, pred):\n    combined = tf.concat([actual, pred], axis=1)\n    masked = tf.boolean_mask(\n        combined,\n        tf.math.equal(combined[:, 2], 1.0)\n    )\n    x_diff = tf.square(masked[:, 0] - masked[:, 3])\n    y_diff = tf.square(masked[:, 1] - masked[:, 4])\n    return K.mean(tf.sqrt(x_diff + y_diff))\n```\n\n\u003e \u003cimg src=\"discussion/lestrade-top-3-models.png\" width=\"350px\" /\u003e\n\u003e\n\u003e *FIG. 13. Top three performing models on the test set.*\n\nThough this mean Euclidean distance is slightly different from the original researchers, it gives the best approximation from the network ensemble chosen. As seen in Fig. 13, the sixth iteration came in at an average of 4.35cm from the actual locations.\u003csup\u003e[13](#footnote13)\u003c/sup\u003e\n\nDespite the test values seen in Fig. 13, much of the training process had values fluctuating from \u003c 100 to over 10\u003csup\u003e24\u003c/sup\u003e. After many iterations, it appears that this unusual behavior was due at least in part to the enormous number of records per data point in TensorBoard. Each point thereby represents only the final batch of 128 in the ∼1.6M frames of the training set.\n\n\u003e \u003cimg src=\"discussion/v6-bounce.png\" width=\"550px\" /\u003e\n\u003e\n\u003e *FIG. 14. Bouncing loss of the sixth iteration. Note that the entire training set is used per epoch (∼3.5 hours/epoch).*\n\n\u003e \u003cimg src=\"discussion/v13-taper.png\" width=\"550px\" /\u003e\n\u003e\n\u003e *FIG. 15. Tapering loss of a partial thirteenth iteration. Note that these values only include 1000 batches per epoch.*\n\nAfter switching just recently to a modification of the sixth iteration using 15 minute segments of only 128K images per epoch, the result of Fig. 15 indicates a more traditional slope of descent.\n\n### Discussion\n\nWith under eight weeks of work on this solo project having almost no prior experience with deep neural networks, I am somewhat satisfied with the results though a higher accuracy would have been much appreciated. That said, the project is certainly in a position where it could quite be continued in order to find a more accurate architecture. For example, I did not experiment much with dropout or L1/L2 normalization layers since I was primarily focused on getting any model to converge. Likewise, experimenting with an activation function that forces an extreme slope at its extremes (i.e. a cubic) could be another path forward.\u003csup\u003e[14](#footnote14)\u003c/sup\u003e\n\nEven with the 4.35cm mean distance, though, the network needs a bit of engineering to even be deployable. The face-alignment library (Bulat and Tzimiropoulos, 2017) uses PyTorch while InspectorLestrade employs keras. Because they both use CUDA for GPU-acceleration but are not made to work together, it is difficult to have them both use the GPU without hogging all of the memory for themselves. To mitigate this problem for evaluation of new images through both networks, I ended up setting PyTorch to use the CPU so that keras could access the GPU.\n\nUnfortunately, time constraints also restricted testing and tuning of the network speed. Though I did perform some exploratory analysis of face-alignment’s speed in FPS, I was focused on accuracy with speed only as a nice to have feature. However, the chosen YOLOv3 weights that I retrained from actually was one of the faster models at an advertized 44 FPS (Redmon and Farhadi, 2018).\n\n#### Acknowledgements\n\nThis 8-week project was made possible courtesy of my Deep Learning professor, Geena Kim, Ph.D., from the Data Science department of Regis University. Thanks also to the researchers of GazeCapture and iTracker for permission to download and explore their data (Krafka et al., 2016).\n\n**References**\n\n```\nBulat, A. and Tzimiropoulos, G., “How far are we from solving the\n  2d \u0026 3d face alignment problem? (and a dataset of 230,000 3d\n  facial landmarks),” in International Conference on Computer\n  Vision (2017).\nKrafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S.,\n  Matusik, W., and Torralba, A., “Eye tracking for everyone,” in\n  2016 IEEE Conference on Computer Vision and Pattern Recog-\n  nition (CVPR) (2016) pp. 2176–2184.\nKrasin, I., Duerig, T., Alldrin, N., Ferrari, V., Abu-El-Haija,\n  S., Kuznetsova, A., Rom, H., Uijlings, J., Popov, S., Ka-\n  mali, S., Malloci, M., Pont-Tuset, J., Veit, A., Belongie,\n  S., Gomes, V., Gupta, A., Sun, C., Chechik, G., Cai,\n  D., Feng, Z., Narayanan, D., and Murphy, K., “Open-\n  images: A public dataset for large-scale multi-label and\n  multi-class image classification.” Dataset available from\n  https://storage.googleapis.com/openimages/web/index.html\n  (2017).\nRedmon, J. and Farhadi, A., “Yolov3: An incremental improve-\n  ment,” arXiv (2018).\n```\n\n**Footnotes**\n\n\u003ca name=\"footnote1\"\u003e\u003csup\u003e1\u003c/sup\u003eThe name “Inspector Lestrade” comes from the Sherlock Holmes series and is employed in this case because the inspector was always confident but not always correct. This describes the result of my project here due to time constraints and having hardly any previous experience with building neural networks.\u003c/a\u003e\n\n\u003ca name=\"footnote2\"\u003e\u003csup\u003e2\u003c/sup\u003eAvailable by permission from http://gazecapture.csail.mit.edu\u003c/a\u003e\n\n\u003ca name=\"footnote3\"\u003e\u003csup\u003e3\u003c/sup\u003eInterestingly, several of my data visualizations ended up coinciding with their diagrams but were not actually based off of them.\u003c/a\u003e\n\n\u003ca name=\"footnote4\"\u003e\u003csup\u003e4\u003c/sup\u003eAll data visualization figures are available from the notebook in the repository: eye-tracking-eda.ipynb (https://git.io/fxoDe)\u003c/a\u003e\n\n\u003ca name=\"footnote5\"\u003e\u003csup\u003e5\u003c/sup\u003ehttps://www.figure-eight.com/dataset/open-images-annotated-with-bounding-boxes/\u003c/a\u003e\n\n\u003ca name=\"footnote6\"\u003e\u003csup\u003e6\u003c/sup\u003eopenimage-eye-detection-yolo-v3.ipynb: https://git.io/fxoMp\u003c/a\u003e\n\n\u003ca name=\"footnote7\"\u003e\u003csup\u003e7\u003c/sup\u003efacial-model.py: https://git.io/fxoS9\u003c/a\u003e\n\n\u003ca name=\"footnote8\"\u003e\u003csup\u003e8\u003c/sup\u003elandmark-generator.py: https://git.io/fxoSG\u003c/a\u003e\n\n\u003ca name=\"footnote9\"\u003e\u003csup\u003e9\u003c/sup\u003egenerator.py: https://git.io/fxoHF\u003c/a\u003e\n\n\u003ca name=\"footnote10\"\u003e\u003csup\u003e10\u003c/sup\u003ehttps://github.com/qqwweee/keras-yolo3\u003c/a\u003e\n\n\u003ca name=\"footnote11\"\u003e\u003csup\u003e11\u003c/sup\u003eAll of these tweaked models are based off of the YOLOv3-320 weights trained on the COCO data set. See https://pjreddie.com/darknet/yolo.\u003c/a\u003e\n\n\u003ca name=\"footnote12\"\u003e\u003csup\u003e12\u003c/sup\u003ehttps://rcd.ai/eye-tracking-nn-activation-functions\u003c/a\u003e\n\n\u003ca name=\"footnote13\"\u003e\u003csup\u003e13\u003c/sup\u003eDetailed results and experimentation playground with various architectures and weights are available in results.ipynb (https://git.io/fxKTI).\u003c/a\u003e\n\n\u003ca name=\"footnote14\"\u003e\u003csup\u003e14\u003c/sup\u003eThis is actually the modification presented in Fig. 15 though the result is much to early to be even anecdotal.\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frcdilorenzo%2Finspector-lestrade","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frcdilorenzo%2Finspector-lestrade","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frcdilorenzo%2Finspector-lestrade/lists"}