{"id":17025913,"url":"https://github.com/spite/FaceMeshFaceGeometry","last_synced_at":"2026-04-13T00:30:19.445Z","repository":{"id":44426462,"uuid":"264466203","full_name":"spite/FaceMeshFaceGeometry","owner":"spite","description":"FaceMeshFaceGeometry for FaceMesh","archived":false,"fork":false,"pushed_at":"2023-01-12T13:52:42.000Z","size":5628,"stargazers_count":401,"open_issues_count":5,"forks_count":63,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-10-04T10:14:23.794Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spite.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-16T15:24:26.000Z","updated_at":"2024-09-18T04:32:34.000Z","dependencies_parsed_at":"2023-02-09T12:15:16.191Z","dependency_job_id":null,"html_url":"https://github.com/spite/FaceMeshFaceGeometry","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spite%2FFaceMeshFaceGeometry","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spite%2FFaceMeshFaceGeometry/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spite%2FFaceMeshFaceGeometry/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spite%2FFaceMeshFaceGeometry/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spite","download_url":"https://codeload.github.com/spite/FaceMeshFaceGeometry/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219847890,"owners_count":16556345,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T07:30:12.506Z","updated_at":"2026-04-13T00:30:19.331Z","avatar_url":"https://github.com/spite.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# FaceMeshFaceGeometry\n\nThree.js helper for FaceMesh https://github.com/tensorflow/tfjs-models/tree/master/facemesh\n\n---\n\nDemo with textured mask: https://spite.github.io/FaceMeshFaceGeometry/examples/mask/index.html\n\nDemo with remapped video: https://spite.github.io/FaceMeshFaceGeometry/examples/video/index.html\n\nDemo with instanced geometry: https://spite.github.io/FaceMeshFaceGeometry/examples/instanced/index.html\n\nDemo of texture mapping from an image: https://spite.github.io/FaceMeshFaceGeometry/examples/face_transfer/index.html\n\nDemo of face switching: https://spite.github.io/FaceMeshFaceGeometry/examples/face_off/index.html\n\n![FaceMeshFaceGeometry](uvmap.png)\n\n## How to use\n\nAfter following the code in the FaceMesh repo, you end up with a face estimation.\n\nFirst import the class.\n\n`import { FaceMeshFaceGeometry } from './face.js';`\n\nCreate a new geometry helper.\n\n`const faceGeometry = new FaceMeshFaceGeometry();`\n\nOn the update loop, after the model returns some faces:\n\n```faceGeometry.setSize(videoWidth, videoHeight);\nconst faces = await model.estimateFaces(video);\nif(faces.length) {\n  faceGeometry.update(faces[0]);\n}\n```\n\nYou have to call `FaceMeshFaceGeometry::setSize` with the width and height of the video to normalise the coordinates so they align with the video source.\n\nThat's all there is. You can use `faceGeometry` as any other `BufferGeometry`:\n\n```const material = new MeshNormalMaterial();\nconst mask = new Mesh(faceGeometry, material);\nscene.add(mask);\n```\n\n## Mirrored video\n\nThis works for the video as it comes. If your source is a webcam, you might want to flip it horizontally.\n\nIn that case -besides flipping the video element, by using `transform: scaleX(-1)`, for instance- you'll have to pass a flag to `estimateFaces` and `update`:\n\n```faceGeometry.setSize(videoWidth, videoHeight);\nconst faces = await model.estimateFaces(video, false, true);\nif(faces.length) {\n  faceGeometry.update(faces[0], true);\n}\n```\n\n## How to use the video/input as a texture for the face\n\nYou can use the input to the FaceMesh model to texture the 3d mesh of the face. Constuct the helper with:\n\n`const faceGeometry = new FaceMeshFaceGeometry({useVideoTexture: true});`\n\nThat will remap the UV coordinates of the geometry to fit the input. The vertex coordinates from the estimation will be projected every frame into the UV space. That means that the UV coordinates for the shader won't work for texture mapping (i.e,, alpha mask, ao map, etc. will be mapped differently and probably wrong).\n\nThere seem to be issues with instanced video in macOs Chrome and Safari.\n\n## How to use as a 3d mesh, independently of the camera\n\nThe range of the vertices is based on the resolution of the input feed, so it changes depending on the chosen video or image input. Construct the FaceGeometry helper with the option normalizeCoords set to true and the mesh will be in a resonable consistent size.\n\n`const faceGeometry = new FaceMeshFaceGeometry({normalizeCoords: true});`\n\n## How to update my threejs camera\n\nFaceMesh data works better with an Orthographic camera. First create a camera:\n\n`const camera = new OrthographicCamera(1, 1, 1, 1, -1000, 1000);`\n\nand then when the video is ready, or the video dimensions change (width, height), run:\n\n```\ncamera.left = -.5 * width;\ncamera.right = .5 * width;\ncamera.top = .5 * height;\ncamera.bottom = -.5 * height;\ncamera.updateProjectionMatrix();\n```\n\n## Track points in geometry\n\nAfter `faceGeometry.update()` you can use `faceGeometry.track()` to place objects relative to the surface of the face.\n\n```const track = faceGeometry.track(5, 45, 275);\ndummy.position.copy(track.position);\ndummy.rotation.setFromRotationMatrix(track.rotation);\n```\n\nIt will calculate a triangle defined by the three provided vertex ids, and return a `position`, a `normal`, and an orthogonal basis `rotation` that cane be used to rotate and object along the correct normal of that triangle.\n\nUse [this image](https://user-images.githubusercontent.com/7452527/53465316-4a282000-3a02-11e9-8e85-0006e3100da0.png) as a reference for vertex Ids.\n\n## API\n\n`FaceMeshFaceGeometry::update(face: FaceEstimation, cameraIsFlipped: boolean): void`\n\nUpdates the vertices and recalculates normals.\n\n`FaceMeshFaceGeometry::setSize(width: number, height: number): void`\n\nSets the internal values to reframe the coordinates.\n\n`FaceMeshFaceGeometry::track(id0: number, id1: number, id2: number): { position: Vector3, normal: Vector3, rotation: Matrix4 }`\n\nCalculates a triangle defined by vertices id0, id1 and id2, return its center, normal and orthogonal basis.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspite%2FFaceMeshFaceGeometry","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspite%2FFaceMeshFaceGeometry","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspite%2FFaceMeshFaceGeometry/lists"}