No pixelbuffer out output on current mediapipe, transform code still missing

[61315]

No pixelbuffer out output on current mediapipe, transform code still missing

[danhd123]

sure thing, this is a diff from 4a20e9909d55838d5630366ce719844cf06ae85c (tag v0.8.10) on google/mediapipe

[danhd123]

Found part of the problem: in the iris tracking pbtxt graph, there's a typo ConcatenateLandmarListVectorCalculator instead of ConcatenateLandmarkListVectorCalculator (missing k in Landmark) which for some reason doesn't break with an error message, but rather just holds up the whole graph. Unfortunately, fixing the typo just results in an error message that this calculator doesn't exist. trying various permutations result in different error messages (-Vector: the next subgraph needs a NormalizedLandmarkList, +Normalized: a different subgraph needs a vector of NormalizedLandmarkLists, with both: calculator does not exist)

EDIT: never mind, the typo is in mediapipe, which indeed registers a ConcatenateLandmarListVectorCalculator. So that's not it.

[61315]

Let me have a look. In the meanwhile, please share the revision id(a commit hash) of the mediapipe.

[danhd123]

Okay, I've got it displaying, by adding "//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata_landmarks.binarypb", to the data section of the objc_library in the framework BUILD file, but now it's only updating the non-eye face landmarks every 5-10 seconds, or if it loses tracking and has to recalculate the non-eye face landmarks. 🤔 This happens even if I replace the face landmark smoothing one euro filter with a no_filter {}

[61315]

@danhd123 You can checkout the latest commit(32b248f).

Repository now contains the following files:

+new file:   src/ios/facegeometry/BUILD
+new file:   src/ios/facegeometry/MPPBFaceGeometry.h
+new file:   src/ios/facegeometry/MPPBFaceGeometry.mm
+new file:   src/ios/facegeometry/graphs/BUILD
+new file:   src/ios/facegeometry/graphs/face_geometry_with_transform.pbtxt

You may find relevant codes that answers your question below. I just parsed the required Protocol Buffers array by hand.

Hope this helps :)

mediapipe-prebuilt/src/ios/facegeometry/MPPBFaceGeometry.mm

Lines 152 to 158 in 32b248f

	const auto& t = faceGeometry.pose_transform_matrix().packed_data();
	const auto& matrix = simd_matrix(
	(simd_float4){ t[0], t[1], t[2], t[3] },
	(simd_float4){ t[4], t[5], t[6], t[7] },
	(simd_float4){ t[8], t[9], t[10], t[11] },
	(simd_float4){ t[12], t[13], t[14], t[15] }
	);