Deepstream Gaze App not functioning on 640x480 video

[angelo-beyk]

I am currently using deepstream’s gaze estimation application from deepstream_tao_apps:
https://github.com/NVIDIA-AI-IOT/deepstream_tao_apps/tree/master/apps/tao_others/deepstream-gaze-app

I have an issue with small format videos, specifically when I try to run the app on a 640x480 video, the gaze direction estimation is false and it is not stable, it oscillates a lot.

In order to use a small size video of 640x480 and perform the inference on it without rescaling it, I modified lines 65 and 66 in deepstream_gaze_app.cpp in the above repo:

#define MUXER_OUTPUT_WIDTH 640
#define MUXER_OUTPUT_HEIGHT 480

The problem is not present when I run the app with a bigger shape format, on the same 640x480 video:

#define MUXER_OUTPUT_WIDTH 1280
#define MUXER_OUTPUT_HEIGHT 720

I also modified lines 958 and 959 to display the video with the original shape:

  g_object_set (G_OBJECT (nvtile), "rows", tiler_rows, "columns",
      tiler_columns, "width", 640, "height", 480, NULL);

Here is the script that I modified:
deepstream_gaze_app_modified.zip

I noticed that the issue is not with video quality, it is more related to shape format size; the bigger the image size, the better it performs. Even on the same 640x480 video, if we rescale it to 1280x960 (double), the model functions much better.

There is also another potential problem, the facial landmarks are not showing with the rescaled video, but with the original one, they are displayed.

In my application, I need to use an image stream with a small shape format. Is there a way to make this work?

Here are some details about my setup:

• Hardware Platform: Jetson Orin NX
• DeepStream Version: deepstream-6.2
• JetPack Version: 5.1.1 - Jetson Linux 35.3.1
• TensorRT Version: tensorrt-8.5.2.2
• Gstreamer Version: 1.16.3

This first example is with the 640x480 video without rescaling:

This second example is with the same 640x480 video but rescaled to 1280x720: