OpenCV using an Intel iGPU (OpenGL/OpenCL/VAAPI interop) on Linux: Custom Optical Flow Visualizatio

Описание

Original Video: https://www.youtube.com/watch?v=ItGwXRCcisA

The source code (https://github.com/kallaballa/GCV) contains several step-by-step demos that will teach you how to do it yourself!

Performance (with this particular video):
*4k: 15+FPS (20+FPS offscreen)*
*1080p: 50+FPS (65+FPS offscreen)*

Explaination: This rendering of sparse optical flow (https://docs.opencv.org/4.6.0/d4/dee/tutorial_optical_flow.html) was created on a Linux system using OpenCV. The special thing about it is, it was rendered on my Intel Tigerlake CPU/iGPU using OpenCL/OpenGL- and OpenCL/VAAPI-interoperability. That basically means that most of the computation (the part that matters) was done on the iGPU, even video decoding and encoding.
That is possible because i maintain a fork of the Intel Compute Runtime (https://github.com/kallaballa/compute-runtime) to support OpenGL sharing extensions and forked OpenCV 4.x to fix a couple of Bugs/Problems. While it is very unlikely that Intel will directly support OpenGL sharing on Linux in the near future, it looks like the required changes to OpenCV (only 2 of them are necessary to make this work: https://github.com/opencv/opencv/pulls/kallaballa) will make it back into the official OpenCV repository.
While this visualization looks good on many videos, it was made with videos in mind that have a rather stable background (so background and foreground can be distinguished easily). Scene changes are no problem because there is explicit detection for it in place.

I contacted the rights holder of the original video (BBTV) several times to ask them what they think of my fair-use of this video for educational purpose. They never answered.