OpenGL, Colourspaces, and Linphone<\/strong><\/u><\/span><\/p>\n What is OpenGL? What is Y’CbCr? And why does Linphone need any of this?<\/strong><\/span><\/p>\n In this article, we will focus on the very last part of this pipeline:\u00a0the video renderer.<\/strong><\/p>\n The renderer takes its input from the video\u00a0decoder\u00a0(H.264, VP8, etc.) and paints its output to a graphical buffer, which is the rectangle you see on your screen (the one with the smiling face of your friend).<\/p>\n This is where we encounter Y’CbCr and colourspaces. For technical reasons, the output of a video decoder is a frame in a Y’CbCr colourspace. At this point, and for details on how this colourspace works, I encourage you to read the summary and the Rationale section of\u00a0the\u00a0Wikipedia article on Y’CbCr<\/em><\/u><\/strong><\/a><\/span>.<\/p>\n All that is then left for the renderer to do is to translate the dataframe to RGB colourspace, scale it to match the resolution of the output buffer, and draw the result onto that buffer. To be able to talk to the graphics card,\u00a0you need OpenGL<\/strong>. So there you have it:\u00a0Linphone uses OpenGL to perform the Y’CbCr\u00a0\u2192\u00a0RGB translation for video rendering.<\/strong><\/p>\n The context of my mission around the video renderer was to create evolutions in the existing code to make it compliant with the OpenGL version 4.1 specification. There was existing code that I needed to either evolve or re-write, but\u00a0in any case I supposed I had a rather good understanding of what it was doing…<\/p>\n I found that the existing code was doing the Y’CbCr to RGB conversion in more or less this way:<\/p>\n This is basically a 3×3 matrix, which is applied to each Y’CbCr pixel to obtain an RGB pixel. A 720P resolution image contains 921,600 pixels, which gives rise to a lot of computations, and that’s where OpenGL helps.<\/p>\n In version 2.0 (2004), OpenGL introduced the concept of\u00a0shaders<\/a><\/strong><\/em><\/u>\u00a0<\/span>through the\u00a0OpenGL Shading Language<\/strong><\/em><\/u><\/a><\/span>, a C-like language for expressing computations to be performed by the GPU. Although initially created to compute the light and colour levels when rendering a 3D object, shaders have evolved to perform a variety of specialised functions. Here, a\u00a0diverted\u00a0but clever usage of shaders is applied: there is no real 3D scene, and we simply draw a flat texture on the screen. The texture, instead of being given as an RGB buffer as expected, is a Y’CbCr image, and the shader program is executed on the graphics card’s GPU to transform this Y’CbCr texture into a RGB one. Simple, isn’t it?<\/p>\n But, where does the transformation matrix above come from?<\/p>\n I searched for “yuv2rgb” on the web, as well as for some of the magic numbers used, like “1.164”. Although I was able to find many examples of code (mostly C) that used similar constants to “convert YUV to RGB” (whatever that meant), none explained how they were obtained.<\/p>\n Refusing to give up, I finally landed on the YUV Wikipedia article, which\u2014past the numbers and formulas that didn’t seem to match the magic numbers in my shader\u2014led me to the\u00a0Y’CbCr article<\/em><\/u><\/strong><\/a><\/span>.<\/p>\n There, everything finally started to make sense. I had finally found where my magic numbers were coming from: they are the result of simplifying (in the mathematical sense) the inverse conversion matrix from the ITU-R BT.601 standard, mixed with partial range shifting and scaling!\u00a0Notably, the 1.164 factor mentioned earlier is a rounded version of the 255\/219 scale factor that can be found in the “ITU-R BT.601 conversion<\/a><\/u><\/em><\/strong><\/span>” section of that article.<\/p>\n I know, I know\u2014I don’t understand half of this either\u2014but the point is that my numbers went from\u00a0magic\u00a0to\u00a0scientific. Example code with documentation<\/strong><\/u><\/span><\/p>\n So after all my research and effort, this is the final code that we use today in the OpenGL 4.1 context:<\/p>\n https:\/\/gitlab.linphone.org\/BC\/public\/mediastreamer2\/-\/blob\/master\/src\/Y…<\/a><\/span><\/p>\n I tried to design it to make it easy to follow along when the Wikipedia article is opened next to it, and used explicit naming.<\/p>\n Conclusion<\/strong><\/u><\/span><\/p>\n Offloading specific processing tasks to the GPU is often a good idea to save CPU time and energy. This colour space conversion, in an unoptimised form (plain C code without SIMD assembly instructions), would consume roughly as much as the whole video encoding process running on the main CPU. This shows you how important this optimisation is in the video stream processing pipeline!<\/p>\n![\"\"](\"https:\/\/linphone.org\/sites\/default\/files\/post_news_520_x_330_px_1920_x_1080_px-13.png\")
\n
\nMost programmers and digital artists are used to representing images in the RGB colourspace, which is a grid of pixels with three channels: the red, green, and blue channels (and sometimes an alpha channel for transparency).
\nA Y’CbCr data frame is almost the same, except it encodes colour with three different channels: luma, chroma blue, and chroma red.<\/p>\n
\nHowever, this is a massively parallel process, as it needs to be done for every pixel. If you have a background in computer graphics, you know what I’m getting at: this is a job for your graphics card!<\/p>\n
\nOpenGL (Open Graphics Library) is a cross-platform library and standard that allows a programmer to control a GPU (graphics processing unit) to carry out tasks such as 2D and 3D rendering.
\nIt is well-known in the video game industry, along with DirectX and Metal (equivalent libraries that are not cross-platform), and is now slowly becoming obsolete due to the new Vulkan standard.<\/p>\n![\"\"](\"https:\/\/linphone.org\/sites\/default\/files\/crea_nl_etroit_1200_x_175_px-4.png\")
<\/strong><\/p>\n![\"\"](\"https:\/\/linphone.org\/sites\/default\/files\/2_2.png\")
<\/p>\n
\nI don’t need to understand the physics of phosphor light emission, nor how Kr, Kg, and Kb were chosen. I just need to know that that’s\u00a0how\u00a0they were chosen.<\/p>\n![\"\"](\"https:\/\/linphone.org\/sites\/default\/files\/post_news_520_x_330_px_1920_x_1080_px-12.png\")
<\/p>\n
![\"\"](\"https:\/\/linphone.org\/sites\/default\/files\/2_0.png\")
<\/a><\/p>\n","protected":false},"author":10,"featured_media":0,"parent":0,"template":"","meta":{"_acf_changed":false,"inline_featured_image":false},"actualites-category":[42],"class_list":["post-13446","actualites","type-actualites","status-publish","hentry","actualites-category-tech"],"acf":[],"yoast_head":"\n