opengl – Saint's Log

2022-04-18 —Categories: Graphics

OpenGL Programming Gotchas

It has been a while since I wrote graphics/rendering code. The bugs are very different from those I typically write/fix since so I thought I might as well share the types of issues I dealt with. Here are some of the pitfalls I encountered:

Not checking all OpenGL API results. Continuing execution when vertex/fragment shader compilation failed, for example, wastes a ton of time debugging downstream failures.
Assuming successful shader compilation means that you can assign values to every shader uniform you declared! If the uniform is not actually used to generate the shader’s output, the compiler (which I learned lives in the graphics driver) can eliminate the uniform, thereby causing attempts to set it to fail.
Using glVertexAttribPointer instead of glVertexAttribIPointer to pass integer IDs needed by a fragment shader. Wasted so much time on this because I was feeling schedule pressure and didn’t carefully read the documentation. Since I set the normalized parameter to GL_FALSE, the IDs were being converted into floats directly without normalization. TODO: study this behavior now to see exactly which floats ended up in the shader.
Passing a count of 0 to glDrawArrays. The count argument specifies the number of indices to be rendered. Took me a while to figure out why nothing was showing up after some refactoring that I did. Turns out the number of vertices in the class I created was 0. An assertion here would have saved a ton of time.
Mismatched vertex attribute formats. Spiky rendered output instead of a shape like a cube/sphere makes this one rather easy to detect. In my case, I was using 2 structs and one had an extra ID field that ended up being vertex data when the other type of struct was passed to the rendering code.
Passing GL_TEXTURE0 to a sampler2D shader instead of 0! This was a typo that I didn’t catch in course slides.
Mixing up sizedInternalFormat and internalFormat values in calls to glTextureStorage2D and glTextureSubImage2D.

Buffer Texture Issues

Shader Compilation Errors

When storing vertex shader data into a buffer texture, the OpenGL APIs were used to create and bind the buffer: glCreateBuffers, glNamedBufferStorage, glCreateTextures, glTextureBuffer, glBindImageTexture. A snippet of the vertex shader code to write into the buffer is shown below. Unfortunately, the shader compilation failed with this error: 0(69) : error C1115: unable to find compatible overloaded function “imageStore(struct image1D4x32_bindless, int, struct DebugData)”.

layout(binding=0, rgba32f) uniform image1D bufferTexture;
...
struct DebugData
{
    int vertexId;
    vec4 vPosition;
    vec4 modelViewVertexPosition;
    vec4 glPosition;
    vec3 vNormal;
    vec3 transformedNormal;
};
...
DebugData debugData;
debugData.vertexId = faceId;
debugData.vPosition = position;
debugData.modelViewVertexPosition = vEyeSpacePosition;
debugData.glPosition = gl_Position;
debugData.vNormal = vNormal;
debugData.transformedNormal = vEyeSpaceNormal;

imageStore(bufferTexture, gl_VertexID, debugData);

I needed up having to define a const int numVec4sPerStruct = 6; and call imageStore for each member of the struct, e.g. imageStore(bufferTexture, gl_VertexID * numVec4sPerStruct + 1, debugData.vPosition);. See related discussions here and here.

Invalid Shader Data in Buffers

There were all 0s in the output written into the texture/shader storage buffers. I tried using imageBuffer instead of image1D to avoid getting all zeros when reading the texture image buffer. The code looked correct but I couldn’t explain why zeros were being read back despite the rendered output looking correct. To figure out why this could be happening, I initialized the texture memory to a known value (integer value -1, which turns out to be a NaN when interpreted as a float). This made it easier to explain the random crap that was being displayed (hint from stack overflow) since it made it obvious that many memory locations were not being written to. Here is a snippet of the fragment shader:

struct FragmentData
{
    uint uniqueFragmentId;
    vec2 texCoords;
    vec4 glFragCoord;
    uint faceId;
};

layout(std430, binding=1) buffer DebugFragmentData
{
    FragmentData fragmentData[];
} outputFragmentData;

The fragment shader’s main method wrote to the shader storage like this:

outputFragmentData.fragmentData[uniqueFragmentId].glFragCoord = gl_FragCoord;

Initializing the storage to all 0xFF bytes made it possible to conclude that the wrong data was being written to the host, more specifically that the wrong locations were being written to! Who knew structs and alignment were a thing (TODO: link to alignment discussion in GLSL spec)! The host needed to interpret the shader storage using this struct:

struct FragmentData
{
    unsigned int uniqueFragmentId;
    unsigned int fourBytesForAlignment1;
    TextureCoord2f texCoords;
    VertexCoord4f glFragCoord;
    unsigned int faceId;
    unsigned int fourBytesForAlignment2;
    unsigned int fourBytesForAlignment3;
    unsigned int fourBytesForAlignment4;
};

Also see discussion in GLSL spec about std430 vs std140 for shader block storage!

C++ Bugs

Some of the bugs I introduced were also plain old C++ bugs (not OpenGL specific), e.g.

Uninitialized variables (the fovy float had a NaN value).
Copy/pasting code and missing a key fact that both Gouraud and Phong shading code paths were calling the Gouraud shader (even though the scene state output in the console showed the state had been correctly updated to Phong). That’s what you get for copy pasting and not having tests…
Wrong array indexing logic. In the bug below (commit 3cfe07aea6914a91), I was multiplying the indices by elementSize but that is wrong because lines 2-5 from the bottom already have a built in multiplication by the element size. Noticed this from the disassembly.

int VolumeDataset3D::GetIndexFromSliceAndCol(uint32_t slice, uint32_t column)
{
    const int sizeOf1Row = width * sizeof(uint8_t);

    int index = slice * sizeOf1Row + column;

    return index;
}

const int elementSize = sizeof(VertexDataPositionedByte);

    for (uint32_t slice = 0; slice < depth - 1; slice++)
    {
        for (uint32_t col = 0; col < width - 1; col++)
        {
            int index = elementSize * GetIndexFromSliceAndCol(slice, col);
            int rightIndex = elementSize * GetIndexFromSliceAndCol(slice, col + 1);
            int diagIndex = elementSize * GetIndexFromSliceAndCol(slice + 1, col + 1);
            int bottomIndex = elementSize * GetIndexFromSliceAndCol(slice + 1, col);

            VertexDataPositionedByte cellData[4] =
            {
                ((VertexDataPositionedByte*)vertexData2D.data)[bottomIndex],
                ((VertexDataPositionedByte*)vertexData2D.data)[diagIndex],
                ((VertexDataPositionedByte*)vertexData2D.data)[rightIndex],
                ((VertexDataPositionedByte*)vertexData2D.data)[index],
            };

Here is the corresponding disassembly:

            int bottomIndex = elementSize * GetIndexFromSliceAndCol(slice + 1, col);
00007FF7A79AE5FC  mov         eax,dword ptr [rbp+0C4h]  
00007FF7A79AE602  inc         eax  
00007FF7A79AE604  mov         r8d,dword ptr [rbp+0E4h]  
00007FF7A79AE60B  mov         edx,eax  
00007FF7A79AE60D  mov         rcx,qword ptr [this]  
00007FF7A79AE614  call        VolumeDataset3D::GetIndexFromSliceAndCol (07FF7A7990046h)  
00007FF7A79AE619  imul        eax,eax,10h  
00007FF7A79AE61C  mov         dword ptr [rbp+164h],eax  

            VertexDataPositionedByte cellData[4] =
            {
                ((VertexDataPositionedByte*)vertexData2D.data)[bottomIndex],
00007FF7A79AE622  movsxd      rax,dword ptr [rbp+164h]  
00007FF7A79AE629  imul        rax,rax,10h  
00007FF7A79AE62D  mov         rcx,qword ptr [this]  
00007FF7A79AE634  mov         rcx,qword ptr [rcx+0E8h]  
00007FF7A79AE63B  lea         rdx,[rbp+190h]  
00007FF7A79AE642  mov         rdi,rdx  
00007FF7A79AE645  lea         rsi,[rcx+rax]  
00007FF7A79AE649  mov         ecx,10h  
00007FF7A79AE64E  rep movs    byte ptr [rdi],byte ptr [rsi]  
                ((VertexDataPositionedByte*)vertexData2D.data)[diagIndex],
00007FF7A79AE650  movsxd      rax,dword ptr [rbp+144h]  
00007FF7A79AE657  imul        rax,rax,10h  
00007FF7A79AE65B  mov         rcx,qword ptr [this]  
00007FF7A79AE662  mov         rcx,qword ptr [rcx+0E8h]  
00007FF7A79AE669  lea         rdx,[rbp+1A0h]  
00007FF7A79AE670  mov         rdi,rdx  
00007FF7A79AE673  lea         rsi,[rcx+rax]  
00007FF7A79AE677  mov         ecx,10h  
00007FF7A79AE67C  rep movs    byte ptr [rdi],byte ptr [rsi]

Crashes

I ran into (currently unexplained) crashes in both the Intel and nVidia OpenGL drivers (I used Windows only). There were also crashes (on a specific commit) from a nullref/access violation on my HP desktop but not on my Surface Pro. Found out later that the desktop was actually right to crash but the difference in behavior was certainly troubling.