Project 5: Zhan Xiong Chin

README.md

@@ -3,26 +3,78 @@ WebGL Deferred Shading
 
 **University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 5**
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
+* Name: Zhan Xiong Chin
+* Tested on: **Google Chrome (Version 54.0.2840.87 m (64-bit))** on
+  Windows 7 Professional, i7-4700 @ 3.40GHz 3.40GHz 16GB, Quadro K600 1GB (SIG Lab)
 
 ### Live Online
 
-[![](img/thumb.png)](http://TODO.github.io/Project5B-WebGL-Deferred-Shading)
+[![](img/thumb.png)](https://czxcjx.github.io/Project5-WebGL-Deferred-Shading-with-glTF/)
 
-### Demo Video/GIF
+### Demo GIF
 
-[![](img/video.png)](TODO)
+![](img/video.gif)
 
-### (TODO: Your README)
+### Features
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
+* WebGL deferred Blinn-Phong shading
+* Scissor test optimization
+* G-buffer packing
+* Bloom shader using gaussian blur with both 2d kernel and convolved 1d kernels
+* Screen space motion blur
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
+### Deferred shading introduction
+A deferred shader packs information about each fragment to be rendered into g-buffers, then each light loops over the fragments rendered to screen to compute the final color of each pixel. This is an optimization over normal forward rendering, where lighting calculations will have to be done for each geometry fragment, even those which will not be rendered to screen due to the depth test. 
 
+### Scissor test optimization
+Each light only has a maximum radius that it would affect. Thus, a simple screen space AABB was used to cull away pixels outside of the maximum radius, to ensure that each light did not have to loop over every pixel in the screen when doing lighting computations. 
+
+![](img/scissor_aabb.png)
+
+Scissor test bounding boxes
+
+The bounding box computation is slightly inaccurate at the edges of screen space, causing unnatural-looking "edges" on lights near the edge of the screen. To remedy this problem, an option was added to scale the bounding boxes up, trading performance for a better visual effect. A further improvement that might be made beyond the current implementation is to use better AABB calculations, or to compute the scale factor according to how close to the center of the screen the bounding box is.
+
+Adding the scissor test for lights improved performance significantly, enabling scenes with hundreds of lights to be rendered with little degradation to render times. The follow graphs show the performance of the scene for different numbers of lights (radius 4.0) and different sizes of bounding box vs no bounding box used.
+
+![](img/scissor_test.png)
+
+### G-buffer packing
+G-buffers are used to store fragment information that the material shaders subsequently use for lighting calculations. The basic "un-packed" data requires 4 g-buffers, storing the world space position, normals, texture color, and normal map direction for each fragment. To reduce the number of g-buffers needed and thus reduce the amount of memory that needs to be moved around to/from the GPU by the shaders, the following improvements were made, reducing the number of required g-buffers to two:
+
+* The normal map was applied by the fragment shader, rather than by the lighting shader. This removed the need to include the normal map direction as part of a g-buffer.
+* Only the x and y positions of the normals were stored, the final position can be calculated by z = sqrt(1 - x^2 - y^2). These two were stored into the first two components of the second g-buffer.
+* The components of the color vector were converted from floating point values to 8-bit integers, then packed into the last two components of the second g-buffer.
+
+While these optimizations did not make a noticeable difference for the standard 800x600 screen, since each individual g-buffer is not very big, with increased screen resolution and size, the packed g-buffers began to perform better (25 lights):
+
+![](img/gbuffer_test.png)
+
+### Bloom shader
+A bloom shader was used for light glow effects. In this, a filter is applied to detect bright parts of the image. Then, a gaussian blur (using a 5x5 kernel) was applied over these bright parts, and they are added back to the original image. This creates a light glow effect, as shown below.
+
+<img src="img/nobloom.png" width=400 height=300></img>
+<img src="img/bloom_brightness.png" width=400 height=300></img>
+<img src="img/bloom_blur.png" width=400 height=300></img>
+<img src="img/bloom.png" width=400 height=300></img>
+
+From left to right: Original image, brightness filter, Gaussian blurred, Image with glow
+
+This was further optimized by using two separate convolutions (horizontal and vertical) using a 1D Gaussian kernel, rather than using a 2D kernel. However, there was no observable difference in rendering time between the two. Also, the extra overhead of the Bloom shader only became apparent at 1600x1200 screen resolution, and even then it only slowed the rendering time from 40 to 45 ms. This suggests that the main performance bottleneck of the deferred renderer is not in the post-processing steps.
+
+### Screen space motion blur
+
+Screen space motion blur was achieved by copying the computed world space positions of the previous frame for each fragment, and projecting it using the current camera matrix instead. From these, we can obtain the new screen space position of the old fragment, and blur the rendered image between those two points. The debug view for motion vectors is shown below; the R and G values are determined by the X and Y difference between the previous position and current position in screen space.
+
+![](img/motion_vectors.gif)
+
+Debug view for motion vectors, note the slight color changes indicating the motion vectors as the image moves
+
+![](img/motion_blur.gif)
+
+Motion blur
+
+Motion blur causes a similar slowdown in performance as Bloom glow; it does not cause slowdown unless the screen resolution is high, as it is a post-processing step that is implemented in a similar fashion. 
 
 ### Credits
 

glsl/clone.frag.glsl

@@ -0,0 +1,17 @@
+#version 100
+precision highp float;
+precision highp int;
+
+uniform sampler2D u_in;
+uniform sampler2D u_depth;
+
+varying vec2 v_uv;
+
+void main() {
+  float depth = texture2D(u_depth, v_uv).x;
+  if (depth >= 1.0) {
+    gl_FragColor = vec4(0.0, 0.0, 0.0, 0.0);
+  } else {
+    gl_FragColor = texture2D(u_in, v_uv);
+  }
+}

glsl/copy-pack.frag.glsl

@@ -0,0 +1,33 @@
+#version 100
+#extension GL_EXT_draw_buffers: enable
+precision highp float;
+precision highp int;
+
+uniform sampler2D u_colmap;
+uniform sampler2D u_normap;
+
+varying vec3 v_position;
+varying vec3 v_normal;
+varying vec2 v_uv;
+
+vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
+    normap = normap * 2.0 - 1.0;
+    vec3 up = normalize(vec3(0.001, 1, 0.001));
+    vec3 surftan = normalize(cross(geomnor, up));
+    vec3 surfbinor = cross(geomnor, surftan);
+    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
+}
+
+vec2 packVec(vec4 v) {
+  v = floor(v * 255.0);
+  return vec2( v.r * 256.0 + v.g, v.b * 256.0 + v.a );
+}
+
+void main() {
+    gl_FragData[0] = vec4( v_position, 1.0 );
+    vec3 geomnor = normalize(v_normal.xyz);
+    vec3 normap = normalize(texture2D(u_normap, v_uv).xyz);
+    vec3 nor = normalize(applyNormalMap(geomnor, normap));
+    vec2 col = packVec(texture2D(u_colmap, v_uv));
+    gl_FragData[1] = vec4( nor.x, nor.y, col.x, col.y );
+}

glsl/copy.frag.glsl

@@ -11,10 +11,8 @@ varying vec3 v_normal;
 varying vec2 v_uv;
 
 void main() {
-    // TODO: copy values into gl_FragData[0], [1], etc.
-    // You can use the GLSL texture2D function to access the textures using
-    // the UV in v_uv.
-
-    // this gives you the idea
-    // gl_FragData[0] = vec4( v_position, 1.0 );
+    gl_FragData[0] = vec4( v_position, 1.0 );
+    gl_FragData[1] = vec4( v_normal, 0.0 );
+    gl_FragData[2] = texture2D(u_colmap, v_uv);
+    gl_FragData[3] = texture2D(u_normap, v_uv);
 }

glsl/deferred-pack/ambient.frag.glsl

@@ -0,0 +1,27 @@
+
+#version 100
+precision highp float;
+precision highp int;
+
+#define NUM_GBUFFERS 4
+
+uniform sampler2D u_gbufs[NUM_GBUFFERS];
+uniform sampler2D u_depth;
+
+varying vec2 v_uv;
+
+void main() {
+    vec4 gb0 = texture2D(u_gbufs[0], v_uv);
+    vec4 gb1 = texture2D(u_gbufs[1], v_uv);
+    float depth = texture2D(u_depth, v_uv).x;
+    ivec2 packedCol = ivec2(int(gb1.z), int(gb1.w));
+    vec3 colmap = vec3(packedCol.x / 256, packedCol.x - (packedCol.x / 256) * 256, packedCol.y / 256) / 255.0;
+
+
+    if (depth == 1.0) {
+        gl_FragColor = vec4(0, 0, 0, 0); // set alpha to 0
+        return;
+    }
+
+    gl_FragColor = vec4(0.1 * colmap, 1.0);
+}

glsl/deferred-pack/blinnphong-pointlight.frag.glsl

@@ -0,0 +1,50 @@
+#version 100
+precision highp float;
+precision highp int;
+
+#define NUM_GBUFFERS 4
+
+uniform vec3 u_lightCol;
+uniform vec3 u_lightPos;
+uniform vec3 u_cameraPos;
+uniform float u_lightRad;
+uniform sampler2D u_gbufs[NUM_GBUFFERS];
+uniform sampler2D u_depth;
+
+varying vec2 v_uv;
+
+vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
+    normap = normap * 2.0 - 1.0;
+    vec3 up = normalize(vec3(0.001, 1, 0.001));
+    vec3 surftan = normalize(cross(geomnor, up));
+    vec3 surfbinor = cross(geomnor, surftan);
+    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
+}
+
+void main() {
+    vec4 gb0 = texture2D(u_gbufs[0], v_uv);
+    vec4 gb1 = texture2D(u_gbufs[1], v_uv);
+    float depth = texture2D(u_depth, v_uv).x;
+    vec3 pos = gb0.xyz;     // World-space position
+    vec3 nor = vec3(gb1.x, gb1.y, sqrt(1.0 - gb1.x * gb1.x - gb1.y * gb1.y)); // Normal map already applied
+    ivec2 packedCol = ivec2(int(gb1.z), int(gb1.w));
+    vec3 colmap = vec3(packedCol.x / 256, packedCol.x - (packedCol.x / 256) * 256, packedCol.y / 256) / 255.0;
+
+    float dist = distance(pos, u_lightPos);
+
+    // If nothing was rendered to this pixel, set alpha to 0 so that the
+    // postprocessing step can render the sky color.
+    if (depth == 1.0 || u_lightRad < dist) {
+        gl_FragColor = vec4(0, 0, 0, 0);
+        return;
+    }
+
+    vec3 L = normalize(u_lightPos - pos);
+    vec3 V = normalize(- pos);
+    vec3 H = normalize(L + V);
+    float attenuation = max(0.0, u_lightRad - dist);
+    float lambert = max(0.0, dot(L, nor));
+    float specular = max(0.0, dot(H, nor));
+    vec3 col = colmap * u_lightCol * lambert + specular * u_lightCol;
+    gl_FragColor = vec4(col * attenuation, 1.0);
+}

glsl/deferred-pack/debug.frag.glsl

@@ -0,0 +1,48 @@
+#version 100
+precision highp float;
+precision highp int;
+
+#define NUM_GBUFFERS 4
+
+uniform int u_debug;
+uniform sampler2D u_gbufs[NUM_GBUFFERS];
+uniform sampler2D u_depth;
+
+varying vec2 v_uv;
+
+const vec4 SKY_COLOR = vec4(0.66, 0.73, 1.0, 1.0);
+
+vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
+    normap = normap * 2.0 - 1.0;
+    vec3 up = normalize(vec3(0.001, 1, 0.001));
+    vec3 surftan = normalize(cross(geomnor, up));
+    vec3 surfbinor = cross(geomnor, surftan);
+    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
+}
+
+void main() {
+    vec4 gb0 = texture2D(u_gbufs[0], v_uv);
+    vec4 gb1 = texture2D(u_gbufs[1], v_uv);
+    float depth = texture2D(u_depth, v_uv).x;
+    // These definitions are suggested for starting out, but you will probably want to change them.
+    vec3 pos = gb0.xyz;     // World-space position
+    vec3 nor = vec3(gb1.x, gb1.y, sqrt(1.0 - gb1.x * gb1.x - gb1.y * gb1.y)); // Normal map already applied
+    ivec2 packedCol = ivec2(int(gb1.z), int(gb1.w));
+    vec3 colmap = vec3(packedCol.x / 256, packedCol.x - (packedCol.x / 256) * 256, packedCol.y / 256) / 255.0;
+
+    if (u_debug == 0) {
+        gl_FragColor = vec4(vec3(depth), 1.0);
+    } else if (u_debug == 1) {
+        gl_FragColor = vec4(abs(pos) * 0.1, 1.0);
+    } else if (u_debug == 2) {
+        gl_FragColor = vec4(nor, 1.0);
+    } else if (u_debug == 3) {
+        gl_FragColor = vec4(colmap, 1.0);
+    } else if (u_debug == 4) {
+        gl_FragColor = vec4(nor, 1.0);
+    } else if (u_debug == 5) {
+        gl_FragColor = vec4(abs(nor), 1.0);
+    } else {
+        gl_FragColor = vec4(1, 0, 1, 1);
+    }
+}

glsl/deferred/ambient.frag.glsl

@@ -16,12 +16,13 @@ void main() {
     vec4 gb2 = texture2D(u_gbufs[2], v_uv);
     vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
-    // TODO: Extract needed properties from the g-buffers into local variables
+    vec3 colmap = gb2.rgb;
+
 
     if (depth == 1.0) {
         gl_FragColor = vec4(0, 0, 0, 0); // set alpha to 0
         return;
     }
 
-    gl_FragColor = vec4(0.1, 0.1, 0.1, 1);  // TODO: replace this
+    gl_FragColor = vec4(0.1 * colmap, 1.0);
 }

glsl/deferred/blinnphong-pointlight.frag.glsl

@@ -6,6 +6,7 @@ precision highp int;
 
 uniform vec3 u_lightCol;
 uniform vec3 u_lightPos;
+uniform vec3 u_cameraPos;
 uniform float u_lightRad;
 uniform sampler2D u_gbufs[NUM_GBUFFERS];
 uniform sampler2D u_depth;
@@ -26,14 +27,27 @@ void main() {
     vec4 gb2 = texture2D(u_gbufs[2], v_uv);
     vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
-    // TODO: Extract needed properties from the g-buffers into local variables
+    vec3 pos = gb0.xyz;
+    vec3 geomnor = gb1.xyz;
+    vec3 colmap = gb2.rgb;
+    vec3 normap = gb3.xyz;
+    vec3 nor = applyNormalMap(geomnor, normap);
+
+    float dist = distance(pos, u_lightPos);
 
     // If nothing was rendered to this pixel, set alpha to 0 so that the
     // postprocessing step can render the sky color.
-    if (depth == 1.0) {
+    if (depth == 1.0 || u_lightRad < dist) {
         gl_FragColor = vec4(0, 0, 0, 0);
         return;
     }
 
-    gl_FragColor = vec4(0, 0, 1, 1);  // TODO: perform lighting calculations
+    vec3 L = normalize(u_lightPos - pos);
+    vec3 V = normalize(- pos);
+    vec3 H = normalize(L + V);
+    float attenuation = max(0.0, u_lightRad - dist);
+    float lambert = max(0.0, dot(L, nor));
+    float specular = max(0.0, dot(H, nor));
+    vec3 col = colmap * u_lightCol * lambert + specular * u_lightCol;
+    gl_FragColor = vec4(col * attenuation, 1.0);
 }

glsl/deferred/debug.frag.glsl

@@ -26,27 +26,25 @@ void main() {
     vec4 gb2 = texture2D(u_gbufs[2], v_uv);
     vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
-    // TODO: Extract needed properties from the g-buffers into local variables
     // These definitions are suggested for starting out, but you will probably want to change them.
     vec3 pos = gb0.xyz;     // World-space position
-    vec3 geomnor = gb1.xyz;  // Normals of the geometry as defined, without normal mapping
+    vec3 geomnor = gb1.xyz; // Normals of the geometry as defined, without normal mapping
     vec3 colmap = gb2.rgb;  // The color map - unlit "albedo" (surface color)
-    vec3 normap = gb3.xyz;  // The raw normal map (normals relative to the surface they're on)
+    vec3 normap = gb3.xyz; // The raw normal map (normals relative to the surface they're on)
     vec3 nor = applyNormalMap (geomnor, normap);     // The true normals as we want to light them - with the normal map applied to the geometry normals (applyNormalMap above)
 
-    // TODO: uncomment
     if (u_debug == 0) {
         gl_FragColor = vec4(vec3(depth), 1.0);
     } else if (u_debug == 1) {
-        // gl_FragColor = vec4(abs(pos) * 0.1, 1.0);
+        gl_FragColor = vec4(abs(pos) * 0.1, 1.0);
     } else if (u_debug == 2) {
-        // gl_FragColor = vec4(abs(geomnor), 1.0);
+        gl_FragColor = vec4(abs(geomnor), 1.0);
     } else if (u_debug == 3) {
-        // gl_FragColor = vec4(colmap, 1.0);
+        gl_FragColor = vec4(colmap, 1.0);
     } else if (u_debug == 4) {
-        // gl_FragColor = vec4(normap, 1.0);
+        gl_FragColor = vec4(normap, 1.0);
     } else if (u_debug == 5) {
-        // gl_FragColor = vec4(abs(nor), 1.0);
+        gl_FragColor = vec4(abs(nor), 1.0);
     } else {
         gl_FragColor = vec4(1, 0, 1, 1);
     }

glsl/post/bloom.frag.glsl

@@ -0,0 +1,14 @@
+#version 100
+precision highp float;
+precision highp int;
+
+uniform sampler2D u_color;
+uniform sampler2D u_glow;
+
+varying vec2 v_uv;
+
+void main() {
+  vec4 color = texture2D(u_color, v_uv);
+  vec4 glow = texture2D(u_glow, v_uv);
+  gl_FragColor = vec4((color + glow).rgb, 1.0);
+}