Project 5: Akshay Shah

README.md

@@ -3,24 +3,181 @@ WebGL Deferred Shading
 
 **University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 5**
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
+* Akshay Shah
+* Tested on: **Google Chrome 54.0.2840.87 m** on Windows 10, i7-5700HQ @ 2.70GHz 16GB, GTX 970M 6GB (Personal Computer)
 
 ### Live Online
 
-[![](img/thumb.png)](http://TODO.github.io/Project5B-WebGL-Deferred-Shading)
+[![Deferred Shader](img/deferred_shader.PNG)](https://aksris.github.io/Project5-WebGL-Deferred-Shading-with-glTF/)
 
 ### Demo Video/GIF
 
-[![](img/video.png)](TODO)
+[![](img/bloom.PNG)](https://vimeo.com/190890932)
 
-### (TODO: Your README)
+### Deferred Renderer
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
+Deferred shading is a screen-space method where the first pass does not actually involve shading and is deferred to the second pass. In the first pass positions, normals, and materials for each surface are rendered into the  g-buffer and the shading happens in the deferred pass computing the lighting at each pixel using textures. The primary advantage is that geometry is separate from lighting giving a significant performance improvement over raytracers or rasterizers.
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
+    This renderer uses several debug passes: Depth, position, normal and color map after packing the geometry normal into the copy stage.
 
+| Depth | Position | Color Map | Blinn-Phong |
+| ----- | -------- | --------- | ----------- |
+| ![](img/depth.PNG) | ![](img/position.PNG) | ![](img/colormap.PNG) | ![](img/blinnphong.PNG) |
 
+This is the [sponza scene](http://graphics.cs.williams.edu/data/meshes.xml), with 20 lights (unless mentioned otherwise).
+Optimizations include: gbuffer packing, improved screen-space aabb scissor test
+
+| Without optimizations | With Optimizations |
+| --------------------- | ------------------ |
+| runs @ 40ms | runs @ 28ms |
+
+Effects
+-------
+
+- [x] Implemented deferred Blinn-Phong shading (diffuse + specular) for point lights
+  - With normal mapping
+- [x] Bloom using post-process blur
+  - [x] Two-pass Gaussian blur using separable convolution using a second postprocess render pass) to improve bloom or other 2D blur performance
+
+![](img/bloomunopt.PNG)
+There are also some undesired blurring artifacts visible.
+
+This does not include the optimizations to the shading pass. With all the optimizations, the bloom runs at 34ms.
+
+| Deferred pass | Bloom with 2-pass blur |
+| ------------- | ---------------------- |
+| runs @ 35ms | runs @ 40ms |
+
+| Bloom without optimizations | Bloom with Optimizations |
+| --------------------------- | ------------------------ |
+| runs @ 40ms | runs @ 32ms |
+
+  The bloom pass added 7.91% to the total computation which was only a ~4.8ms of the render time.
+
+- [x] implemented efficient gaussian blur with linear sampling
+
+| Bloom unoptimized | Bloom optimized with efficient linear sampling |
+| ----------------- | ---------------------------------------------- |
+| runs @ 40ms | runs @ 31ms |
+
+There wasn't a very significant improvement in the performance as noted by [this][1] article.
+
+![](img/bloomopt.PNG)
+No more artifacts.
+
+[![](img/bloom.PNG)](https://vimeo.com/190890932)
+
+Optimizations
+-------------
+
+- [x] Scissor test optimization: when accumulating shading from each point light source, only   render in a rectangle around the light.
+  - [x] Improved screen-space AABB for scissor test
+
+  ![](img/scissor_debug.PNG)
+
+  The scissor test is an optimization that discards fragments that are out of the light's bounding box portion of the screen, thus improving the shading performance.
+
+  > Polygons are clipped to the edge of projection space, but other draw operations like glClear() are not. So, you use glViewport() to determine the location and size of the screen space viewport region, but the rasterizer can still occasionally render pixels outside that region.
+
+  Borrowed from [postgoodism](http://gamedev.stackexchange.com/users/19286/postgoodism) from [gamedev.stackexchange.com](http://gamedev.stackexchange.com/questions/40704/what-is-the-purpose-of-glscissor)
+
+```javascript
+ a.fromArray(l.pos);
+a.w = 1;
+a.applyMatrix4(view);
+a.x -= l.rad;
+a.y -= l.rad;
+// a.z += l.rad;
+
+// front bottom-left corner of sphere's bounding cube
+b.fromArray(l.pos);
+b.w = 1;
+b.applyMatrix4(view);
+b.x = a.x + l.rad * 2.0;
+b.y = a.y + l.rad * 2.0;
+b.z = a.z;
+a.applyMatrix4(proj);
+a.divideScalar(a.w);
+b.applyMatrix4(proj);
+b.divideScalar(b.w);
+```
+
+Here I'm not using `a.z`, but instead just reusing the variables to fill the `b` array. This neat little trick can improve the performance by a healthy 5~7ms.
+
+- [x] Optimized g-buffer format - reduced the number and size of g-buffers:
+  - Reduce number of properties passed via g-buffer, by:
+    - Applying the normal map in the copy shader pass instead of copying both geometry normals and normal maps
+
+So instead of using 4 GBUFFERS, I pack the normal in the copy pass as the geometry normal was only being used to calculate the surface normals which was what I needed all along.
+
+So in the copy fragment shader:
+
+```glsl
+#version 100
+#extension GL_EXT_draw_buffers: enable
+precision highp float;
+precision highp int;
+
+uniform sampler2D u_colmap;
+uniform sampler2D u_normap;
+
+varying vec3 v_position;
+varying vec3 v_normal;
+varying vec2 v_uv;
+
+vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
+    normap = normap * 2.0 - 1.0;
+    vec3 up = normalize(vec3(0.001, 1, 0.001));
+    vec3 surftan = normalize(cross(geomnor, up));
+    vec3 surfbinor = cross(geomnor, surftan);
+
+    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
+}
+void main() {
+
+    gl_FragData[0] = vec4( v_position, 1.0 );
+    gl_FragData[1] = vec4(applyNormalMap(v_normal, texture2D(u_normap, v_uv).rgb), 1.0);
+    gl_FragData[2] = texture2D(u_colmap, v_uv).rgba;
+
+}
+
+```
+
+| With 4 GBuffers | With 3 GBuffers |
+| --------------- | --------------- |
+| runs @ 40ms | runs @ 36ms |
+
+I figured most of the time spent calculating normals in the deferred pass is now being done in the copy pass and more registers on the GPU are freed up.
+
+More stuff
+-----------
+
+- [x] Toon shading (ramp shader + simple edge detection)
+![](img/toon.png)
+![](img/toon2edit.PNG)
+Simple edge detection with ramp shader
+![](img/toon2edge.PNG)
+
+- [x] Improved screen-space AABB for scissor test
+- [x] Two-pass Gaussian blur using separable convolution (using a second postprocess render pass) to improve bloom or other 2D blur performance
+- [x] Implemented a sobel edge filter
+![](img/sobel_edge.PNG)
+
+GPU tracing on Chrome
+---------------------
+
+Trying to make heads and tails of the gpu tracing in chrome.
+![](img/gputrace.PNG)
+
+### References
+
+* [Three.js](https://github.com/mrdoob/three.js) by [@mrdoob](https://github.com/mrdoob) and contributors
+* [stats.js](https://github.com/mrdoob/stats.js) by [@mrdoob](https://github.com/mrdoob) and contributors
+* [webgl-debug](https://github.com/KhronosGroup/WebGLDeveloperTools) by Khronos Group Inc.
+* [glMatrix](https://github.com/toji/gl-matrix) by [@toji](https://github.com/toji) and contributors
+* [minimal-gltf-loader](https://github.com/shrekshao/minimal-gltf-loader) by [@shrekshao](https://github.com/shrekshao)
+* [Toon shader](http://in2gpu.com/2014/06/23/toon-shading-effect-and-simple-contour-detection/) by Sergiu Craitoiu
+* [Silhouette Extraction](http://prideout.net/blog/?p=54) by Philip Rideout
+* [Bloom effect](http://learnopengl.com/#!Advanced-Lighting/Bloom) by Joey de Vries
+* [Sobel and Frei-Chen edge detector](http://rastergrid.com/blog/2011/01/frei-chen-edge-detector/) by Daniel Rákos
+[1]: http://rastergrid.com/blog/2010/09/efficient-gaussian-blur-with-linear-sampling/ "Efficient Gaussian blur with linear sampling"

glsl/clear.frag.glsl

@@ -3,7 +3,7 @@
 precision highp float;
 precision highp int;
 
-#define NUM_GBUFFERS 4
+#define NUM_GBUFFERS 3
 
 void main() {
     for (int i = 0; i < NUM_GBUFFERS; i++) {

glsl/copy.frag.glsl

@@ -10,11 +10,22 @@ varying vec3 v_position;
 varying vec3 v_normal;
 varying vec2 v_uv;
 
+vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
+    normap = normap * 2.0 - 1.0;
+    vec3 up = normalize(vec3(0.001, 1, 0.001));
+    vec3 surftan = normalize(cross(geomnor, up));
+    vec3 surfbinor = cross(geomnor, surftan);
+
+    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
+}
 void main() {
     // TODO: copy values into gl_FragData[0], [1], etc.
     // You can use the GLSL texture2D function to access the textures using
     // the UV in v_uv.
 
     // this gives you the idea
-    // gl_FragData[0] = vec4( v_position, 1.0 );
+    gl_FragData[0] = vec4( v_position, 1.0 );
+    gl_FragData[1] = vec4(applyNormalMap(v_normal, texture2D(u_normap, v_uv).rgb), 1.0);
+    gl_FragData[2] = texture2D(u_colmap, v_uv).rgba;
+    // gl_FragData[3] = texture2D(u_normap, v_uv).xyzw;
 }

glsl/deferred/ambient.frag.glsl

@@ -3,25 +3,29 @@
 precision highp float;
 precision highp int;
 
-#define NUM_GBUFFERS 4
+#define NUM_GBUFFERS 3
 
 uniform sampler2D u_gbufs[NUM_GBUFFERS];
 uniform sampler2D u_depth;
 
 varying vec2 v_uv;
 
+const vec3 ambient = vec3(0.05, 0.07, 0.12);
+
 void main() {
-    vec4 gb0 = texture2D(u_gbufs[0], v_uv);
-    vec4 gb1 = texture2D(u_gbufs[1], v_uv);
+    // vec4 gb0 = texture2D(u_gbufs[0], v_uv);
+    // vec4 gb1 = texture2D(u_gbufs[1], v_uv);
     vec4 gb2 = texture2D(u_gbufs[2], v_uv);
-    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
+    // vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
     // TODO: Extract needed properties from the g-buffers into local variables
 
-    if (depth == 1.0) {
-        gl_FragColor = vec4(0, 0, 0, 0); // set alpha to 0
-        return;
-    }
+    vec3 colmap = gb2.rgb;
+
+      if (depth == 1.0) {
+          gl_FragColor = vec4(0, 0, 0, 0); // set alpha to 0
+          return;
+      }
 
-    gl_FragColor = vec4(0.1, 0.1, 0.1, 1);  // TODO: replace this
+      gl_FragColor = vec4(vec3(0.1) * colmap, 1);
 }

glsl/deferred/blinnphong-pointlight.frag.glsl

@@ -2,38 +2,89 @@
 precision highp float;
 precision highp int;
 
-#define NUM_GBUFFERS 4
+#define NUM_GBUFFERS 3
+// #extension GL_OES_standard_derivatives : enable
 
 uniform vec3 u_lightCol;
 uniform vec3 u_lightPos;
 uniform float u_lightRad;
+uniform vec3 u_viewDir;
 uniform sampler2D u_gbufs[NUM_GBUFFERS];
 uniform sampler2D u_depth;
+uniform int u_toon;
 
 varying vec2 v_uv;
 
-vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
-    normap = normap * 2.0 - 1.0;
-    vec3 up = normalize(vec3(0.001, 1, 0.001));
-    vec3 surftan = normalize(cross(geomnor, up));
-    vec3 surfbinor = cross(geomnor, surftan);
-    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
-}
+// const vec3 ambientColor = vec3(0.1, 0.1, 0.1);
+// const vec3 diffuseColor = vec3(0.5, 0.5, 0.5);
+// const vec3 specColor = vec3(1.0, 1.0, 1.0);
+// const float shininess = 16.0;
+// const float screenGamma = 2.2; // Assume the monitor is calibrated to the sRGB color space
+
+// vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
+//     normap = normap * 2.0 - 1.0;
+//     vec3 up = normalize(vec3(0.001, 1, 0.001));
+//     vec3 surftan = normalize(cross(geomnor, up));
+//     vec3 surfbinor = cross(geomnor, surftan);
+//     return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
+// }
 
 void main() {
     vec4 gb0 = texture2D(u_gbufs[0], v_uv);
     vec4 gb1 = texture2D(u_gbufs[1], v_uv);
     vec4 gb2 = texture2D(u_gbufs[2], v_uv);
-    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
+    // vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
     // TODO: Extract needed properties from the g-buffers into local variables
+    vec3 pos = gb0.xyz;     // World-space position
+    // vec3 geomnor = gb1.xyz;  // Normals of the geometry as defined, without normal mapping
+    vec3 nor = normalize(gb1.xyz);
+    vec3 colmap = gb2.rgb;  // The color map - unlit "albedo" (surface color)
+    // vec3 normap = gb3.xyz;  // The raw normal map (normals relative to the surface they're on)
+    // vec3 nor = applyNormalMap (geomnor, normap);     // The true normals as we want to light them - with the normal map applied to the geometry normals (applyNormalMap above)
 
     // If nothing was rendered to this pixel, set alpha to 0 so that the
     // postprocessing step can render the sky color.
-    if (depth == 1.0) {
-        gl_FragColor = vec4(0, 0, 0, 0);
+    // if (depth == 1.0) {
+    //     gl_FragColor = vec4(0, 0, 0, 0);
+    //     return;
+    // }
+
+    float lightDistance = distance(u_lightPos, pos);
+    if (lightDistance  > u_lightRad) {
         return;
     }
+    float edgeDetection = 1.0;
+
+    vec3 lightDir = normalize(u_lightPos - pos);
+    float lambertian = clamp(dot(lightDir, normalize(nor)), 0.0, 1.0);
+    vec3 viewDir = normalize(u_viewDir - pos);
+     // this is blinn phong
+    vec3 halfDir = normalize(lightDir + viewDir);
+    float specAngle = max(dot(halfDir, nor), 0.0);
+    float specular = float(lambertian > 0.0) * pow(specAngle, 16.0);
+    float falloff = clamp(1.0 - (lightDistance * lightDistance)/(u_lightRad * u_lightRad), 0.0, 1.0);
+     if(u_toon == 1) {
+        /* Reference: http://prideout.net/blog/?p=22 */
+        // float cutoff = 3.0;
+        // lambertian = ceil(lambertian * cutoff) / cutoff;
+        // specular = ceil(specular * cutoff) / cutoff;
+        // falloff = ceil(falloff * cutoff) / cutoff;
+        const float A = 0.1;
+        const float B = 0.3;
+        const float C = 0.6;
+        const float D = 1.0;
+
+        if (lambertian < A) lambertian = 0.1;
+        else if (lambertian < B) lambertian = 0.35;
+        else if (lambertian < C) lambertian = 0.7;
+        else lambertian = D;
+        specular = step(0.5, specular);
+        edgeDetection = (dot(viewDir, nor) > 0.4) ? 1.0 : 0.0;
+    }
 
-    gl_FragColor = vec4(0, 0, 1, 1);  // TODO: perform lighting calculations
+    gl_FragColor = vec4(
+        (colmap * lambertian +
+        specular * vec3(1.0)) * u_lightCol * falloff * edgeDetection
+        , 1);
 }