GSoC 2024: Rendering Performance Infrastructure Improvements #1773
Replies: 13 comments
-
Week 1 ReportIn my first week, I've started by assessing the current state of the gpu execution features. As all of the code was written before the adaptive resolution system was developed, I first had to add some bandaids to get the code to compile. Once that was done, I discovered that the way we currently deal with surfaces only allows for one global surface configuration, which is insufficient. Next week:
|
Beta Was this translation helpful? Give feedback.
-
Week 2 ReportThis week, I made significant progress on several fronts, primarily focusing on debugging and improving the GPU execution features, particularly in web browsers. I started by finalizing the A major portion of the week was spent debugging and resolving execution errors across different environments. I ported functionality from the node-graph-executor to the cli-client, which helped identify and fix errors in the node definitions. This process led to a more consistent approach in handling surface/window types for both WASM and native environments. I encountered and resolved several texture-related issues, including implementing a conversion to upload sRGBA8 values to texture buffers and experimenting with half-precision color formats because Rgba16Float is not support by any browser I tested and we'd like to not have to do gamma conversions when we can avoid doing so. But this did ultimately not fix the issue. A significant breakthrough came when I finally resolved browser compatibility issues. After extensive testing and troubleshooting, I managed to get the GPU features working in both Firefox and Chrome on different devices. This was due to Vulkan library packaging issues in NixOS and forcing Chrome to launch under Wayland. Towards the end of the week, I focused on improving rendering performance, particularly during panning operations. I identified that a significant amount of time was being spent on memory copying and UI updates, likely due to resource storage in the node graph. To address this, I began moving the network change detection to the node-graph executor instead of the node runtime, though this introduced some challenges with message deduplication. Next week:
|
Beta Was this translation helpful? Give feedback.
-
Week 3 ReportThis week, I started working on integrating vello as a rendering backend for Graphite. This did include a bunch of plumbing to get the basic data structures in place. I then spend a bunch of time debugging why we didn't get any output. After I discovered how to render to a surface directly I did eventually manage to render an artboard. I then moved the vello renderer into the gpu executor to reduce the overhead of initializing it every frame. After that I kept working on getting more things to render correctly. I managed to get viewport panning to work by appending the rendered scene into a parent scene with the viewport transform. I then also made some progress on rendering other shapes, but the transforms seem to still be wrong. There are also issues related to the translation of our stroke styling to velllo. On Friday we discovered some Undefined Behaviour which breaks our ci runs. Next week:
|
Beta Was this translation helpful? Give feedback.
-
Week 4 ReportThis week was mostly spend on trying to fix the flaky ci tests. Early on I discovered while running miri that there was a use afetr free caused by the let node. I then spent a few days developing a global system to replace the Let binding concept which did in the end resolve the miri issue but not the CI segfaults. I then continued to rebase my open pull request until the gpu restoration pr ran into a ci issue caused by the thread local node runtime. I then worked on replacing the thread local data with a Mutex and made the nodes on non wasm targets return Next week:
|
Beta Was this translation helpful? Give feedback.
-
Week 5 ReportThis week, I've continued to push the vello rendering further. We can now render basic scenes and open the Demo Artworks without crashing. There is some work that needs to be done to support gradients and paths as currently using the gradient tool breaks them. Next week:
|
Beta Was this translation helpful? Give feedback.
-
Week 6 ReportAt the beginning of last week, I workend on improving the ux of the vello preference by graying out the button on non-supported plattforms. I then worked on sandboxing the nodegraph execution such that a panic of the node graph does not bring down tho whole editor. On non wasm32 targets this is implemented pretty cleanly using Next week:
|
Beta Was this translation helpful? Give feedback.
-
Week 7 ReportLast week I worked on fixing more of the blending issues and the offset in the potted cactus artwork. Next week:
|
Beta Was this translation helpful? Give feedback.
-
Week 8 ReportAfter managing to basically finish the vello integration and fixing the remaining issues, I worked on reviving the imaginate node. After that was done I shifted my focus towards fixing the Mandelbrot and noise nodes and ported the noise node to work with the adaptive resolution system. I then tried to integrate proper footprint based arboard clipping but took a break after making little progress on that front. I then worked on creating benchmarks for our compilation process and spend the rest of the week optimizing our compilation process which was pretty effective.
Next week:
|
Beta Was this translation helpful? Give feedback.
-
Week 9 ReportSince my last update, I finished my pending prs to add clippy and performance regression testing to the ci runs. #[node_fn(category("Value")]
async fn construct_vector2<IY: Into<f64> + Send + 'n, IC: Into<u64> + Send + 'n>(
footprint: Footrpint,
#[default(1.3)] x: f64,
#[implementations(f32, f64)] y: IY,
#[implementations(((), u32),((), u64),)] c: impl Node<'n, (), Output = IC>,
) -> glam::DVec2 {
glam::DVec2::new(x, y.into())
}This should also make it easier to iterate on variations in the future. Next week:
|
Beta Was this translation helpful? Give feedback.
-
Week 10 ReportThis week I have been focused on actually migrating the codebase to use the new macro and adding new features to the macro as needed #[node_fn(category("Value")]
#[node_macro::new_node_fn(category("Style"))]
async fn blend_mode<T: SetBlendMode>(
footprint: Footprint,
#[expose]
#[implementations((Footprint, crate::vector::VectorData), (Footprint, crate::GraphicGroup), (Footprint, ImageFrame<Color>))]
value: impl Node<Footprint, Output = T>,
blend_mode: BlendMode,
) -> T {
let mut value = value.eval(footprint).await;
value.set_blend_mode(blend_mode);
value
}One of the main things I've worked on is the automatic generation of document nodes and properties panel entries for the nodes. This is done based on the type of the input arguments which allows us to reduce the boilerplate in the codebase significantly and also gives us a single location where the entire node is defined, making development easier. Next week:
|
Beta Was this translation helpful? Give feedback.
-
Week 11 ReportThis week I took a small detour from finishing the node fn macro refactor and worked on porting our boolean operations to rust. I did also port the vector nodes and then made the demo art compile again. For the boolean ops the provided test suite does now pass, but when hooking it up to graphite, I got a couple of panics which I am currently investigating. I have now also reached my 350h worked so my main focus will be on getting my current work merged and then writing the report. Next week:
|
Beta Was this translation helpful? Give feedback.
-
Week 12 ReportThis week, I've mainly worked on fixing bugs in the boolean operation implementation, diving deep into the algorithms and developing new methods to debug the various intermediate steps. After first focussing my effort to make the rust version behave the same way to the js version and pinning the divergence to an arbitrary difference in the sorted sequence, I discovered that the reference implementation did not produce the correct results either. And it was detecting the faces incorrectly: (this should be the outer face) But and the end of the Week, I got the face detection mostly fixed (not their usage though) and then pivoted to continuing work on the node macro |
Beta Was this translation helpful? Give feedback.
-
Week 12 & 13 ReportThe last two weeks of my GSoC project were focussed on fixing the remaining issues in the path bool library and the node fn refactor pr. We managed to get both of them merged in time The process of authoring nodes is now much more ergonomic than it was before and we have not found any major regressions introduced by prs which is pretty good considering their large scope. |
Beta Was this translation helpful? Give feedback.
-
GSoC timeline shifted to accommodate weeks of availability. Week 1 will begin June 17 and end the three months later.
About
I'm Dennis and study computer science in Germany. I have contributed for Graphite quite a bit so far and am excited to be able to put aside some time this summer to push the project further.
Synopsis
This project aims to significantly enhance Graphite's rendering performance and capabilities by implementing GPU-accelerated graphics rendering using Vello as a rendering backend, optimizing the existing node graph execution system, and improving the overall editor performance. Key objectives involve improving the speed and ergonomics of the internal compiler infrastructure and laying the groundwork for future raster editing features. My focus is removing rough edges around the adaptive resolution system for users in the node graph and for developers writing efficient nodes.
Benefits
Deliverables
GSoC 2024 Final Report: Enhancing Graphite's Rendering and Node Graph System
Integrate Vello for vector rendering #1802
This pull request integrated Vello as a new rendering backend for Graphite, marking a significant milestone in the project's development. The implementation brought GPU-accelerated rendering to Graphite, dramatically improving performance for complex scenes, containing raster data.
Key achievements included:
Challenges faced during this integration included ensuring browser compatibility across different platforms and optimizing the rendering pipeline for both performance and quality.
Refactor the node macro and simplify most of the node implementations #1942
This PR involved a comprehensive refactoring of the node macro system, representing one of the largest changes in the project. The new system significantly simplified node implementations, making it easier to define and work with nodes in the graph. This pr gives us a single source of truth for the node definition instead of having to keep 4 files in sync whenever you work on a node. This should also make it easier for new contributors to start writing new nodes.
Key improvements included:
This refactoring was challenging due to the need to maintain backwards compatibility while improving the system's performance and flexibility.
Add path-bool library #1952
This pull request introduced a new path-bool library for handling boolean operations on paths. Before this, we used paper.js as a JavaScript module which limits the performance and platform support. The main motivation for me to work on this port was to allow us to add proper graph execution benchmarking, but this will also be crucial to support a desktop release of graphite.
A major challenge encountered was ensuring the correct detection and handling of complex path intersections and edge cases. This required deep diving into geometric algorithms and developing new debugging methods to visualize the intermediate steps of the boolean operations.
Restore functionality of GPU infrastructure #1797
This PR restored and improved the GPU functionality in Graphite, preparing the system for advanced GPU-accelerated features and ensuring compatibility with the new Vello rendering backend. It involved refactoring existing GPU-related code to work with the current architecture of Graphite. While working on this I did also remove the vulkan executor and removed an abstraction layer to simplify working with the code.
Add profiling metrics for, and speed up, Graphene graph compilation #1924
Implemented profiling metrics for the Graphene graph compilation process, allowing for targeted optimizations. This resulted in significant speed improvements in graph compilation, enhancing the overall performance of the node graph system.
Make noise generation resolution aware #1909
Updated the noise generation system to be resolution-aware. This is the first use case of an infinitely scalable raster generator node.
Add benchmarks for runtime profiling #2005
Implemented a system for benchmarks to profile runtime performance. This tool is crucial for ongoing optimization efforts, allowing developers to easily identify and address performance bottlenecks.
Beta Was this translation helpful? Give feedback.
All reactions