Qt screenshots and scaling

[hmaarrfk]

On our qt application, we would like to take a "screenshot" of the window.

However, wgpu seems to draw directly onto the surface and bypasses qt's paintEvent so qt can't do it all on its own.

I've modified the triangle_qt_embed.py demo to allow it to fire off a request for a screenshot using Qt's grab() method.
hmaarrfk/wgpu-py#1

(PS. I'll continue this draft of an issue in a bit, I'm going to try to provide an example of working code to "pull a screenshot" forcifully from the renderer's internal buffer)

[Korijn]

You might want to compare your approach to solutions for Qt with pyopengl as well

[hmaarrfk]

Hmm. Great tip!

Truthfully I had a much harder translating my pygfx demo into a wgpu demo......

For the onlookers browsing. The symptom are that the widget on which the rendering is happening in wgpu is just blank.

My "fix" is to grab the rendered texture on the GPU and then rescale it myself prior to pasting it into the rest of the buffer provided by Qt.

It's pretty tricky because on pygfx at least, the internal texture can be of a different size than the Qt widget. So you have to make sure to resize things correctly (at least in my strategy)

[almarklein]

The best solution would be to take the GPU screenshot from the canvas' frame buffer. However, that texture has a lifetime that is bound to a "draw event". Or maybe it lives until the next draw event? I'd have to try/check.

Another possible route might be to have a public method to draw (I think this came up somewhere else to), and perhaps such a method could provide a custom texture. Basically a method to take a screenshot (to a texture) of any size you want.

[hmaarrfk]

I think this came up somewhere else to

I saw this issue today, pygfx/pygfx#754 so it encouraged me to post my qt issue here to give an other perspective.

I updated my example above to generate:

The red and blue rectangles are there to give you a sense that if we can somehow get the texture (correctly scaled for qt ....) then we can place it in the right position ourselves manually as a workaround. Of course, the best way would be to play nice with Qt, but sometimes thats just hard....

[hmaarrfk]

In pygfx, I have access to renderer.snapshot() which makes it possible to get the right data in there. I just don't know how to do it (yet) with wgpu.

[panxinmiao]

The usage flag of the surface texture (obtained from the context.get_current_texture() method) does not include COPY_SRC, so the texture data cannot be read directly from it.

If you need to capture a screenshot of the rendered scene, you should create a render target texture yourself with the usage flag set to COPY_SRC | RENDER_ATTACHMENTS and use it as the color attachment in the render pass. Then, use command_encoder.copy_texture_to_texture() to copy the data to the surface texture.
When you call your screenshot() method, you can read the data from the target texture you created.

For QT applications, if you want to capture the entire GUI interface, not just the rendered scene, the best approach is to customize a QT widget by overriding its paintEvent method. Offscreen rendering with wgpu-py can easily assist you in achieving this.
Here is a sample code snippet (modified from triangle_qt_embed.py):

import importlib

# For the sake of making this example Just Work, we try multiple QT libs
for lib in ("PySide6", "PyQt6", "PySide2", "PyQt5"):
    try:
        QtWidgets = importlib.import_module(".QtWidgets", lib)
        break
    except ModuleNotFoundError:
        pass


from wgpu.gui.offscreen import WgpuCanvas

from triangle import main


from PySide6.QtWidgets import QWidget
from PySide6.QtGui import QPainter, QImage

class RenderableCanvas(QWidget):
    def __init__(self, parent=None):
        super().__init__(parent)
        self._canvas = WgpuCanvas()

    def paintEvent(self, event):
        main(self._canvas) # do the animation and render logic
        frame = self._canvas.draw()
        painter = QPainter(self)
        width, height = frame.shape[1], frame.shape[0]
        img = QImage(frame, width, height, 4 * width,  QImage.Format_RGBA8888)
        painter.drawImage(event.rect(), img)


    def resizeEvent(self, event):
        self._canvas.set_logical_size(event.size().width(), event.size().height())


class ExampleWidget(QtWidgets.QWidget):
    def __init__(self):
        super().__init__()
        self.resize(640, 480)
        self.setWindowTitle("wgpu triangle embedded in a qt app")

        splitter = QtWidgets.QSplitter()

        self.button = QtWidgets.QPushButton("screnshot", self)
        self.button.clicked.connect(self.screenshot)
        self.canvas1 = RenderableCanvas(self)
        self.canvas2 = RenderableCanvas(self)

        splitter.addWidget(self.canvas1)
        splitter.addWidget(self.canvas2)

        layout = QtWidgets.QHBoxLayout()
        layout.addWidget(self.button, 0)
        layout.addWidget(splitter, 1)
        self.setLayout(layout)

        self.show()

    def screenshot(self):
        self.grab().save("screenshot.png")


app = QtWidgets.QApplication([])
example = ExampleWidget()

# Enter Qt event loop (compatible with qt5/qt6)
app.exec() if hasattr(app, "exec") else app.exec_()

[hmaarrfk]

Thank you for the detailed response, it will take me time to digest it.

[almarklein]

Since pygfx/wgpu-py#586 you can instantiate a Qt Canvas with QWgpuWidget(present_method="image"), which should make it possible to make screenshots via Qt that include the rendered content.

You'll need to specify this when the widget is being created. At the time I looked into being able to change the method at runtime, but this complicated the logic in the GpuCanvasContext a lot. Since the "screen" method incurs a performance penalty, I'd recommend only turning it on when you want to create screenshots.

[hmaarrfk]

Interesting, i didn't expect you to be so active on this repo. I'll have to watch for all notifications here too!

[hmaarrfk]

thanks for the warning about the performance penalty. i don't really think we can afford it. we run most of our stuff on Intel Integrated....

[almarklein]

thanks for the warning about the performance penalty. i don't really think we can afford it. we run most of our stuff on Intel Integrated....

If you want to make screenshot for say a presentation, then you can run your app with present_method="image", take your shots, and turn it off again. But if you want to be able to take a screenshot at any moment in the normal workflow, yeah, we'd need something else. I'll leave the issue open to cover that case.

[almarklein]

Correction: above I mentioned QWgpuWidget(present_method="screen"), but the arg should be "image"!

[almarklein]

I was working on stuff that touches on this. In order for canvas content to show up on Qt's screenshots (QWidget.grab()), without compromising performance, we should set present_method='image' temporarily. Making this work is technically possible, but makes the code related to presenting quite a bit more complex. I think I consider this problem out of scope for wgpu/rendercanvas.

I think this problem must be solved another way, by taking a screenshot of the application from the framebuffer. I found a tiny pure-Python, cross-platform, zero-deps, library that does this: mss. It's API is a bit quirky, but it even has a tiny pure-Python png exporter!

For Qt:

import mss

def screenshot(toplevelwidget, filename):
    g = toplevelwidget.geometry()
    rect = {"top": g.top(), "left": g.left(), "width": g.width(), "height": g.height()}
    with mss.mss() as sct:
        screenshot = sct.grab(rect)
    mss.tools.to_png(screenshot.rgb, screenshot.size, output=filename)