Camera support?

[Boscop]

I'm trying to find a cross-platform crate to read fullhd webcam frames at 30fps.
On windows I'm using a custom version of escapi (hacked to support MJPEG decoding with the mozjpeg-sys crate), but I also want to support linux..

Any idea when this crate will support camera capture? :)

[AldaronLau]

Thanks for your interest in my crate! Right now, adding camera support isn't super high on my priority list, but I have a small C library that does camera capture on Linux which wouldn't be too difficult to port to Rust (https://github.com/OxyDeadbeef/lib-car-wreck).

Honestly, I could probably port it in a small amount of time (unless you want to). Then, I could make a crate (probably called adi_camera) that incorporates your custom version of escapi and my ported C code, which this crate can depend on.

What do you think?

[Boscop]

I just tested the rscam crate on linux, and I wish there was something like that for windows.
The problem is, my modified version of escapi is currently blind because escapi doesn't allow querying for supported formats, so I had to hardcode it for my specific camera, but I want it to work with all cameras.
More details here: jarikomppa/escapi#14

So what I would ideally want to do with escapi is: find the highest resolution with at least 20 fps and use that format (and when it's MJPEG I can decode it myself).

With rscam, it is possible, but not with escapi right now.. But I need windows support much more than linux support because my linux laptop has no GPU and I need to heavily postprocess the webcam frames etc., and I need to load a lot of windows-specific VST dlls for audio processing..

[AldaronLau]

It probably wouldn't be too difficult to add as a feature. Just find the right winapi functions and wrap them in Rust. I'd need to look at the escapi project a bit more. It looks like it is not entirely written in Rust? Ideally, webcam support in this crate would be pure Rust (depending on the winapi crate on WIndows).

If possible, I would like to see the diff file of your hacks on the escapi crate.

[Boscop]

escapi is written in C++, I had written some Rust bindings for v2 but then v3 came out with its own Rust bindings but which weren't really suitable for my use case because they didn't allow getting a device's name before opening it, which I need because before I'm opening the camera device, I have to find its number by the name that I'm reading from a config file.. So I adapted my own Rust bindings to work with v3, which was easy, because the API didn't really change, only the implementation switched to windows media foundation (from directshow)..
Then I had to modify the C++ code of escapi to impl mjpeg support (pass-thru of the buffer in escapi and decoding in my Rust app)..
My modified version of escapi is here:
Boscop/escapi@0a25e95

My Rust bindings are here: https://github.com/Boscop/escapi-rs

In my app I'm using my Rust bindings with my escapi fork + mozjpeg-sys to decode the frames.

By winapi do you mean directshow? I've never worked with directshow or WMF directly but IIRC, the guy who wrote OBS said on IRC that directshow is much easier to work with, and there's no real reason to use WMF, because directshow won't be deprecated..
He also wrote this lib: https://github.com/obsproject/libdshowcapture
It's used in OBS for camera capture.
Maybe it would make sense to write Rust bindings for that? But it's C++..
But this is the most flexible and working solution for webcam reading I found, also allows querying the cam for supported formats/resolutions/framerates..
Or maybe it wouldn't be too difficult to just port this lib to Rust, and then write a wrapper crate that abstracts over the platform differences by providing a uniform cross-platform interface and uses the rscam crate on linux.

[AldaronLau]

directshow won't be deprecated? https://en.wikipedia.org/wiki/DirectShow says "Microsoft plans to completely replace DirectShow gradually with Media Foundation in future Windows versions". It sounds like deprecation to me. But OBS is a respectable source, so I could be wrong.

I actually haven't done any webcam stuff on Windows before. My experience is entirely on Linux in this field, so I thought webcam stuff might be built into the winapi which seems to be huge an all encompassing.

Either way, I would like to avoid writing wrapper crates around C++ APIS, and have a rust crate that calls to directshow or WMF directly, which will be abstracted by a cross platform crate.

[AldaronLau]

directshow is good enough for now. If Windows deprecates it in the future, we can always add WMF support later.

[AldaronLau]

Are you sure directshow is easier? This stackoverflow post says otherwise: https://stackoverflow.com/a/1259459/5286593

[Boscop]

Not sure, since I also haven't worked with it directly but I IIRC that's what the OBS creator said on IRC..
But it probably makes sense to look more into it before making a decision.
Btw, the wiki article also says:

Microsoft's MSFT Becky Weiss also confirms that "you'll notice that working with the Media Foundation requires you to work at a slightly lower level than working with DirectShow would have. And there are still DirectShow features that aren't (yet) in Media Foundation".

Not sure if the last sentence is still true at this point in time, but the first one probably is.

[AldaronLau]

I'm starting to lean towards WMF, since it's newer (And it looks like it's still higher level than v4l2).

[Boscop]

Makes sense..