Support for Handwritten Notes/ Drawings

[pfmephisto]

Hey,

has anyone considered adding support for handwriting/drawing using a table to the notes app? Or has any idea what that would require?
I was considering getting myself and e-ink note-taking tablet and was looking at open source self hosted solution for storing and managing my notes. Some of those devices run regular android. So my idea was just finding and android based app. And since I am already using nextcloud, notes would have been a great solution.

I'm just not sure how this could be added nicely together with the already existing regular typed text in the notes app or how it would translate to a web based interface (non touch/pen sensitive devices).

But I though we could start a discussion here.

[stefan-niedermann]

I have considered this for the Notes Android client, but it required stuff like a Samsung SDK to support the S-Pen - and this is proprietary unfortunately.

I am not sure how this might work for the Notes web app - usually a website doesn't have direct access to such devices. Usually this is / should provided as an input method by the operating system. Will be interesting to hear opinions and maybe some tricks & knowledge about this topic 🙂

[pfmephisto]

Does every manufacturer implement their own library for supporting pens? I was hoping that maybe android had some rudimentary support build in.
But otherwise my starting point would have been taking a look at this repository.

https://github.com/onyx-intl/OnyxAndroidDemo

[J4gQBqqR]

@stefan-niedermann
Maybe I am wrong, but I think Xournalpp has Android and web client that supports stylus already? Can the knowledge be borrowed?

xournalpp/xournalpp#2128

[stefan-niedermann]

@J4gQBqqR as far as i can see, they support only generating graphics, not for interpreting the handwritten notes as actual text content. Since the Notes app does not yet support attachments or other file types, this isn't helpful at this point. We need some kind of text recognition, so this is unfortunately not helpful right now. 🙁

[pfmephisto]

@stefan-niedermann Onyx's own Notes app supports handwritten text recognition. So the hardware itself seems to be capable of doing it.
But I would agree that it would be nicer to have support for attachments inside Nextcloud Note and handle any text recognition oneself or server-side. That at least in my opinion would allow for the most flexibility of being able to store drawings/sketches, but also to search ones notebooks based on text.

[poperigby]

Once #785 is merged, could this be handled by embedding SVGs?

[dd88w]

and maybe users can add audios as a part of a text note too, it could be very helpful when people in a discussion need to be recorded or a student taking notes in a class..
for example, audio clips lined up follow the timeline of a single note, and there would be text content additionally explain what the audio clip saying.

[szaimen]

Cc @juliushaertl should we move this over to text now that it is planned to change notes to use text?

[thezanderlv]

Needs Apple Pencil support.

[relink2013]

What would be the best place to track this feature going forward?

Plenty of web-apps can use Apple Pencil, i don't know if its full featured or not, but its enough for my needs.

I think my needs are fairly simple, if i could just simply draw freeform in different colors id be happy. If i could draw on top of embedded attachments like images and PDFs i could finally get rid of Notability!

[Himmelxd]

Pen in web has good support
https://developer.mozilla.org/en-US/docs/Web/API/PointerEvent/pressure
I already worked on a notes app for web. But I'm struggling yet implementing it into a Nextcloud App.
Simple note taking (without many features) would probably be not that hard to implement into the existing one.
Only problem would be the mobile app, displaying may be somewhat possible, but editing it without opening it in web again might be hard.
As drawing representation backbones it is possible to use (in my experience) canvas/rastered and svg.

[heluca]

SVG support in Nextcloud Notes would be amazing as I take handwritten notes all the time using Stylus Labs Write and store the files in Nextcloud today. Their app uses SVG (actually compressed svgz by default), but doesn't support Markdown. The real challenge is not having a platform that brings Markdown and SVG together, and I think Nextcloud Notes could do that successfully.

Stylus Labs Write isn't open source, but is available for Linux, Windows, and Android for free (as in beer), and is only a few dollars on IOS. It also supports adding embed images (photos/screenshots) and after using it for a few years I think it serves as an excellent example. Developer support has been very good and the app is very customizable. https://www.styluslabs.com/

Also, I think it is common knowledge that SVG works in web browsers out of the box, but they also can be edited in LibreOffice Draw & Inkscape.

Bringing these files together is an important step to a much more interesting goal. Analysis with deep learning to find words, summarize notes into markdown, identify to dos and due dates and automatically create them in task and calendar apps.