Non-US keyboard layouts

[ditron]

A nice feature for me and all others with non-US keyboards is to add a keyboard layout option list and/or trying to auto detect locale settings from the browser.

[kanaka]

That's a great idea. Except I have no experience in that area and no easy way to validate it even if I did implement it. Any examples of good implementations? Or even better would be working patches :-)

[ghost]

http://guacamole.svn.sourceforge.net/viewvc/guacamole/trunk/web/javascript/

[kanaka]

A while ago Colin Dean of xvpsource.org sent me a patch that attempted to add Non-US keyboard layout support.

Here is the original patch: https://gist.github.com/763752

Here is the patch with some modifications I made to it: https://gist.github.com/2e7f20be852a4527df63

The patch switches from using keydown and keyup events to using the keypress event to get the charCode value which is then mapped through a Unicode table to get the keysym value.

Unfortunately, after playing around with it I discovered a couple of issues that prevent it from being a solution for noVNC:

• Using the keypress event means that we can't distinguish and send separate keydown and keyup events. This means the key events sent to the server don't truely reflect what the user is doing on the client side.
• To receive the keypress event we have to allow the keydown event to propagate normally. If we stop the keydown event, the keypress event will never fire. However, stopping the event at keypress time is too late to prevent certain browser hot-key actions from firing. For example, if the user presses Ctrl-A and we don't stop the keydown event from propagating, then everything on the page is selected. This is an annoyance, but there are other cases that are worse such as Ctrl-W.

The implication is that in order to do the right thing we will have to determine the right keysym from the keyCode of the keydown event and not rely on the charCode of the keypress event.

Most of Colin's patch probably won't be used (although the u2x11 script probably will be), but it was invaluable to me to figure out what really needs to be done.

[kanaka]

Some useful links:

• quirksmode writeup of keystroke issues: http://www.quirksmode.org/js/keys.html
• quirksmode page on events in general: http://www.quirksmode.org/dom/w3c_events.html
• more cross-browser info mentioned by that quirksmode page: http://www.hallvord.com/opera/keyevents.htm
• detecting shift/ctrl/alt keys: http://www.cross-browser.com/x/examples/shift_mode.php
• more on detecting modifier keys: http://www.javascripter.net/faq/ctrl_alt.htm
• still more on modifiers with specific info for older browsers: http://www.webreference.com/js/column11/modifierkeys.html

It sounds like the altKey, shiftKey and ctrlKey properties of the key events may be fairly cross-browser (in the browser versions we care about) which is very good if true.

[kanaka]

Also, I added tests/keyboard.html which prints keyboard event data (keyCode, charCode, which, altKey, shiftKey, ctrlKey, translated keysym).

[kanaka]

One thought I've had for how to generate keyboard layout data is that we may be able to create a page that generates fake keyboard events to determine what the keycode+modifiers to charCode data is (i.e. generate keydown event and trap the keypress event),

It's possible this is fast enough (and reliable enough) in all browsers we need to support that we might be able to do this on page load in noVNC. But at least we could have a publicly accessible page where people could generate and then submit their layout data.

But at the very least we might be able to use the technique with just a few key events to binary search for the keyboard layout that the user is using. Assuming there is no better way to determine that (I haven't searched in detail for how JavaScript can determine keyboard layout).

[ghost]

i would do both:
a. use key sequences to guess the keyboard
b. let them select

[moobyfr]

if the keyboard layout could be configurable, it could resolve many problems...
for non-us keyboard, it can be very hard to type

[moobyfr]

I tried to integrate the patch from Colin Dean, as noVNC is mostly no usuable for typing text. The git is https://github.com/moobyfr/noVNC
Fell free to comment if it works for you. I tested on french layout, and is ok for normal keys + modifiers. The only issue is AltGr key

[kanaka]

moobyfr, the change in your fork is not actually functional. You have only added the x11.js file without using it. Perhaps you forget to push the rest of the changes.

Also, as I mentioned in the 3rd comment there are two major issues with Colin's approach: the keydown and keyup messages are no longer detected and sent separately, unintended browser events are no longer disabled (e.g. Ctrl-W will close the browser tab rather than sending a Ctrl-W).

I am very willing to accept additional keyboard layouts, however the layout must key off off keydown and keyup events and not key off of the keypress event as with Colin's patch. If you would like to develop a French layout that keys off of keydown/keyup then I would gladly pull it into the project.

[moobyfr]

I've spent some time to look for the way to go. The keydown/keyup is the way to intercept ~all keys. But there is something bad in the management with javascript for key events. keyCode contain a key reference for a qwerty keyboard, and the real char is in CharCode.
So to be realistic, noVNC should transmit keyCode to say which key was pressed.
The basic idea is, as you said , and I tried, is to map keyCode to other values depending to the key pressed. But layout are more than just keys moved: on a azerty keyboard, two differents key can return the same value, and there is no way to distinguish them only with keydown/keyup: Example, the "-" and "_" on a qwerty is on the same key, and differentied with the shift modifier. the keyCode value is 189. On a azerty layout, these two symbols are on two differents keys. the keyCode returned by the browser for each key is 189!
So only with the keyCode (and modifiers), it's not possible to make a table to manage differents layout.
That's why the charCode has to be used.
I've looked in guacamole code, the same is made: trap keydown to know that a key has been pressed, and use keypress, if needed, to get the real charCode to send.

Javascript doesn't give us all the power :/

[kanaka]

noVNC has to use keyCode and not charCode for two reasons: first so that we can send the keydown and keyup separately (for some keys on some layouts we may have to work around this due to keyCode limitations as you describe), but the second and bigger issue is that using charCode requires the key event to propagate/bubble which means that certain key sequences are seen by the browser and cause unintended behavior such as tab, space, enter, Ctrl-W, Ctrl-Tab, etc. If Guacamole is using charCode then they have this problem.

Here is a concrete example of the second problem. On most browsers, Ctrl-Minus will shrink the font size and Ctrl-Plus will increase the font size. In order to prevent this from happening in noVNC (and just send the Ctrl-Minus/Plus on to the server), we stop the keyDown event and don't allow it to propagate. However, another effect of stopping the event is that there is no keyPress event generated. I don't know of any way to both generate a keyPress event and prevent bubbling into default browser actions.

That is really terrible that on certain keyboard layouts multiple keys generate the same keyCode in Javascript.

If anybody has a sudden stroke of brilliance about how to solve the keyboard layout problems without bad side-effects, I'm all ears. In the meantime, we may just have to provide a way (button) for the user to send a '_' character when using an azerty keyboard.

Do we know of other keys that have the same overlapping key layout abomination?

[kanaka]

I may ask this question on stackoverflow.

[kanaka]

I've asked the question on stackoverflow here: http://stackoverflow.com/questions/5306132/translate-javascript-keycode-into-charcode-for-non-u-s-keyboard-layout-i-e-aze

[moobyfr]

more informations: 3x 2 keys have same keyCode: ";" and ":" are differents keys, and "!" and shift + "&"
Some more fun: 6 other keys send a 0 value for keyCode: (in the qwerty mapping for having a better idea of the mapping) : the key 2 7 9 0 and the two keys right of "p" and "m"
And the best: the key 4 (" ' " on a azerty ) only show keypress events, and no up/down!!

I have another proposition: add an option to noVNC to use the up/down event only for qwerty users, or the keypress for international users with less functionalities...

[kanaka]

moobyfr, thanks for your work on this. It's a really messy area and I just use a standard U.S. keyboard layout so it's difficult for me to determine exactly where the issues lie.

It looks like it's going to have to be something like a fallback for non-US layouts. Although, again, the biggest issue is not getting the mapping correct, the biggest problem is preventing unintended default browser behaviors. So it's likely that keypress will only be used for non-US keys when no modifier key (other than shift) is pressed at the same time. When a modifier (other than shift) is pressed we still can't use keypress on any key layout.

[kanaka]

Okay, it's a huge challenge but I think I've made some forward progress here:

c96f900

In summary:

• I've added a keyPress handler and I report a key down for normal (safe) key combinations when the keyPress event happens.
• For special (browser action) key combinations I report the key down in the keyDown event and suppress further action.
• In order to report the key up properly I add a key event to a list in keyDown, give that element a keysym value in keyPress, and then pop that item off the list and use the keysym value during keyUp to report the key up.
• I've also incorporated Colin Deans character code to keysym lookup table for values above 255.

This isn't a 100% solution since key combinations that cause browser events (tab, enter, backspace, Ctrl-, Alt-, etc) are not translated before being sent, but for normal typing we should be able to get keyboard mapping correct now.

I'm going to leave the bug open for a while since I have not been able to test this much on alternate keyboards so I'm sure there are issues, but I think the framework is in place to be able to solve most of the issues.

[kanaka]

Also, see this commit and the comments with it: https://github.com/moobyfr/noVNC/commit/c0615b935cc1e0e5cc9be222405185bbf829294a

[kanaka]

I'm closing this issue. I don't have a great way of testing this myself. I can switch keyboard layouts and poke around a bit and things seem to be working fairly well, but I'm not a real user of non-US layouts so I'm relying on community feedback. If anybody listening uses non-US keyboards and is having issues, please re-open. Thanks.

[tenchman]

I'm having problems here with a german keyboard layout and at least the backslash key. Instead of a backslash noVNC sends a left bracket [.

The codes send with different Browsers:

• IE9: FFE3, FFE9, 005B down and 005B, FFE3, FFE9 up
• FF4: FFE3, FFE9, 005B, 005C down and 005C, FFE3 up
• Opera 11.11: FFE3, FFE9, 00DF, 005C down and 005C, FFE3 up

Unfortunately i have not the permissions to reopen this issue.

[kanaka]

@tenchman, I've reopened the bug. I can't seem to replicate your problem on Linux (Ubuntu Maverick) with Chrome 12, firefox 4.0.1, or Opera 11.11. I'll have to try with browsers under Windows at some point (I assume that is where you are seeing this?)

Just to make sure, is this the keyboard layout you are using? http://en.wikipedia.org/wiki/Keyboard_layout#Germany_and_Austria_.28but_not_Switzerland.29
And are you pressing Alt_Gr and then the key two over from the backspace?

[moobyfr]

the current version of novnc can't handle correctly AltGr key. In USA, right ctrl + alt can do the same, but these keys don't exist on european keyboards

[laggarcia]

@benbracha @DirectXMan12 Have you tested with @danielhb patches referenced above? They might fix your issue.

[benbracha]

@laggarcia - yes, I tried @danielhb version

[danielhb]

@benbracha can you specify which German layout are you using? Because I just tested here with my version, with the layout I believe it's the German default for Gnome and the "@" works fine. However it's AltGr+Q that gives "@", not AltGr+I.

[danielhb]

@benbracha don't know if this is exactly the layout you're using, but just tried it here with German - Austria (Macintosh) and it worked too (with AltGr + L ).

What browser are you using? The code will not activate in any browser but latest stable Firefox, Chrome and Opera

[benbracha]

I just installed a German keyboard on my mac (see its layout below) - and locally it prints "@" using alt+l, but I don't see this when using your noVNC version.

Am I missing something?

[danielhb]

What VNC server are you trying to connect to? Are you trying to connect to xvnc, tigervnc, QEMU vnc ...?

Supposing that you're connecting to a QEMU compatible VNC server, using Firefox >= 44 or Chrome >= 48 or Opera >= 36 and mapping the exact same keyboard layout inside the VM, it should work.

I've tested it a lot with Linux and Windows 10 but haven't tried Apple. I'll fire it up an old Mac Air here and see if there's something I didn't predict.

[benbracha]

Thanks. Sorry for not being verbose with the details. So:
I'm connecting to a QEMU vnc, Chrome latest version.
Probably what I'm missing is... "mapping the exact same keyboard layout inside the VM" - what do you mean by that?

(I'm used to work with guacamole, and at least as I understand - the keyboard layout of the VM is not important, as eventually the actual characters (charCode that are raised from keypressed JS events) are sent to the remote VM. I may miss something here though..
I tried your solution as I cannot figure out how to solve "dead-keys" issues which are not working with gacamole, and hoped you solved it).

[laggarcia]

(I'm used to work with guacamole, and at least as I understand - the
keyboard layout of the VM is not important, as eventually the actual
characters (charCode that are raised from keypressed JS events) are
sent to the remote VM. I may miss something here though..

Keyboard layout is not important only when you are using the same
graphical user interface on both sides (client and server), e.g. X11,
which is what VNC protocol uses by default. charCode raised from
keypressed JS event is way high level and a number of transformations
happen between your key press, the operating system getting it and
passing it to the JS engine. If you are connecting to a VM, the best way
to deal with this is to send the actually keycode received by the
operating system to the remote server (that's what QEMU provides through
a VNC extension because on the VNC server end, you actually have a full
operating system running, not only a graphical user interface). But if
you do that you need to set the keyboard layout on the VM properly, as
you are not sending charCodes to the VM, you are sending actually
keycodes (which are much lower level and need to be mapped to a keyboard
layout to be meaningful).

I tried your solution as I cannot figure out how to solve "dead-keys"
issues which are not working with gacamole, and hoped you solved it).

Yeah, that's one of the issues you might see when passing charCodes to
the VM instead of actual low level keycodes.

[benbracha]

@laggarcia Thanks for the explanation! Just to be clear, when you are saying:

Keyboard layout is not important only when you are using the same
graphical user interface on both sides (client and server)

What do you mean by "client and server"? My set up has:

• A client (my local machine, that use noVNC via browser) - Mac OS / Chrome
• A server (the VM that runs QEMU and the noVNC proxy) - Ubuntu 14.04
• A "client VM" (the VM that runs on top QEMU) - Ubuntu 14.04

In this setup - should I set the the "client VM" keyboard layout to German as well (like it is set on my Mac)?

[danielhb]

@benbracha I see where you're coming from. I'll explain why as succinct as I can, because this is a long story/topic and @laggarcia already explained a bit.

The implementation you're talking about is the regular VNC implementation of key events. You send the key symbol (keysym) to the remote server, the remote server shows it and that's it. This is trivial for VNC clients that have direct access to the keyboard map of the OS and can tell, for example, that AltGr+L does nothing in an English layout but it produces the keysym "@" in yours.

QEMU VNC servers, on the other hand, can't be treated as regular VNC servers. QEMU is a hardware emulator which emulates everything, including the keyboard. In this regular VNC implementation, when connecting to the QEMU server, sending a "@" keysym would cause QEMU to transform it back to the actual key strokes that generated it. In en_us layout, it would be Shift+2. But you haven't pressed shift+2, you pressed AltGr+L. But there's no way QEMU knows your layout beforehand. That leads to a lot of problems, as you might have noticed.

Then it comes the QEMU extension, the work I've done in noVNC. It does not rely on sending keysyms, but the actual physical key pressed. It's a different code from the regular noVNC keyboard handler. In your case for example, AltGr+L will be sent as 'key of row 6 column 5' + 'key of row 4 column 10'. This is what the QEMU VNC server receives and sends to the VM. So, in order to get the "@" in the VM, the VM must match your own keyboard layout. Supposing that you didn't change the VM keyboard, AltGr+L in a en_us layout gives you no symbol.

So, in order to properly make the keyboard work in your case, the VM layout must match the "source" layout.

[danielhb]

"In this setup - should I set the the "client VM" keyboard layout to German as well (like it is set on my Mac)?"

Exactly. Not only "German" but the exact same layout you're using. In Ubuntu15 it's called "German-Austria-Macintosh"

[benbracha]

Thanks @danielhb - this really sheds more light on this, and improve my understanding.
I tried the set up the keyboard on the "client VM" as you suggested, using "sudo dpkg-reconfigure keyboard-configuration", and still couldn't get it work.

I you think this is not related to this issue, I would be happy to take this offline with you (email), as it can help me a lot.

I did notice this thing:

Even if I don't change my "client VM" keyboard and leave it as is (english), when I switch my mac between EN and DE keyboards and send "y" - the "client VM" behaves as expected - types "y" or "z" accordingly. According to your explanation, this shouldn't be possible unless I'll change the "client VM" keyboard to DE.. Maybe I'm missing something more basic with my setup?
For example - I assume your changes relates only the the server-side of noVNC, as I still use regular noVNC client from http://kanaka.github.io/noVNC/noVNC/vnc.html. Should I use your vnc.html?

[danielhb]

On 04/13/2016 11:20 AM, Ben Bracha wrote:

Thanks @danielhb https://github.com/danielhb - this really sheds
more light on this, and improve my understanding.
I tried the set up the keyboard on the "client VM" as you suggested,
using "sudo dpkg-reconfigure keyboard-configuration", and still
couldn't get it work.

I don't know what dpkg-reconfigure keyboard-configuration does. I would
rather set the keyboard manually. In Ubuntu 15 it's under 'Keyboard' ->
'Input sources'. Then select language and layout type. You can see a
drawing of the keyboard layout to ensure that it matches yours. You're
using Ubuntu 14 but I am guessing it is similar.

I did notice this thing:

Even if I don't change my "client VM" keyboard and leave it as is
(english), when I switch my mac between EN and DE keyboards and send
"y" - the "client VM" behaves as expected - types "y" or "z"
accordingly. According to your explanation, this shouldn't be possible
unless I'll change the "client VM" keyboard to DE.. Maybe I'm missing
something more basic with my setup?

The reason is that the QEMU VNC extension isn't enabled. Otherwise
pressing the 'Y' key (row 3 column 7 key) would always result in the "y"
symbol in the VM set with English layout. In fact, the extension doesn't
care about the client keyboard layout. It's the VM layout that counts.

Another thing that really counts is the browser version. You said that
you're using latest Chrome. Which version is that in your case? You can
see the version by issuing chrome://version in the address bar.

For example - I assume your changes relates only the the server-side
of noVNC, as I still use regular noVNC client from
http://kanaka.github.io/noVNC/noVNC/vnc.html. Should I use your vnc.html?

It's actually the opposite. My changes are on the client side, in the
javascript files.

If you want to be sure that you're running the right code, just use the
code from https://github.com/danielhb/noVNC . It's the noVNC master code

• my patches and you use it the same way.

I you think this is not related to this issue, I would be happy to
take this offline with you (email), as it can help me a lot.

I am interested to see your case working or at least figured because
there's always a chance that I've missed some detail, perhaps something
specific of Mac.

Feel free to email me at [email protected] if you want to take the
discussion offline.

[benbracha]

Thanks, now I see it work (I didn't use your noVNC client side, only the server side, as I thought your changes were on the server side..).
After setting the "client VM" to match my exact Mac layout (BTW - dpkg-reconfigure keyboard-configuration just let you do that in the Ubuntu terminal, as I don't have a desktop there..), I see the "@" sent when using AltGr+l.
Nice!

It seems like dead-keys are working as well.. at least I tried the AltGr+n that generates a "~" and AltGr+u that generates quotes, although I couldn't get them to add "accent"
For example: doing AltGr+n+n should do: "ñ" But I got ~n (it may be that Ubuntu terminal can't print it?)

[danielhb]

Nice to see it worked!

Ubuntu terminal doesn't print all the available characters. Try opening
'gedit' and test it there

On 04/13/2016 11:57 AM, Ben Bracha wrote:

Thanks, now I see it work (I didn't use your noVNC client side, only
the server side, as I thought your changes were on the server side..).
After setting the "client VM" to match my exact Mac layout (BTW -
dpkg-reconfigure keyboard-configuration just let you do that in the
Ubuntu terminal, as I don't have a desktop there..), I see the "@"
sent when using AltGr+l.
Nice!

It seems like dead-keys are working as well.. at least I tried the
AltGr+n that generates a "~" and AltGr+u that generates quotes,
although I couldn't get them to add "accent"
For example: doing AltGr+n+n should do: "ñ" But I got ~n (it may be
that Ubuntu terminal can't print it?)

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#21 (comment)

[dkelson]

@danielhb so in libvirt, with your patch, there should be no keymap set, correct?

For example:

<graphics type='vnc' listen='0.0.0.0' port='-1' autoport='yes'/>

instead of

<graphics type='vnc' listen='0.0.0.0' port='-1' autoport='yes' keymap='fr-ca'/>

See also: https://www.redhat.com/archives/libvir-list/2009-May/msg00234.html

[danielhb]

@dkelson yes, you're right. No keymap set in libvirt or QEMU, following the recommendation in the link you sent:

"Our recommendation is to never set the '-k' /keymap option at all these
days. Recent QEMU / KVM and GTK-VNC releases support a VNC extension for
sending raw keycodes, instead of localized keysyms. Thus is best to leave
off the keymap in the config, and just configure it inside the guest OS."

[brenard]

@danielhb : it's work fine for me ! I port your patched version of noVNC in my webvirtcloud fork to try it : https://github.com/brenard/webvirtcloud/tree/html5_console

I make my tests with Chromium Version 49.0.2623.108 Built on 8.3, on multiple libvirt/KVM VNC instances without specific keymap configure in libvirt XML configuration, with french keymap configure in VM and french keyboard (azerty).

[danielhb]

I'm glad it worked @brenard ! Thanks for testing it!

[danielhb]

For anyone interested, I've backported the my implementation of the QEMU extension that fixes this bug to the latest noVNC release, 0.5.1. The code can be fetched here:

https://github.com/danielhb/noVNC/tree/0.5.1_qemu_extension

Same conditions apply (recent Firefox/Chrome/Opera browsers, must connect to a QEMU VNC server).

[neufeind]

Article by Daniel with further defails and background: https://www.ibm.com/developerworks/library/wa-support-multiple-keyboard-layouts-in-web-based-vnc-apps-trs/index.html