Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix IM and virtual keyboard handling #1169

Merged
merged 2 commits into from
Oct 24, 2023
Merged

Conversation

kchibisov
Copy link
Member

@kchibisov kchibisov commented Oct 17, 2023

The series is split into 2 commits, the first one addresses the virtual keyboard issues I've faced and the second tries to solve IM issues, which were left on the table. Each commit has a description on what it tried to fix.

--

This is by far the ugliest thing I ever wrote, but to make it nice I'd need to delete all the input handling in smithay, which I don't have time to do.

Comment on lines +94 to +163
pub(crate) popup_geometry_callback: fn(&D, &WlSurface) -> Rectangle<i32, Logical>,
pub(crate) new_popup: fn(&mut D, PopupSurface),
pub(crate) dismiss_popup: fn(&mut D, PopupSurface),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Otherwise you force IME into the users.

@kchibisov
Copy link
Member Author

The other question is that the parent_geometry doesn't really work, at least the way it's used.

The issue is that parent geometry could change without client losing focus and the popup needs a way for it to be updated. The most reliable solution for popup to actually ask geometry during space renderer each time from the compositor, so it ensures that everything is up to date.

How something like that should be approached? I'm pretty sure it's not the first time something like that happens and some other elements may also need to query for the parent geometry all the time.

@kchibisov kchibisov marked this pull request as draft October 18, 2023 06:16
@kchibisov
Copy link
Member Author

Converting to draft since it needs more tuning to actually work, and probably will have more to come.

I'd still like to know what to do with geometry.

@codecov-commenter
Copy link

codecov-commenter commented Oct 18, 2023

Codecov Report

Attention: 272 lines in your changes are missing coverage. Please review.

Comparison is base (7e2e3c4) 22.02% compared to head (3c33822) 21.94%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1169      +/-   ##
==========================================
- Coverage   22.02%   21.94%   -0.08%     
==========================================
  Files         154      154              
  Lines       23972    24053      +81     
==========================================
  Hits         5279     5279              
- Misses      18693    18774      +81     
Flag Coverage Δ
wlcs-buffer 18.99% <0.00%> (-0.07%) ⬇️
wlcs-core 18.61% <0.00%> (-0.07%) ⬇️
wlcs-output 7.74% <0.00%> (-0.03%) ⬇️
wlcs-pointer-input 20.80% <0.00%> (-0.08%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
src/input/keyboard/mod.rs 12.69% <ø> (ø)
src/desktop/wayland/popup/manager.rs 8.02% <0.00%> (ø)
src/wayland/virtual_keyboard/mod.rs 39.70% <0.00%> (-0.60%) ⬇️
src/desktop/wayland/popup/mod.rs 0.00% <0.00%> (ø)
src/wayland/input_method/mod.rs 24.67% <0.00%> (-1.01%) ⬇️
src/wayland/text_input/mod.rs 17.46% <0.00%> (-0.58%) ⬇️
anvil/src/state.rs 36.74% <0.00%> (-0.18%) ⬇️
...wayland/input_method/input_method_keyboard_grab.rs 0.00% <0.00%> (ø)
...wayland/input_method/input_method_popup_surface.rs 0.00% <0.00%> (ø)
src/wayland/seat/keyboard.rs 7.87% <0.00%> (-0.53%) ⬇️
... and 3 more

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@kchibisov kchibisov marked this pull request as ready for review October 18, 2023 10:48
@kchibisov kchibisov force-pushed the fix-ime-popup branch 2 times, most recently from aa67f5e to 9e86a1d Compare October 18, 2023 12:24
@kchibisov kchibisov changed the title Fix IME popup handling Fix IM and virtual keyboard handling Oct 18, 2023
Copy link
Member

@Drakulix Drakulix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

#1156 was just merged and causes a slight conflict.

src/wayland/virtual_keyboard/virtual_keyboard_handle.rs Outdated Show resolved Hide resolved
@rano-oss
Copy link
Contributor

Sorry I am a bit late to the party, it can probably be merged the way it is for now I guess?
Placement of popup surface is incorrect as it should be the compositor that decides where to place the surface.
Text input only communicates placement of the text cursor. Technically my solution is also incorrect, I just made it correct in 90% of cases, but I guess we I should start looking into how to handle collision, see picture:
image
There is also a weird bug sometimes were no text is written and keys pressed go for menu items, not sure why yet, maybe a modifier is stuck?
The last thing I found is that when I open and close an ime a couple of times, the keyboard spits out rubbish.
I haven't had the time to chase down why this happens

@rano-oss
Copy link
Contributor

Do we really need to have an implementation of dismiss_popup? Can we not just call "PopupManager::dismiss_popup" in the ime code?

@kchibisov
Copy link
Member Author

Placement of popup surface is incorrect as it should be the compositor that decides where to place the surface.
Text input only communicates placement of the text cursor. Technically my solution is also incorrect, I just made it correct in 90% of cases, but I guess we I should start looking into how to handle collision, see picture:

I expose all the state needed to handle collisions, there's a protected_region, which downstream should deal with.

There is also a weird bug sometimes were no text is written and keys pressed go for menu items, not sure why yet, maybe a modifier is stuck?

It could be a commit logic discarding, basically or how we count commits. There could be a modifier stuck, but it would mean that the smithay input stuff is bugged. Also, IME works during that you can like re-trigger it just fine...

The last thing I found is that when I open and close an ime a couple of times, the keyboard spits out rubbish.
I haven't had the time to chase down why this happens

Given that we just pass things around it could be your IME doing so for whatever reason. I don't have that issue with my IME, for example. This is also has nothing to do with the way things work on this PR in general, since it just uses the smithay infra for that and what I did is basically moved the whole logic to bring up/down IME into the activate stuff.

@kchibisov
Copy link
Member Author

Do we really need to have an implementation of dismiss_popup? Can we not just call "PopupManager::dismiss_popup" in the ime code?

Yes, because PopupManager is not mandatory and could be excluded.

@rano-oss
Copy link
Contributor

I used fcitx5, which IME are you using?

@rano-oss
Copy link
Contributor

Do we really need to have an implementation of dismiss_popup? Can we not just call "PopupManager::dismiss_popup" in the ime code?

Yes, because PopupManager is not mandatory and could be excluded.

but would it not be sufficient to check if we have a popup, then just call it?

@kchibisov
Copy link
Member Author

but would it not be sufficient to check if we have a popup, then just call it?

Call what? We don't have PopupManager in a wayland module, it's a desktop module which could be excluded. Users also may not even use the PopupManager.

@rano-oss
Copy link
Contributor

Placement of popup surface is incorrect as it should be the compositor that decides where to place the surface.
Text input only communicates placement of the text cursor. Technically my solution is also incorrect, I just made it correct in 90% of cases, but I guess we I should start looking into how to handle collision, see picture:

I expose all the state needed to handle collisions, there's a protected_region, which downstream should deal with.

Could we at least make a correct example in anvil then? Like just the 90% correct one or something?

There is also a weird bug sometimes were no text is written and keys pressed go for menu items, not sure why yet, maybe a modifier is stuck?

It could be a commit logic discarding, basically or how we count commits. There could be a modifier stuck, but it would mean that the smithay input stuff is bugged. Also, IME works during that you can like re-trigger it just fine...

Well, I guess I should just create an issue for this then

@rano-oss
Copy link
Contributor

but would it not be sufficient to check if we have a popup, then just call it?

Call what? We don't have PopupManager in a wayland module, it's a desktop module which could be excluded. Users also may not even use the PopupManager.

Alright, in that case, could we just not check if the popup manager is used? Like we do not really have an alternative for it right now anyway?

The virtual keyboard was always sending modifiers resulting in erronious
client breakages. The keymap was also read via regular file routines
instead of mmap'ing.
This commit fixes multiple issues identified during using the IM:

1. Make IM activation/deactivation based on text-input::{enable/disable}
   and keyboard focus leave, instead of just keyboard focus changes.

2. Send `text_input::enter` on text-input creation, since some bind
   global after getting focus.

3. Don't activate IME for clients which don't have text_input
   instance, otherwise keyboard input will be disabled in general due to
   grabbing.

4. Add `new_popup` and `dismiss_popup` to update IM popup in response
   to IM text_input::{enable/disable} events.

5. Use text-input coordinates for input-method popup positioning
   IM popup positioning. The width and height mark the region you
   shouldn't obscure.

6. Fix serial handling to match input-method-v2 and text-input-v3
   specifications. input-method-v2 also requires counting amount
   of done events and using it to discard commit when it doesn't
   match text-input. text-input-v3 just wants serial with value
   equal to the number of commits being sent for the client.
@kchibisov
Copy link
Member Author

Could we at least make a correct example in anvil then? Like just the 90% correct one or something?

I don't want to do that myself, but feel free. Only cosmic can probably handle popup re constaints logic correctly.

Alright, in that case, could we just not check if the popup manager is used? Like we do not really have an alternative for it right now anyway?

I just tell downstream that popup should be dissmissed, how it's done is up to the downstream. Anvil uses PopupManager for example, catacomb doesn't use it. I don't know what you really want from me here, sorry. So propose a patch.

@kchibisov kchibisov requested a review from Drakulix October 24, 2023 11:21
@kchibisov
Copy link
Member Author

I used fcitx5, which IME are you using?

https://github.com/tadeokondrak/anthywl

@rano-oss
Copy link
Contributor

Alright I will try and see if I can figure something out on my own then. Anyway great work 🙂 , focus seems to work a lot better now at least.

@Drakulix Drakulix merged commit 9cefe4f into Smithay:master Oct 24, 2023
13 checks passed
@kchibisov kchibisov deleted the fix-ime-popup branch October 29, 2023 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants