Evil breaks visual-line-mode

[TheBB]

Originally reported by: Titus von der Malsburg (Bitbucket: tmalsburg, GitHub: tmalsburg)

---

In Evil, line operations always affect buffer lines instead of visual lines. This behavior is consistent with Vim. While I think that consistency is important, the reason why people use Evil instead of Vim is that they would like to take advantage of features provided by Emacs. One such feature is visual-line-mode. The purpose of visual-line-mode is:

1. automatic, visual word wrapping without changing the buffer content,
2. making visual lines behave as if they were buffer lines.

In other words, the fact that visual lines are not buffer lines is supposed to be completely transparent to the user. Evil breaks visual-line-mode because it operates on buffer lines regardless of whether visual-line-mode is on or not.

Visual-line-mode may not be important for people who mainly write code, because in code, you want to manage line breaks yourself. However, when writing prose, visual-line-mode can be essential. One reason why I left Vim was that I was sick and tired of doing gqap hundreds of times a day. Commands that I would like to see working with visual lines instead of with buffer lines are of course dd and yy, but also I and A, etc.

What are possible solutions? I guess, one possibility would be to change all kinds of Evil commands to deal with visual lines instead buffer lines when visual-line-mode is switched on. This is what I currently implemented for some important commands. However, this leads to a lot of changes and messy code. Another possibility may be to let Evil always operate on visual lines irrespective of whether visual-line-mode is switched on or not. The obvious draw-back is that Evil's behavior would then be inconsistent with Vim's when visual-line-mode is turned off.

A more elegant and more correct solution might therefore be the following: The idea is to enclose line-operations in a function that does the following:

1. Word wrap the current buffer line using fill-region.
2. Execute the line operation on the newly created buffer line (dd, yy, I, A, ...).
3. "Unwrap" the region corresponding to the original line by removing the line breaks introduced by fill-region.

For this to work properly, fill-column has to be set to the current width of the window (in characters). Filling should then not change the appearance of the text but simply replace visual line breaks by real line breaks (i.e. "\n" characters). So the user shouldn't notice what is going on. Here's some code to illustrate what I mean:

(defmacro fill-unfill (&rest body)
 `(lambda (&rest args)
    (interactive "p")
    (let ((fill-column-orig fill-column)
          (fill-column (- (window-width) 2))
          (start (save-excursion (beginning-of-line) (point)))
          (end (save-excursion (end-of-line) (point)))
          (end-of-buffer (point-max)))
      (if visual-line-mode
        (save-excursion (fill-region start end nil 1)))
      (apply ,@body args)
      (if visual-line-mode
        (let ((fill-column (point-max))
              ; Account for additional newline chars in the region:
              (new-end (+ end (- (point-max) end-of-buffer))))
          (save-excursion (fill-region start end nil 1)))))))

(define-key evil-normal-state-map "I" (fill-unfill 'evil-insert-line))
(define-key evil-normal-state-map "A" (fill-unfill 'evil-append-line))

This code may break some things like macros (q and friends) but perhaps this can be dealt with. In case that this solution can be generalized to handle other operations like yy and dd as well, this may be a nice approach because it doesn't require changes in too many places. The whole issue is rather beautifully factored out from the actual line operations. The line operations themselves would be agnostic to the whole issue and would not have to deal with visual lines at all.

Frank considered a similar solution for VimMode. See https://bitbucket.org/lyro/vim-mode/issue/33/visual-line-mode

I think that having Evil play nicely with visual-line-mode would be a major improvement.

---

• Bitbucket: https://bitbucket.org/lyro/evil/issue/188

[TheBB]

Original comment by Michael Markert (Bitbucket: cofi, GitHub: cofi):

---

I think your code could be a lot more easier considering there is beginning-of-visual-line and end-of-visual-line.

If I understand Frank's thoughts at the vim-mode issue correctly it's less about how to select the text but how to deal with it conceptually.
A yanked visual line is no line at all because it does not end with a new-line .. that line of thought.

[TheBB]

Original comment by Titus von der Malsburg (Bitbucket: tmalsburg, GitHub: tmalsburg):

---

The point of my code is that it attempts to solve this conceptual issue once and for all by equating visual lines with real lines. For instance, if you yy, then you have copied a real line (including a new line character). If you subsequently P, my code would first wrap, then insert the real line, and then unwrap, which may of course cause a word to slip to another line. This occasional slipping of words is the only point where the illusion (visual lines == real lines) breaks, but that's a feature not a bug.

From the perspective of the user, the fact that visual lines are not real lines should not matter at all, because it's the very purpose of visual-line-mode to give visual lines the semantics of real lines. If you switch off visual-line-mode, everything should of course behave as it usually does in Vim. That means that the above code would have to be amended to do this wrapping and unwrapping business only if visual-line-mode is switched on. (Edit: duh, I just see that the code already does that.) I hope that clarifies my idea, sorry if the first post was confusing.

[TheBB]

Original comment by Titus von der Malsburg (Bitbucket: tmalsburg, GitHub: tmalsburg):

---

Just edited the text a bit.

[TheBB]

Original comment by Titus von der Malsburg (Bitbucket: tmalsburg, GitHub: tmalsburg):

---

Minor formatting.

[TheBB]

Original comment by Titus von der Malsburg (Bitbucket: tmalsburg, GitHub: tmalsburg):

---

Markdown, arg.

[TheBB]

Original comment by Titus von der Malsburg (Bitbucket: tmalsburg, GitHub: tmalsburg):

---

Hopefully final edit. Sorry, bitbucket changed the markup and there's no preview facility.

[TheBB]

Original comment by Christoph Paulik (Bitbucket: cpaulik, GitHub: cpaulik):

---

Is there any status or decision on this? I'm just curious if this will be implemented.

[TheBB]

Original comment by Frank Fischer (Bitbucket: lyro, GitHub: lyro):

---

Nothing new. AFAIK nobody ever worked on this.

[TheBB]

Original comment by Titus von der Malsburg (Bitbucket: tmalsburg, GitHub: tmalsburg):

---

minor rephrasing

[TheBB]

Original comment by sampablokuper (Bitbucket: sampablokuper, GitHub: sampablokuper):

---

@tmalsburg, thank you for reporting this issue, which has been bugging me for a while! Thank you also for your draft patch. It would be great for this to be fixed, along the lines you describe.

As a possible alternative to fixing this directly within Evil, have you considered creating an Emacs plugin? (Maybe you could call it "evil-intuitive-visual-lines" or suchlike?)

Regardless of approach, all the better if it the fix is done in a way that allows it to be easily integrated into Spacemacs, perhaps even as part of the default Spacemacs install.

(I'm not yet sufficiently competent at Emacs Lisp to do this work myself, sorry!)

[TheBB]

Closing this in light of #1173 and #1181. Please let me know if evil-respect-visual-line-mode is still unreliable in some way.

[tmalsburg]

evil-repect-visual-line-mode looks promising but it is currently woefully incomplete and partly broken. In the current state, I'm afraid, the implementation behaves too erratically to be useful. I request that this issue is reopened.

Here are some things that appear to work after some short tests (with evil from MELPA and up-to-date Emacs from development master branch):

• Motion to previous line with k and to next line with j in normal mode.
• Motion to beginning of line 0, motion to end of line $, motion to first non-whitespace in line ^ in normal mode. Consequently, these do work as expected: d0, d$, d$, D, c$, …
• I insert at beginning of line (normal mode).
• A insert at end of line (normal mode).

However, a whole bunch of other things do not work at all or incorrectly:

• Motion with <up> and <down> in normal mode. Goes to previous/next buffer line, not previous/next visual line.
• dd kill current line. Kills from beginning of visual line to end of buffer line. Should kill from beginning of visual line to end of visual line. (Can't reproduce reliably. Will investigate.)
• yy yank current line. Kills from beginning of visual line to end of buffer line. Should kill from beginning of visual line to end of visual line. (Can't reproduce reliably. Will investigate.)
• D should be a synonym for d$ but it deletes the rest of the buffer line not the rest of the visual line.
• With yanked line, P inserts before buffer line not before visual line. Likewise p inserts yanked line after buffer line not after visual line.
• Also both P and p insert buffer lines even when visual lines were yanked/killed. On a line in the middle of a paragraph,dd followed by p should get us back to were we started but it doesn't.
• :g/pattern/d kills matching buffer lines, not matching visual lines.

Both lists above are likely incomplete. One would have to go through the vim manual for a complete list of operations that are affected by this.

Regarding implementation: I had a quick look at the code behind evil-repect-visual-line-mode. It seems to address the issue on a case-by-case basis which means that a complete implementation might be a lot of work. It might be preferable to find a solution that addresses the visual/buffer line equivalence on a deeper level such that individual editing operations can be left untouched and just do their thing as usual. If such a solution is viable, this may not just provide us with a once-and-for all complete solution but would also produce code that is much easier to maintain. In my initial description, I sketched such a solution as a proof-of-concept but there may of course be better ways to do this.

[tmalsburg]

@TheBB, I apologize for not responding to your comment from 2016. I must have overlooked the notification. Responses below.

As a possible alternative to fixing this directly within Evil, have you considered creating an Emacs plugin? (Maybe you could call it "evil-intuitive-visual-lines" or suchlike?)

Perhaps an implementation as a package is possible but it might be complicated and not the most natural solution as this package would have to modify core evil behaviour not just add behaviour. Also, since we have evil-respect-visual-line-mode now (introduced in 2017) it might be best to just build on that.

Regardless of approach, all the better if it the fix is done in a way that allows it to be easily integrated into Spacemacs, perhaps even as part of the default Spacemacs install.

I agree but should say that I'm not using Spacemacs myself and I'm not really familiar with it.

[TheBB]

@tmalsburg This is a migrated issue from Bitbucket. The actual authors of the migrated comments are in the text, so that's not mine.

I'll reopen.

[tmalsburg]

Thank you (also for the clarification).

[jeyj0]

Hello, I am very much interested in this, as this is still one of the main reasons making me not entirely happy with emacs. However, I don't just want to complain here, but rather offer my assistance in resolving this issue, if there's interest. I don't have any experience in elisp apart from writing a 104 line emacs config right now, so I could use some assistance (I've used Doom until recently, and am just starting to create my own config from scratch).

I'd also be interested to hear other people's opinions on what they think the behavior of dj or dk should be. Right now they behave like dgj and dgk in vim. However, I expected them treat visual lines as buffer lines when evil-respect-visual-line-mode is turned on (causing two visual lines to be deleted, instead of the text that happens to be between original cursor position and new cursor position).

[muello]

@tmalsburg Something else for the 'not working correctly' list:

• zt and zb: evil-scroll-line-to-top and -bottom. When point is near the end of a long (visual) paragraph, zt can actually have the effect of scrolling the text downwards instead of up. (Typing zt in the first empty line after the paragraph then works as it should.)

[tgbugs]

I have a related issue with evil-respect-visual-line-mode. The behavior that I would like by default is for vertical navigation j and k to follow visual lines so that navigation behavior is always consistent with the visualization of the file, otherwise you wind up having to navigate 100s of "lines" using h and l which is an exercise in eternal frustration, especially if you happen to accidentally move away from the line. evil-respect-visual-line-mode provides exact this behavior.

Unfortunately evil-respect-visual-line-mode also applies to all text editing operations, which is equally frustrating because now I cannot use dd to delete a logical line, which is all that dd has ever and should ever do (for my use case). If you have a single line of code that wrapped, suddenly you can chop it in half and your file now has malformed garbage sitting in it.

At the moment there is a nasty hack that can be used to partially obtain this behavior which is to (setq evil-respect-visual-line-mode t) at startup, and then (setq evil-respect-visual-line-mode nil) it after loading evil. This preserves the navigation behavior so that j and k respect visual lines, but allows dd and yy to operate on logical lines.

However, this is exploiting some strange persistent state and is completely unreliable. Also, it does not fix the behavior of operations such as $ and ^ which still operate over the visual line instead of the logical line. This is a case where different users are going to have different expectations and may want different behavior in different modes. Thus it seems that more configuration is needed.

[tmalsburg]

Unfortunately evil-respect-visual-line-mode also applies to all text editing operations, which is equally frustrating because now I cannot use dd to delete a logical line, which is all that dd has ever and should ever do (for my use case). If you have a single line of code that wrapped, suddenly you can chop it in half and your file now has malformed garbage sitting in it.

I understand your point and use case, but keep in mind that visual-line-mode is primarily attractive for writing text not so much for code. Personally, I'd not use visual-line-mode in any code buffers because there I want to manage line breaks manually and I need the buffer to show exactly what's in the file without any extra virtual line breaks. For instance, visual-line-mode could totally mess up the apparent syntax of Python source files. However, if you write text, it's perfectly reasonable for dd to delete a visual line, not a buffer line.

More generally I think it's best to implement visual-line-mode in evil such that all line operations act on visual lines instead of buffer lines. This way visual-line-mode behaves perfectly predictably. If visual-line-mode mode would change only some line operations to visual lines but not others, that will be confusing, difficult to learn, and there will be potential for endless debates about which operations should act on visual and which on buffer lines. Doesn't seem desirable. Having said that, it would also be nice if it was possible for users with different preferences to choose and pick.

[tgbugs]

I understand your point and use case, but keep in mind that visual-line-mode is primarily attractive for writing text not so much for code.

The main place that I run into these issues is in org-mode buffers where I have text and code at the same time. Most of the time I manually wrap my blocks, but I can't always, especially when there are urls that are quite a bit longer than my fill column setting.

There are also cases where I need retain a single line e.g. github issues and various other web text fields, where manual line wrapping has nasty interactions with the automated rendering algorithms those fields employ.

[hraban]

@tgbugs what about this

(defun evil-redo-no-respect-visual-line ()
  (interactive)
  (undo)
  (let ((evil-respect-visual-line-mode nil))
    (evil-repeat 1)))

Bind it to a key, to quickly redo your last evil command on a logical line instead of a visual line.