Indentation commands don't indent literal text in treesit modes

[antifuchs]

I have to add my voice to the chorus of issues (e.g. #73 and #80): I've not managed to create a snippet yet for rust or nix modes that indents its literal or prompted contents to a column other than 0.

I use rust-ts-mode and nix-ts-mode, with active eglot LSP connections, if that indicates anything

This rust snippet:

rust-mode rust-ts-mode
(test "#[test]" n > "fn " (p "test_name") "() {" n> (r> "body") > n "}")

In rust-ts-mode, entering "test" on column zero and M-x tempel-expand, results in:

#[test]
fn test_name() {
body
}

I've tried all sorts of combinations on that "body" prompt, using r>, plain >, n> before it or not, none of these things result in an indented text.

Similar with this nix snippet:

nix-mode nix-ts-mode
(darwinmod > "{ config, pkgs, lib, ... }:" > n
           "with lib;" > n
           "let" > n
           "cfg = config.boinkornet." (p "name" module-name) ";" > n
           "stdenv = pkgs.stdenv;" > n
           "in" > n
           "{" > n
           "options = {" > n
           "boinkornet." (s module-name) ".enable = mkOption {" > n
           "description = \"Enable " (s module-name) "\";" > n
           "default = stdenv.isDarwin;" > n
           "example = false;" > n
           "};" > n
           "};" n > n
           "config = mkIf (cfg.enable) {" > n
           r> > n
           "};" > n
           "}")

And again, "darwinmod" in nix-ts-mode, on column zero, results in:

{ config, pkgs, lib, ... }:
with lib;
let
cfg = config.boinkornet.name;
stdenv = pkgs.stdenv;
in
  {
options = {
boinkornet.name.enable = mkOption {
  description = "Enable name";
  default = stdenv.isDarwin;
  example = false;
};
};

config = mkIf (cfg.enable) {

};
}

...which is even weirder, because some parts are indented!

I tried doing the same thing tempel is doing, with M-: (progn (insert "\n") (tempel--protect (indent-according-to-mode))) - and that does result in the correct indentation. So something must be going on, but I'm not sure what.

Can you tell me what's happening there?

[antifuchs]

Thanks for converting this to a discussion, I didn't even see that there's that tab on the repo.

I've been experimenting with more variations of that snippet in rust-ts-mode, and none of them seem to indent the "bar" token:

(foo "{" n> "bar" n> "}")
(foo "{" n "bar" > n "}")
(foo "{" n> "bar" n> "}" >)
(foo "{" n "bar" n "}")

When invoked on column 0, all of these put "bar" on column zero (and you can probably tell that I was getting pretty desparate towards the end)...

However, this one works when invoked outside column 0, inside a block (that's the third from the top above):

(foo "{" n> "bar" n> "}" >)

on column zero:

{
bar
}

inside a block:

{
    // below is the expansion of foo on the base indent level:
    {
        bar
    }
}

[antifuchs]

Update: It looks like indent-according-to-mode doesn't work correctly at all with treesit modes. In rust-ts-mode, running M-: (progn (insert "{\n")(indent-according-to-mode)) at column zero results on (point) on column zero again, which is ... not ideal. Guess it's not tempel's fault that the major-mode is misbehaving like that.

[antifuchs]

So given that tempel needs can only do indentation on a line-by-line basis, while it's inserting characters (creating syntax errors along the way, which sometimes cause treesit modes to be unable to correctly indent), I think the only way to get indentation of column-0 based forms to work is to wrap them in "indent this region, please" code.

Here's what I have, but it feels really dirty (note that this doesn't use :post because that only runs when the template is finalized, and that could be quite some time on my enormous templates above):

rust-mode rust-ts-mode
(test (p (point) template-start t)
      "#[test]" > n "fn " (p "test_name") "() {" n
      > (p "test_body") n
      "}"
      (progn (indent-region template-start (point)) nil))

The (p (point) template-start t) saves point at the start of the template expansion in a variable that's available to the template evaluator; then when the template is fully inserted, the (progn (indent-region template-start (point)) nil) runs and indents the correctly-balanced expression.

I'm somewhat dissatisfied with this; would this warrant and :indent t flag on the template, instructing the tempel evaluator to do what I'm doing there, but with fewer words?

[antifuchs]

Think we can do even better, in fact:

(defun asf--tempel-indent-template (elt)
  (when (eq '>> (car-safe elt))
    (let ((template-start (gensym)))
     `(l (p (point) ,template-start t)
         ,@(cdr elt)
         (progn (indent-region ,template-start (point)) nil)))))

(add-to-list 'tempel-user-elements 'asf--tempel-indent-template)

This defines a template sigil >> that can be invoked like so:

rust-mode rust-ts-mode

(test (>>
       "#[test]" n "fn " (p "test_name") "() {" n
       (p "test_body") n
       "}"))

...which properly indents the inserted template, but THEN, when I edit the test_name element, text that comes after test_body gets deleted. I imagine that adjusting indentation of placeholders after they have been inserted doesn't work too well?

[minad]

I imagine that adjusting indentation of placeholders after they have been inserted doesn't work too well?

Very well. If you can get this to work, >> would be a great example for a user element which should also be documented in the README. Or maybe it makes even sense to provide this element out of the box.

[antifuchs]

Hrm, yeah, so it seems that two things don't interact nicely with each other:

• The indent-region call causes the placeholder field overlay to expand for no discernable reason. What's worse, it expands both to the left (where spaces would be inserted) and to the right (so that it grows to include the literal text inserted after the placeholder.
• When editing the placeholder in that state, the tempel--synchronize-replace function runs (seemingly redundantly) and deletes the text that comes after the overlay (even if the overlay didn't expand to the length to cover the text that the function deletes).

I'm not completely sure how to go about debugging this. Maybe I'll have to introduce a new variable to the insert-in-front-hooks and insert-behind-hooks overlay function to move the overlay when it gets reindented?

[antifuchs]

Ah, I sent the PR in already; that has a bit of justification for the internals changes: One reason that this needs internal changes is that inserting text before the overlay causes the overlay to "extend left", causing the placeholder to look out of place (and start at the beginning of the indented region). The other is that indenting placeholders causes them to get cleared because moving them via indentation counts as an edit.

I think it's fine to not add the >> operator to the language; but in that case, tempel probably should add something that user code can use to opt into these workarounds.

[minad]

To take a step back - do I understand correctly that this problem is only related to treesit modes? #160 (comment)

Is it possible to fix it there? I don't recall but I may have already talked with the treesit maintainer about this indentation problem.

[minad]

One reason that this needs internal changes is that inserting text before the overlay causes the overlay to "extend left"

Maybe the extension behavior of overlays could be changed, or at least try to handle the case when whitespace is inserted via indentation?

The other is that indenting placeholders causes them to get cleared because moving them via indentation counts as an edit.

Could indentation be detected somehow and handled generally?

Maybe we could get a more general solution enhancing the existing hooks?

[antifuchs]

To take a step back - do I understand correctly that this problem is only related to treesit modes?

The problem with line-by-line indentation happens most easily in treesit modes (but other modes that are sensitive to paired braces could be affected also).

The issue where re-indenting a templated chunk of text causes placeholders to go wrong is universal: The reproducer above works in emacs-lisp-mode (definitely not a treesit one (-:). There, placeholder fields that get reindented have their extent altered such that they enclose text that follows (e.g. braces) and editing those fields clears that falsely-included text.

[minad]

The reproducer above works in emacs-lisp-mode (definitely not a treesit one (-:).

Yes, but the reproducer artificially uses indent-region and uses custom user elements. I use normally written templates in Elisp mode and other modes with the normal operators >, n> and r>. They don't have any problems.

The issue where re-indenting a templated chunk of text causes placeholders to go wrong is universal:

So can we solve this problem at the root, and not only for a newly added >> operator, which even requires a special inhibit variable?

Let me explain why my motivation to add workarounds is limited. First, I think problems are always better solved at the right place to improve Emacs overall. So far this approach has worked quite well and numerous bugs got fixed at the right place. Second, your PR adds more special casing and edge cases - code which I have to understand, review and which I will maintain in the end. This costs me time and energy. Such requests come in quite regularly for my packages, while support is not sufficiently increasing to dedicate even more time.

[DivvyCr]

I stumbled upon this thread while debugging a different indentation issue (which I solved by tinkering with my template definition).

Anyway, I got curious and found what I believe to be the core issue – the variable treesit-simple-indent-rules, or rather, the mode-specific variables LANG-ts-mode--indent-rules. In short, it's a list of (MATCHER ANCHOR OFFSET), where offset dictates whether the indentation level should change and anchor specifies the relative column (as far as I understand). For Rust, it begins with:

(defvar rust-ts-mode--indent-rules
  `((rust
     ((parent-is "source_file") column-0 0) ;; <== Curious ==========
     ((node-is ")") parent-bol 0)
     ((node-is "]") parent-bol 0)
     ((node-is "}") (and parent parent-bol) 0)
     ((and (parent-is "comment") c-ts-common-looking-at-star)
      c-ts-common-comment-start-after-first-star -1)
     ((parent-is "comment") prev-adaptive-prefix 0)
     ((parent-is "arguments") parent-bol rust-ts-mode-indent-offset)
     ...
     ((parent-is "block") parent-bol rust-ts-mode-indent-offset)
     ...
)

Immediately, we see that at the top-level of the AST the anchor is column zero and offset is zero. As a result, everything is reset to column zero (by tree sitter). Changing the offset to something like rust-ts-mode-indent-offset resulted in all lines being shifted one indent's worth (which makes intuitive sense).

Next, considering that at some point the indentation level has to change, I examined the AST with malformed code blocks:

AST                       PROGRAM

(source_file              {
 (ERROR {))

AST                       PROGRAM

(ERROR {                  {
 (expression_statement      {
  (block { })))           }

Notice that in the second case the error is relegated to the top-level (as the top-most brace is now unmatched), while the 'inside' can be an indent-able structure (wrt. to the indent rules).

Hence, to solve the core of this issue, you would have to either:

• write a custom indentation rule (either at the tree sitter level or maybe above it), or
• modify how the AST handles malformed code blocks at the top-level.

Of course, we are using Emacs, so we can possibly tackle this at a higher level, such as:

• change indent-line-function at a relevant point, temporarily overriding the default treesit-indent (in tree sitter modes);
  the issue is in deciding when to change it. Arguably, this reduces to an indentation rule anyway.

I agree with minad that modifying Tempel to indent the whole template at any point is not the best idea, mainly because that will introduce a heap of other issues. Having said that, a possible middle ground could be for Tempel to track the first and last line of the template and also provide a pre and/or post template insertion hook to allow users to hack away at it. I am not sure how easy it would be to track the first and last lines, but this is the best I could come up with at the moment.

I think any of the listed solutions are much safer than any modification to Tempel could do.