Current master does not properly highlight Rust source file

[mouse07410]

What version of bat are you using?

bat --version
bat 0.15.0

Describe the bug you encountered:
I'm trying to display/highlight the following Rust source file const-generics.rs:

#![feature(const_generics)]

mod state {
    pub struct State<const S: &'static str> {
        total: u32,
    }

    impl State<"init"> {
        pub fn new() -> Self {
            Self { total: 0 }
        }

        pub fn accumulate(self) -> State<"accumulate"> {
            State { total: self.total }
        }
    }

    impl State<"accumulate"> {
        pub fn add(&mut self, add: u32) {
            self.total += add
        }

        pub fn freeze(self) -> State<"freeze"> {
            State { total: self.total }
        }
    }

    impl State<"freeze"> {
        pub fn unwrap(self) -> u32 {
            self.total
        }
    }
}

use state::State;

fn main() {
    let init = State::new();
    // init.add(120); // nope

    let mut acc = init.accumulate();
    acc.add(10);
    acc.add(20);

    let frozen = acc.freeze();
    // frozen.add(120); // nope
    let val = frozen.unwrap();

    println!("{}", val);
}

What is happening - it stops syntax-highlighting after the first impl<> is processed:

Describe what you expected to happen?
The entire source file highlighted appropriately. Same way, as, e.g., the above source is highlighted by this web page.

How did you install bat?

$ cargo install bat

---

system
------

**$ uname -srm**
Darwin 19.4.0 x86_64  

**$ sw_vers**
ProductName:	Mac OS X  
ProductVersion:	10.15.4  
BuildVersion:	19E266  

bat
---

**$ bat --version**
bat 0.13.0  

**$ env**

bat_config
----------

bat_wrapper
-----------

No wrapper script.

bat_wrapper_function
--------------------

No wrapper function.

tool
----

**$ less --version**
less 487 (POSIX regular expressions)  

[keith-hall]

This is not a problem with bat (or syntect), but the Rust.sublime-syntax syntax definition being used - it currently doesn't correctly handle strings inside angle brackets. It may be worth logging it at https://github.com/sublimehq/Packages/issues/

[thomcc]

https://github.com/rust-lang/rust-enhanced might be better to use (although I don't know if it does any better here)

[mouse07410]

https://github.com/rust-lang/rust-enhanced might be better to use (although I don't know if it does any better here)

Probably it would. How can I try it with the Cargo-installed bat (not build from local repo)?

Also, whatever's highlighting the code on this web page, appears to do a better job. Perhaps it could be used...

[mouse07410]

@sharkdp can you switch to https://github.com/rust-lang/rust-enhanced ? Based on the opinion of the 'netizens, it is likely to give better results.

If you do switch (and I hope you do) - to get your updates, I should just reinstall bat via $ cargo install bat?

[sharkdp]

There is no need to reinstall/recompile bat if you want to try a new syntax. Simply follow the steps at https://github.com/sharkdp/bat#adding-new-syntaxes--language-definitions

[mouse07410]

Following your instructions above and installing the syntaxes - I can confirm that both work fine:

• Cloned https://github.com/sublimehq/Packages.git and applied clp is a faster cat with syntax highlighting than bat #2305 PR to it (worked fine);
• Downloaded RustEnhanced.sublime-syntax file from https://github.com/rust-lang/rust-enhanced repo (it also worked fine).

So, either of the two proposed fixes would work (tested).

[sharkdp]

Thank you for reporting this upstream (sublimehq/Packages#2323).

Given that you are trying to highlight Rust source code with a language feature (const generics) that is not stable and hasn't even finished the design phase (rust-lang/rust#44580), I think that this doesn't really constitute a bug.

In this case, I think we can just wait for sublimehq/Packages#2305 to be merged and close this?

[mouse07410]

Given that you are trying to highlight Rust source code with a language feature (const generics) that is not stable and hasn't even finished the design phase (rust-lang/rust#44580), I think that this doesn't really constitute a bug

I disagree - the problem seems unrelated to the unstable feature. The fact that you can remove several lines of code (reduce the sample) and still observe the problem seems to confirm it. And it is remedied by the current https://github.com/rust-lang/rust-enhanced

I think we can just wait for sublimehq/Packages#2305 to be merged and close this?

Since you gave me a good workaround (copying the relevant ...-syntax file and bat cache --build), I have no problem with this approach.

[sharkdp]

I disagree - the problem seems unrelated to the unstable feature. The fact that you can remove several lines of code (reduce the sample) and still observe the problem seems to confirm it.

How would the smaller example look like? The problem is apparently caused by the string within angle brackets. Without const generics, strings inside angle brackets wouldn't be allowed, right?

[mouse07410]

The problem is apparently caused by the string within angle brackets. Without const generics, strings inside angle brackets wouldn't be allowed, right?

You may be right. Anyway, this thing is supposed to syntax-highlight the source, not to play lint or clippy on it. IMHO, at least.

Regardless, the only thing I care about here is that the sources are properly and comprehensively highlighted. The workaround you suggested works perfectly for me, with a minimal inconvenience.

Therefore, IMHO it's OK to patch (or switch from Sublime to RustEnhanced) whenever you think the right time is, if at all. Since my machines all now incorporate the workaround - timeline doesn't matter (for me).

[mouse07410]

Current release 0.15 still fails:

Using syntax file from ExtendedRust still succeeds (with 0.15 release as well):

I repeat my suggestion: ditch Sublime syntax module for Rust, and pull one from ExtendedRust instead.

[sharkdp]

I repeat my suggestion: ditch Sublime syntax module for Rust, and pull one from ExtendedRust instead.

I deliberately decided against this for the reasons given above. If there is a Rust source code file that compiles on stable Rust, but is highlighted incorrectly by bat, I'm happy to reconsider.

If someone tells me that it is extremely likely that this unstable syntax will make it into stable Rust, I'm also OK with patching the current Rust syntax. I'd rather not have to maintain two Rust syntaxes, if possible.

[mouse07410]

I deliberately decided against this for the reasons given above...

Respectfully disagree. bat is not a syntax validator - that's what clippy is for (which, might I remind, exists in the "nightly" toolchain also).

bat is supposed to be usable, e.g., for the source written for "nightly" toolchain - and there is no good reason that I'm aware of why it shouldn't be so. Especially since it won't cost you a single extra line of code.

I'd rather not have to maintain two Rust syntaxes, if possible.

Of course! Nobody's suggesting that. What I recommend is that you ditch the current
Sublime-based syntax and replace it with ExtendedRust, which, as shown above, works perfectly well for both "stable" and "nightly".

[sharkdp]

Respectfully disagree. bat is not a syntax validator

I'm not here to argue, but I think that both tools (syntax validators and syntax highlighters) have a lot in common. You cannot expect a completely invalid syntax to be highlighted properly (whatever that would mean). Some syntaxes in bat even have a "invalid syntax" scope/color, further demonstrating that syntax validation and highlighting (ideally) is pretty much the same.

bat is supposed to be usable, e.g., for the source written for "nightly" toolchain - and there is no good reason that I'm aware of why it shouldn't be so.

We're not going to add syntaxes for every possible flavor of every possible programming language out there. That is not feasible (from a maintenance and performance point of view).

"Rust nightly" is not (yet) the official version of the language. Its syntax is prone to frequent changes. There is no way we could keep up with that.

Especially since it won't cost you a single extra line of code.

Adding new syntaxes adds a maintenance cost and a performance cost (#951). The current Rust syntax has served us well for years. Using a new syntax could lead to all kinds of surprises (e.g. #677 #914 #924 #750).

What I recommend is that you ditch the current Sublime-based syntax and replace it with ExtendedRust

Note that the Rust syntax is part of sublimehq/Packages, which we either include completely or not at all. But that's not the main point. Maintenance cost (see above) is.

Also, we have invested a lot of time in the customization features of bat. If users prefer another syntax, it's easy to replace it.

[mouse07410]

both tools (syntax validators and syntax highlighters) have a lot in common.

So? Highlighters are expected to, well, highlight the code even with errors in it.

You cannot expect a completely invalid syntax to be highlighted properly (whatever that would mean)

Let's not go down this rabbit hole. "Completely" invalid and such...

Note that the Rust syntax is part of sublimehq/Packages, which we either include completely or not at all. But that's not the main point. Maintenance cost (see above) is.

I'm not familiar enough with bat internals, but I thought that the whole idea of using somebody else's syntax packages was that somebody else maintains them (and incurs the cost), while bat merely pulls them in as-is.

Your point about "either include completely or not at all" is well-taken. That explains why my proposal may not be feasible.

Also, we have invested a lot of time in the customization features of bat. If users prefer another syntax, it's easy to replace it.

Yes, you're right. Thanks!