Scrolling rendered aka "CLI" mode, ideally a replacement for direct mode

[dankamongmen]

#1823 gave us the ability to scroll in rendered mode, by setting the default plane to scroll. Together with retrieving the cursor position at startup and preserving it as a logical scroll, this gives us the ability to write scrolling CLI-like (as opposed to TUI-like) applications in rendered mode. This means the full power of rendered mode can be brought to bear, and ideally that Direct Mode might one day be cast off into the lands of backwards compatibility (maybe not -- scrolling rendered mode still doesn't want other sources emitting output).

The first major problem I can see is stderr. Anything dropped onto stderr is still going to screw us, and Notcurses can do that itself on errors (or with a high loglevel, because it's Tuesday). But maybe we just say, yep, as always, you can't interleave other output freely. idk

What else? #1837 hasn't been particularly reassuring, though any such new technique will need a few bugs shaken out.

[dankamongmen]

Alright, I've put a chunk of thought into this, and think I have a good plan.

I'll be calling this "CLI mode" for want of a better name. I think of TUIs as fullscreen deals, and CLIs as scrolling and working exclusively with stdin/stdout. CLIs are things you can (sometimes) pipe, pipes being best used with streams and CLI being a newline-delimited stream. A (roughly) fundamental property of streams: you're not going backwards. A (roughly) fundamental property of terminals: you're not scrolling up (these both have exceptions, as will we). I'd call it "line mode", except we're not really working with lines, just turning on them.

Remember, CLI mode still does not happily coexist with other programs freely generating output. Unlike direct mode, everything must be run through Notcurses. We will, however, provide for some recovery in the event of wayward output.

There is no single bit that says "CLI mode". There are a set of existing behaviors which, following some refinement, can be sensibly composed to achieve this goal. The prototypical CLI-style Notcurses application will:

• use NOTCURSES_OPTION_PRESERVE_CURSOR,
• use NOTCURSES_OPTION_NO_ALTERNATE_SCREEN,
• use NOTCURSES_OPTION_NO_CLEAR_BITMAPS, and
• set the standard plane to scroll at startup.

Of these, scrolling standard plane and PRESERVE_CURSOR are really the hinges upon which everything turns (if you want classic scrolling behavior without any of these effects, you can ncplane_dup() the standard plane, and give it the ncplane_resize_maximize resize callback, and use it instead of the standard plane. Anyone who comes to Notcurses without preconceptions, and sets the standard plane scrolling, is now going to get the behavior they'd expect, I think).

The standard plane remains how it is -- always having geometry equal to the visual area, always with its origin matching that of the visual area, and always present. That's fundamental, because actions taken relative to the standard plane (when it is in scrolling mode) can be interpreted as actions against the visual area.

• ncpiles other than the standard pile may not be rasterized in CLI mode, though they may be used for offpile rendering that is reparented to the standard pile. Rationale: the output of a pile would now require feedback to the standard pile, which would require locking. Furthermore, double buffering doesn't make much sense in a scrolling environment. This constraint might be lifted in the future.
• At startup, when NOTCURSES_OPTION_PRESERVECURSOR is used, the standard plane's virtual cursor will be initialized to the current position of the physical cursor (this is already being done). Rationale: we want our output to start wherever we were when we invoked the program, as it does for most CLI programs.
• At the end of each rasterization, the physical cursor ought be placed at the end of the output we've generated across any rasterization, taking any interim scrolling into account. Rationale: we don't want interleaved output from other sources, but if there is some, it ought go to the right place. when NCOPTION_PRESERVE_CURSOR is used, place at logical end of output #1906
• After a render, the virtual cursor of the standard plane will be updated to match the position of the physical cursor. This is likewise keyed off of NOTCURSES_OPTION_PRESERVECURSOR. Rationale: the act of rasterizing might result in terminal scrolling, but that scrolling might not necessarily take place directly due to the standard plane (more on this shortly). But the standard plane, remember, is supposed to represent our visual area. Note that the term "virtual cursor" here is specific, according to these definitions in HACKING.md.
• Scrolling of the physical terminal beyond that caused by the standard plane will be reflected in logical scrolling of the standard plane at rasterization time. This will not require a cursor location report, because we will know exactly how much scrolling is going to happen. Rationale: the standard plane is being kept in sync with the visual area.
• The use of logging will force a cursor lookup request immediately prior to the next rasterization (it might perhaps be . Rationale: we won't otherwise know how many rows (if any) have been scrolled. This will be implemented via posting to a context-wide atomic, which will be checked at rasterization time. Q: how to deal with a logging event that takes place while rasterizing?
• We will likely want to make use of terminal margin/protected areas (fixed sections of protected non-scrolling data) where available to support efficient, flicker-free implementation of planes fixed relative to the visual area. SUMode might achieve this where protected regions are unavailable, but we'll need feed more information into the SUMode decision. Rationale: this wasn't necessary before, because we didn't have physical scrolling. We now will.
• The lastframe damage cache will scroll along with the standard plane. This will be one of the bigger chunks of new functionality, though we already have scrolling logic, so it ought not be much new code. Rationale: this is critical for performance, as otherwise a scroll will force a manual redraw of the entire screen, since the damage buffers won't line up.
• Planes bound to a scrolling plane move along with scrolls. When a plane scrolls N lines, the plane's immediate children move N lines up, recursing from those points out naturally. Rationale: we fundamentally expect things next to one another to remain next to one another, and this will be necessary for planes to scroll along with the standard plane. Unlike most of these statements, this is a general principle, and will be put into place for scrolling planes in all cases.
• Pile rendering is currently limited to the visual area's overlap. In CLI mode, rendering is expanded to the visual area and any content within its horizontal bounds, virtually below the visual area. This might result in a rasterization larger than the visual area, even multiples of the visual area's size. That's exactly what we expect when we generate a bunch of output at once. Rationale: if this was not done, the previous principle could not be honored without splitting planes, so that the part in the visual area scrolled with the standard plane, but the part below did not.
• If the user has explicitly placed the cursor, that placement ought not scroll with the standard plane. Rationale: what would we do when it logically scrolls off the screen? I'm not certain about this; the rationale is certainly a weak one.

[dankamongmen]

eradicate

see that works if it's like family_eradicate() but i think ncplane_eradicate just sounds like you're power-washing the ncplane, like don't just free it but banish it from existence, make it an unplane

[joseluis]

eradicate

see that works if it's like family_eradicate() but i think ncplane_eradicate just sounds like you're power-washing the ncplane, like don't just free it but banish it from existence, make it an unplane

that would be obliterate or annihilate, exterminate... eradicating is to pull up by the roots; destroy at the roots; root out; extirpate: as, to eradicate weeds

[dankamongmen]

see that works if it's like family_eradicate() but i think ncplane_eradicate just sounds like you're power-washing the ncplane, like don't just free it but banish it from existence, make it an unplane

sigh nothing else really summons up visions of troops coming across the border in black para outfits, lining everyone up against a barn and wiping out your planes with automatic weapons. i mean i guess ncplane_ethnic_cleanse() would but for obvious reasons that's right out.

[joseluis]

or maybe just ncplane_drop_family or ncplane_destroy_family...

[dankamongmen]

What would be a realistic scenario where that could happen? Can there be a way to capture all those loose ends in order to avoid disrupting the terminal?

most certainly. we could allow them to provide a diagnostic callback, which defaults to fprint otherwise. remember that none of these will be emitted by default, or at least shouldn't (file a bug/PR if so, i've been converting fprintf()s to log*() as i come across them), because by default we use NCLOGLEVEL_SILENT. so we could probably stick the diagnostic right next to loglevel in notcurses_options. that's probably where we want it, since you'd want it to be available during initialization. maybe allow them to change it later, idk, so many symbols already ugh.

i figured out what to do here; see #1930 for the solution.

[dankamongmen]

I think we ought document that people using CLI mode probably want to use NCOPTION_NO_CLEAR_BITMAPS, though I don't think PRESERVE_CURSOR ought imply it or anything.

[dankamongmen]

maybe it should. idk. nah

[dankamongmen]

@klamonte (as always) has a lot of good thoughts on this at https://gitlab.com/klamonte/jexer/-/issues/75. in particular:

By "command line", I mean pipe-able (composable) programs that read from stdin, write to stdout, and do a sequence of command / response / command / response. Just like the "read-eval-print-loop" (REPL) used by Lisp, Python, and many other programming languages...In contrast we have the "terminal", meaning a grid of fixed cells comprising a screen of X columns by Y rows. There are many kinds of terminals such as mainframe, DOS, and now most commonly VT100 or "xterm" (really ANSI X3.64 / ECMA-48 type terminals). Nowadays every major desktop operating system has a good or getting-better terminal: the new Windows Terminal for Windows 10, Terminal.app for Mac, and dozens of fantastic open-sources choices for Windows/Mac/Linux/BSD/other (xterm itself, rxvt-unicode, gnome-terminal, konsole, mintty, iTerm2)...The evidence to me shows that people, in general, do not like the "terminal" part of the "command line". Not enough to replicate the DOS-era full-screen applications like Turbo Vision in the 27 years since its initial release, or in the 22 years since Borland abandoned it. Not enough to create a new Turbo Pascal or Turbo C++ IDE experience for other languages. Not enough to request that the most popular terminals pass vttest. Not enough to look around at other terminals and applications to make theirs notably better.

and

So what has the market said to me about what people need in a terminal? What they do not need are: vttest compliance, custom keybindings, Zmodem/Kermit file transfers, scripting, screen dump, scrollback dump, session capture, sixel graphics, and text-based windowing systems. What they do need are: full-width glyph support (CJK), emojis, 24-bit RGB, tabbed windows, reflowable text, clickable URLs, and background transparency...When you compare these lists of features, it looks an awful lot like what people actually need from their terminal are the same underlying features of SMS/iMessage mobile apps: language support, colors, emojis, and pictures. iMessage is in fact a "command line", just with another person (presumably) on the other end rather than a computer. Again: "command line", not "terminal".

so i think this is going to be pretty important. with that said, if we can truly bring the full power of Notcurses to bear, we're going to be able to unleash the most kick-ass command line tools ever seen.