Implement session recording and playback support in Cockpit

[spbnick]

Implement support for session recording and playback in Cockpit, for the
controlled machine.

For the start don't implement configuring session recording in Cockpit.
Leave that to users to do manually, then add that to SSSD configuration when
it is added to Cockpit.

Implement local session recording in tlog, so that actual recorded sessions
could be listed.

For example, consider recording to journal with extra meta-data, so that it is
then possible to list all recorded sessions and play them back. See
Scribery/tlog#88.

If that doesn't work, consider recording to per-session files in some
directory in the filesystem.

Implement an interface in Cockpit listing available recorded sessions, where
you could click on and play back a specific session.

Implement playback by running tlog-play on the local machine and showing its
output in a JavaScript terminal emulator, similarly to how it's done with
local console in Cockpit. Try to reach @dperpeet for that.

Also, support forwarding recordings somewhere else using the regular logging
channel. If not recording to journal, implement support for multiple writers
(Scribery/tlog#76).

This might also need support for rewind/fast-forward in tlog-play (see Scribery/tlog#90).

[spbnick]

To implement playback resizing we can make tlog-play output resize control sequences (see Scribery/tlog#94) and make the term.js library interpret them. This way we won't need any other channel to communicate the resizes.

[spbnick]

Rough plan for the start:

• Add another Cockpit "package"
• Add menu item "Sessions" or "Session recordings"
• Simply list files in a directory as sessions
• Files contain pre-recorded sessions for testing
• Clicking on a "session" opens a terminal, which plays back the session
• We won't have the progress bar for the start, likely only with a full
  web-based player

Later:

• Check if we can use journald for storing and querying sessions, ask on
  systemd-devel.
• Once/if we implement playback controls in tlog-play, the user should be able
  to press the controlling key combinations while the terminal is in focus
• We can also have buttons surround the terminal, which would send particular
  key combinations to the terminal and thus tlog-play

[spbnick]

Rather than invent file storage and write all the management code around it, and write custom tests for journal performance I just went ahead and implemented recording and playback to/from journal in tlog. Seems to work so far, and wasn't hard. I'll go ahead and start writing Cockpit code for listing sessions from the journal now.

[dperpeet]

fyi, we're currently working on providing a demo out of tree plugin for Cockpit. We're tracking this effort on trello and we're using subscription-manager-cockpit as the case study.

[spbnick]

Thanks a lot, @dperpeet! I'll look into making an out-of-tree plugin when I figure out more about Cockpit. So far I'm just trying to make something work :)

[dperpeet]

If you have a working installation of cockpit, you should be able to:

• check out the sample repo I linked
• run npm install (see build notes)
• run make install

And after the next cockpit login you should see the sample page.

Otherwise I recommend building from cockpit's git repo and editing the playground. But that is more complex: https://github.com/cockpit-project/cockpit/blob/master/HACKING.md.

[spbnick]

Thank you. So far I've been adding my page directly to the cockpit tree, borrowing from systemd/logs and systemd/terminal. I'll try the plugin route too.

[dperpeet]

Whatever works for you! The plugin is definitely a work in progress, so if you have something that works, you can definitely stick to that for now.

[spbnick]

I got some basic list of sessions to appear in Cockpit based on the journal contents, but found a systemd problem, apparently related to systemd/systemd#1347, which manifests in some required fields missing in last entries tlog produces right before exit. Will need to add them in tlog itself, even though systemd normally adds them anyway.

[spbnick]

We will likely need to distinguish separate tlog recordings using the combination of boot ID, recording process PID and start time. We might have to implement Scribery/tlog#95 and Scribery/tlog#103.

[spbnick]

I got a very rudimentary "Session Recording" page working in Cockpit in this branch: https://github.com/Scribery/cockpit/tree/hacking

This still needs a lot of work and likely changes in tlog, but we got a proof that something is possible.

I'll make a short video demo soon.

[spbnick]

A video demo I made a while back: https://youtu.be/BPiqFKvW0q4

[spbnick]

I implemented logging a "recording ID" which would let us uniquely identify a recording on a host. This was necessary, because just audit session ID is not enough, such as across reboots.

[spbnick]

See https://github.com/Scribery/cockpit/milestone/3 for detailed tracking.