workspace: recover from missing operation

If the operation corresponding to a workspace is missing for some reason
(the specific situation in the test in this commit is that an operation
was abandoned and garbage-collected from another workspace), currently,
jj fails with a 255 error code. Teach jj a way to recover from this
situation.

When jj detects such a situation, it prints a message and stops
operation, similar to when a workspace is stale. The message tells the
user what command to run.

When that command is run, jj loads the repo at the @ operation (instead
of the operation of the workspace), creates a new commit on the @ commit
with the same tree as the @ commit, and then proceeds as usual - in
particular, including the auto-snapshotting of the working tree, which
creates another commit that obsoletes the newly created commit.

There are several design points I considered.

1) Whether the recovery should be automatic, or (as in this commit)
manual in that the user should be prompted to run a command. The user
might prefer to recover in another way (e.g. by simply deleting the
workspace) and this situation is (hopefully) rare enough that I think
it's better to prompt the user.

2) Which command the user should be prompted to run (and thus, which
command should be taught to perform the recovery). I chose "workspace
update-stale" because the circumstances are very similar to it: it's
symptom is that the regular jj operation is blocked somewhere at the
beginning, and "workspace update-stale" already does some special work
before the blockage (this commit adds more of such special work). But it
might be better for something more explicitly named, or even a sequence
of commands (e.g. "create a new operation that becomes @ that no
workspace points to", "low-level command that makes a workspace point to
the operation @") but I can see how this can be unnecessarily confusing
for the user.

3) How we recover. I can think of several ways:
a) Always create a commit, and allow the automatic snapshotting to
create another commit that obsoletes this commit.
b) Create a commit but somehow teach the automatic snapshotting to
replace the created commit in-place (so it has no predecessor, as viewed
in "obslog").
c) Do either a) or b), with the added improvement that if there is no
diff between the newly created commit and the former @, to behave as if
no new commit was created (@ remains as the former @).
I chose a) since it was the simplest and most easily reasoned about,
which I think is the best way to go when recovering from a rare
situation.

cli/src/cli_util.rs

@@ -692,20 +692,81 @@ impl CommandHelper {
         WorkspaceCommandHelper::new(ui, self, workspace, repo)
     }
 
+    fn create_and_check_out_recovery_commit(
+        &self,
+        ui: &mut Ui,
+        workspace: &mut Workspace,
+    ) -> Result<Arc<ReadonlyRepo>, CommandError> {
+        let operation = self.resolve_operation(ui, workspace.repo_loader())?;
+        let repo = workspace.repo_loader().load_at(&operation)?;
+        let commit_id = repo
+            .view()
+            .get_wc_commit_id(workspace.workspace_id())
+            .ok_or_else(|| CommandError::InternalError("Could not load commit ID".to_string()))?;
+        let commit = repo.store().get_commit(commit_id)?;
+        let tree_id = commit.tree_id();
+
+        let mut tx = repo.start_transaction(&self.settings);
+        let mut_repo = tx.mut_repo();
+        let new_commit = mut_repo
+            .new_commit(&self.settings, vec![commit_id.clone()], tree_id.clone())
+            .write()?;
+        mut_repo.set_wc_commit(workspace.workspace_id().clone(), new_commit.id().clone())?;
+        let ret = Ok(tx.commit("recovery commit"));
+
+        let mut locked_workspace = workspace.start_working_copy_mutation()?;
+        locked_workspace
+            .locked_wc()
+            .reset(&repo.store().get_root_tree(tree_id)?)
+            //.check_out(&new_commit)
+            .map_err(|err| {
+                CommandError::InternalError(format!(
+                    "Could not check out commit in workspace: {err}"
+                ))
+            })?;
+        locked_workspace.finish(operation.id().clone())?;
+
+        writeln!(
+            ui.stderr(),
+            "Created and checked out recovery commit {}",
+            new_commit.id().hex()
+        )?;
+
+        ret
+    }
+
     /// Loads workspace that will diverge from the last working-copy operation.
     pub fn for_stale_working_copy(
         &self,
         ui: &mut Ui,
-    ) -> Result<WorkspaceCommandHelper, CommandError> {
-        let workspace = self.load_workspace()?;
+    ) -> Result<(WorkspaceCommandHelper, bool), CommandError> {
+        let mut workspace = self.load_workspace()?;
         let op_store = workspace.repo_loader().op_store();
-        let op_id = workspace.working_copy().operation_id();
-        let op_data = op_store
-            .read_operation(op_id)
-            .map_err(|e| CommandError::InternalError(format!("Failed to read operation: {e}")))?;
-        let operation = Operation::new(op_store.clone(), op_id.clone(), op_data);
-        let repo = workspace.repo_loader().load_at(&operation)?;
-        self.for_loaded_repo(ui, workspace, repo)
+        let (repo, recovered) = {
+            let op_id = workspace.working_copy().operation_id();
+            match op_store.read_operation(op_id) {
+                Ok(op_data) => (
+                    workspace.repo_loader().load_at(&Operation::new(
+                        op_store.clone(),
+                        op_id.clone(),
+                        op_data,
+                    ))?,
+                    false,
+                ),
+                Err(e) => {
+                    writeln!(
+                        ui.stderr(),
+                        "Failed to read working copy's current operation; attempting recovery. \
+                         Error message from read attempt: {e}"
+                    )?;
+                    (
+                        self.create_and_check_out_recovery_commit(ui, &mut workspace)?,
+                        true,
+                    )
+                }
+            }
+        };
+        Ok((self.for_loaded_repo(ui, workspace, repo)?, recovered))
     }
 }
 
@@ -1412,6 +1473,14 @@ See https://github.com/martinvonz/jj/blob/main/docs/working-copy.md#stale-workin
                         short_operation_hash(&old_op_id)
                     )));
                 }
+                WorkingCopyFreshness::MissingOperation => {
+                    return Err(user_error_with_hint(
+                        "Could not read working copy's operation.",
+                        "Run `jj workspace update-stale` to recover.
+See https://github.com/martinvonz/jj/blob/main/docs/working-copy.md#stale-working-copy \
+                         for more information.",
+                    ));
+                }
             };
         self.user_repo = ReadonlyUserRepo::new(repo);
         let progress = crate::progress::snapshot_progress(ui);
@@ -1917,14 +1986,18 @@ pub enum WorkingCopyFreshness {
     WorkingCopyStale,
     /// The working copy is a sibling of the latest operation.
     SiblingOperation,
+    /// The working copy's operation could not be loaded.
+    MissingOperation,
 }
 
 impl WorkingCopyFreshness {
     /// Returns true if the working copy is not updated to the current
     /// operation.
     pub fn is_stale(&self) -> bool {
         match self {
-            WorkingCopyFreshness::Fresh | WorkingCopyFreshness::Updated(_) => false,
+            WorkingCopyFreshness::Fresh
+            | WorkingCopyFreshness::Updated(_)
+            | WorkingCopyFreshness::MissingOperation => false,
             WorkingCopyFreshness::WorkingCopyStale | WorkingCopyFreshness::SiblingOperation => true,
         }
     }
@@ -1942,9 +2015,10 @@ pub fn check_stale_working_copy(
         // The working copy isn't stale, and no need to reload the repo.
         Ok(WorkingCopyFreshness::Fresh)
     } else {
-        let wc_operation_data = repo
-            .op_store()
-            .read_operation(locked_wc.old_operation_id())?;
+        let wc_operation_data = match repo.op_store().read_operation(locked_wc.old_operation_id()) {
+            Ok(o) => o,
+            Err(_) => return Ok(WorkingCopyFreshness::MissingOperation),
+        };
         let wc_operation = Operation::new(
             repo.op_store().clone(),
             locked_wc.old_operation_id().clone(),

cli/src/commands/workspace.rs

@@ -289,22 +289,28 @@ fn cmd_workspace_update_stale(
     // operation, then merge the concurrent operations. The wc_commit_id of the
     // merged repo wouldn't change because the old one wins, but it's probably
     // fine if we picked the new wc_commit_id.
-    let known_wc_commit = {
-        let mut workspace_command = command.for_stale_working_copy(ui)?;
+    let (known_wc_commit, recovered) = {
+        let (mut workspace_command, recovered) = command.for_stale_working_copy(ui)?;
         workspace_command.maybe_snapshot(ui)?;
+
         let wc_commit_id = workspace_command.get_wc_commit_id().unwrap();
-        workspace_command.repo().store().get_commit(wc_commit_id)?
+        (
+            workspace_command.repo().store().get_commit(wc_commit_id)?,
+            recovered,
+        )
     };
     let mut workspace_command = command.workspace_helper_no_snapshot(ui)?;
 
     let repo = workspace_command.repo().clone();
     let (mut locked_ws, desired_wc_commit) =
         workspace_command.unchecked_start_working_copy_mutation()?;
     if !check_stale_working_copy(locked_ws.locked_wc(), &desired_wc_commit, &repo)?.is_stale() {
-        writeln!(
-            ui.stderr(),
-            "Nothing to do (the working copy is not stale)."
-        )?;
+        if !recovered {
+            writeln!(
+                ui.stderr(),
+                "Nothing to do (the working copy is not stale)."
+            )?;
+        }
     } else {
         // The same check as start_working_copy_mutation(), but with the stale
         // working-copy commit.

cli/tests/test_workspaces.rs

@@ -384,6 +384,98 @@ fn test_workspaces_updated_by_other() {
     "###);
 }
 
+#[test]
+fn test_workspaces_current_op_discarded_by_other() {
+    let test_env = TestEnvironment::default();
+    // Use the local backend because GitBackend::gc() depends on the git CLI.
+    test_env.jj_cmd_ok(
+        test_env.env_root(),
+        &["init", "main", "--config-toml=ui.allow-init-native=true"],
+    );
+    let main_path = test_env.env_root().join("main");
+    let secondary_path = test_env.env_root().join("secondary");
+
+    std::fs::write(main_path.join("file"), "contents\n").unwrap();
+    test_env.jj_cmd_ok(&main_path, &["new"]);
+
+    test_env.jj_cmd_ok(&main_path, &["workspace", "add", "../secondary"]);
+
+    // Create an op by abandoning the parent commit. Importantly, that commit also
+    // changes the target tree in the secondary workspace.
+    test_env.jj_cmd_ok(&main_path, &["abandon", "@-"]);
+
+    let stdout = test_env.jj_cmd_success(
+        &main_path,
+        &[
+            "operation",
+            "log",
+            "--template",
+            r#"id.short(10) ++ " " ++ description"#,
+        ],
+    );
+    insta::assert_snapshot!(stdout, @r###"
+    @  6c22dbc43c abandon commit 479653b3216ad2af6a8ffcbe5885203b7bb3d0d49175ffd2be58a454da95493b0e8e44885ac4f7caacbeb79b407c66d68991f5a01666e976687c6732b3311780
+    ◉  72b8975c75 Create initial working-copy commit in workspace secondary
+    ◉  b816f120a4 add workspace 'secondary'
+    ◉  332e901b2a new empty commit
+    ◉  c07b7d7796 snapshot working copy
+    ◉  c5295044c8 add workspace 'default'
+    ◉  0c73edb541 initialize repo
+    ◉  0000000000
+    "###);
+
+    // Abandon ops, including the one the secondary workspace is currently on.
+    test_env.jj_cmd_ok(&main_path, &["operation", "abandon", "..@-"]);
+    test_env.jj_cmd_ok(&main_path, &["util", "gc", "--expire=now"]);
+
+    insta::assert_snapshot!(get_log_output(&test_env, &main_path), @r###"
+    @  3bebcd0f8335f735c00063290247d6c3332106261d0dd4810105906cdf29878990ee717690750ec203a7c4763a36968a0d9de799f46bc37fcf62dde13a52b73f default@
+    │ ◉  1d558d136f4d90ec2fe9beac8eb638eb509d0e378c4546f7b2634f93d0c86cfdd08b7efe4fd3aeb383f41fb7ebb2311a14754f617e71389c3aa9519c05c17dd8 secondary@
+    ├─╯
+    ◉  00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
+    "###);
+
+    let stderr = test_env.jj_cmd_failure(&secondary_path, &["st"]);
+    insta::assert_snapshot!(stderr, @r###"
+    Error: Could not read working copy's operation.
+    Hint: Run `jj workspace update-stale` to recover.
+    See https://github.com/martinvonz/jj/blob/main/docs/working-copy.md#stale-working-copy for more information.
+    "###);
+
+    let (stdout, stderr) = test_env.jj_cmd_ok(&secondary_path, &["workspace", "update-stale"]);
+    insta::assert_snapshot!(stderr, @r###"
+    Failed to read working copy's current operation; attempting recovery. Error message from read attempt: Object 72b8975c7573303a258c292ad69f93dad1922532a2839000c217102da736ed7c6235872bdfa8c747a723df78547666686722d0d301da2d70d815798634c8eedc of type operation not found: No such file or directory (os error 2)
+    Created and checked out recovery commit ba75b52f282b5bd12fca18f7e59c0a9ef34459c01a1b6cd384114508c83c3bd2364b0aa5ec87b503862138603a81b87b4156a8b49fb754ad76b572c0aa3c1bad
+    "###);
+    insta::assert_snapshot!(stdout, @"");
+
+    insta::assert_snapshot!(get_log_output(&test_env, &main_path), @r###"
+    ◉  18352d32a66810b10e2ea210807fe23b570c3dfa50776075f2b5efde69f39fddda1a7f7c78b34aea0b59bf2e8ec96f0ee4a2507b929c6ddbee30efc86f8358aa secondary@
+    ◉  1d558d136f4d90ec2fe9beac8eb638eb509d0e378c4546f7b2634f93d0c86cfdd08b7efe4fd3aeb383f41fb7ebb2311a14754f617e71389c3aa9519c05c17dd8
+    │ @  3bebcd0f8335f735c00063290247d6c3332106261d0dd4810105906cdf29878990ee717690750ec203a7c4763a36968a0d9de799f46bc37fcf62dde13a52b73f default@
+    ├─╯
+    ◉  00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
+    "###);
+
+    let (stdout, stderr) = test_env.jj_cmd_ok(&secondary_path, &["st"]);
+    insta::assert_snapshot!(stderr, @"");
+    insta::assert_snapshot!(stdout, @r###"
+    Working copy changes:
+    A file
+    Working copy : znkkpsqq 18352d32 (no description set)
+    Parent commit: pmmvwywv 1d558d13 (empty) (no description set)
+    "###);
+
+    let (stdout, stderr) = test_env.jj_cmd_ok(&secondary_path, &["obslog"]);
+    insta::assert_snapshot!(stderr, @"");
+    insta::assert_snapshot!(stdout, @r###"
+    @  znkkpsqq [email protected] 2001-02-03 04:05:16.000 +07:00 secondary@ 18352d32
+    │  (no description set)
+    ◉  znkkpsqq hidden [email protected] 2001-02-03 04:05:16.000 +07:00 ba75b52f
+       (empty) (no description set)
+    "###);
+}
+
 #[test]
 fn test_workspaces_update_stale_noop() {
     let test_env = TestEnvironment::default();