workspace: recover from missing operation

If the operation corresponding to a workspace is missing for some reason
(the specific situation in the test in this commit is that an operation
was abandoned and garbage-collected from another workspace), currently,
jj fails with a 255 error code. Teach jj a way to recover from this
situation.

When jj detects such a situation, it prints a message and stops
operation, similar to when a workspace is stale. The message tells the
user what command to run.

When that command is run, jj loads the repo at the @ operation (instead
of the operation of the workspace), creates a new commit on the @
commit with an empty tree, and then proceeds as usual - in particular,
including the auto-snapshotting of the working tree, which creates
another commit that obsoletes the newly created commit.

There are several design points I considered.

1) Whether the recovery should be automatic, or (as in this commit)
manual in that the user should be prompted to run a command. The user
might prefer to recover in another way (e.g. by simply deleting the
workspace) and this situation is (hopefully) rare enough that I think
it's better to prompt the user.

2) Which command the user should be prompted to run (and thus, which
command should be taught to perform the recovery). I chose "workspace
update-stale" because the circumstances are very similar to it: it's
symptom is that the regular jj operation is blocked somewhere at the
beginning, and "workspace update-stale" already does some special work
before the blockage (this commit adds more of such special work). But it
might be better for something more explicitly named, or even a sequence
of commands (e.g. "create a new operation that becomes @ that no
workspace points to", "low-level command that makes a workspace point to
the operation @") but I can see how this can be unnecessarily confusing
for the user.

3) How we recover. I can think of several ways:
a) Always create a commit, and allow the automatic snapshotting to
create another commit that obsoletes this commit.
b) Create a commit but somehow teach the automatic snapshotting to
replace the created commit in-place (so it has no predecessor, as viewed
in "obslog").
c) Do either a) or b), with the added improvement that if there is no
diff between the newly created commit and the former @, to behave as if
no new commit was created (@ remains as the former @).
I chose a) since it was the simplest and most easily reasoned about,
which I think is the best way to go when recovering from a rare
situation.

cli/examples/custom-working-copy/main.rs

@@ -240,6 +240,10 @@ impl LockedWorkingCopy for LockedConflictsWorkingCopy {
         self.inner.reset(commit)
     }
 
+    fn reset_to_empty(&mut self) -> Result<(), ResetError> {
+        self.inner.reset_to_empty()
+    }
+
     fn sparse_patterns(&self) -> Result<&[RepoPathBuf], WorkingCopyStateError> {
         self.inner.sparse_patterns()
     }

cli/src/cli_util.rs

@@ -1423,9 +1423,9 @@ Set which revision the branch points to with `jj branch set {branch_name} -r <RE
         let mut locked_ws = self.workspace.start_working_copy_mutation()?;
         let old_op_id = locked_ws.locked_wc().old_operation_id().clone();
         let (repo, wc_commit) =
-            match check_stale_working_copy(locked_ws.locked_wc(), &wc_commit, &repo)? {
-                WorkingCopyFreshness::Fresh => (repo, wc_commit),
-                WorkingCopyFreshness::Updated(wc_operation) => {
+            match check_stale_working_copy(locked_ws.locked_wc(), &wc_commit, &repo) {
+                Ok(WorkingCopyFreshness::Fresh) => (repo, wc_commit),
+                Ok(WorkingCopyFreshness::Updated(wc_operation)) => {
                     let repo = repo.reload_at(&wc_operation)?;
                     let wc_commit = if let Some(wc_commit) = get_wc_commit(&repo)? {
                         wc_commit
@@ -1435,7 +1435,7 @@ Set which revision the branch points to with `jj branch set {branch_name} -r <RE
                     };
                     (repo, wc_commit)
                 }
-                WorkingCopyFreshness::WorkingCopyStale => {
+                Ok(WorkingCopyFreshness::WorkingCopyStale) => {
                     return Err(user_error_with_hint(
                         format!(
                             "The working copy is stale (not updated since operation {}).",
@@ -1446,14 +1446,23 @@ See https://github.com/martinvonz/jj/blob/main/docs/working-copy.md#stale-workin
                          for more information.",
                     ));
                 }
-                WorkingCopyFreshness::SiblingOperation => {
+                Ok(WorkingCopyFreshness::SiblingOperation) => {
                     return Err(internal_error(format!(
                         "The repo was loaded at operation {}, which seems to be a sibling of the \
                          working copy's operation {}",
                         short_operation_hash(repo.op_id()),
                         short_operation_hash(&old_op_id)
                     )));
                 }
+                Err(OpStoreError::ObjectNotFound { .. }) => {
+                    return Err(user_error_with_hint(
+                        "Could not read working copy's operation.",
+                        "Run `jj workspace update-stale` to recover.
+See https://github.com/martinvonz/jj/blob/main/docs/working-copy.md#stale-working-copy \
+                         for more information.",
+                    ))
+                }
+                Err(e) => return Err(e.into()),
             };
         self.user_repo = ReadonlyUserRepo::new(repo);
         let progress = crate::progress::snapshot_progress(ui);

cli/src/commands/workspace.rs

@@ -15,21 +15,23 @@
 use std::fmt::Debug;
 use std::fs;
 use std::io::Write;
+use std::sync::Arc;
 
 use clap::Subcommand;
 use itertools::Itertools;
 use jj_lib::file_util;
 use jj_lib::object_id::ObjectId;
-use jj_lib::op_store::WorkspaceId;
+use jj_lib::op_store::{OpStoreError, WorkspaceId};
 use jj_lib::operation::Operation;
-use jj_lib::repo::Repo;
+use jj_lib::repo::{ReadonlyRepo, Repo};
 use jj_lib::rewrite::merge_commit_trees;
 use jj_lib::workspace::Workspace;
 use tracing::instrument;
 
 use crate::cli_util::{
-    self, check_stale_working_copy, internal_error_with_message, print_checkout_stats, user_error, CommandError, CommandHelper,
-    RevisionArg, WorkingCopyFreshness, WorkspaceCommandHelper,
+    self, check_stale_working_copy, internal_error_with_message, print_checkout_stats,
+    short_commit_hash, user_error, CommandError, CommandHelper, RevisionArg, WorkingCopyFreshness,
+    WorkspaceCommandHelper,
 };
 use crate::ui::Ui;
 
@@ -280,20 +282,68 @@ fn cmd_workspace_root(
     Ok(())
 }
 
+fn create_and_check_out_recovery_commit(
+    ui: &mut Ui,
+    command: &CommandHelper,
+) -> Result<Arc<ReadonlyRepo>, CommandError> {
+    let mut workspace_command = command.workspace_helper_no_snapshot(ui)?;
+    let workspace_id = workspace_command.workspace_id().clone();
+    let repo = workspace_command.repo().clone();
+    let tree_id = repo.store().empty_merged_tree_id();
+    let (mut locked_workspace, commit) =
+        workspace_command.unchecked_start_working_copy_mutation()?;
+    let commit_id = commit.id();
+
+    let mut tx = repo.start_transaction(command.settings());
+    let mut_repo = tx.mut_repo();
+    let new_commit = mut_repo
+        .new_commit(command.settings(), vec![commit_id.clone()], tree_id.clone())
+        .write()?;
+    mut_repo.set_wc_commit(workspace_id, new_commit.id().clone())?;
+    let repo = tx.commit("recovery commit");
+
+    locked_workspace.locked_wc().reset_to_empty()?;
+    locked_workspace.finish(repo.op_id().clone())?;
+
+    writeln!(
+        ui.stderr(),
+        "Created and checked out recovery commit {}",
+        short_commit_hash(new_commit.id())
+    )?;
+
+    Ok(repo)
+}
+
 /// Loads workspace that will diverge from the last working-copy operation.
 fn for_stale_working_copy(
     ui: &mut Ui,
     command: &CommandHelper,
-) -> Result<WorkspaceCommandHelper, CommandError> {
+) -> Result<(WorkspaceCommandHelper, bool), CommandError> {
     let workspace = command.load_workspace()?;
     let op_store = workspace.repo_loader().op_store();
-    let op_id = workspace.working_copy().operation_id();
-    let op_data = op_store
-        .read_operation(op_id)
-        .map_err(|e| internal_error_with_message("Failed to read operation", e))?;
-    let operation = Operation::new(op_store.clone(), op_id.clone(), op_data);
-    let repo = workspace.repo_loader().load_at(&operation)?;
-    command.for_loaded_repo(ui, workspace, repo)
+    let (repo, recovered) = {
+        let op_id = workspace.working_copy().operation_id();
+        match op_store.read_operation(op_id) {
+            Ok(op_data) => (
+                workspace.repo_loader().load_at(&Operation::new(
+                    op_store.clone(),
+                    op_id.clone(),
+                    op_data,
+                ))?,
+                false,
+            ),
+            Err(e @ OpStoreError::ObjectNotFound { .. }) => {
+                writeln!(
+                    ui.stderr(),
+                    "Failed to read working copy's current operation; attempting recovery. Error \
+                     message from read attempt: {e}"
+                )?;
+                (create_and_check_out_recovery_commit(ui, command)?, true)
+            }
+            Err(e) => return Err(e.into()),
+        }
+    };
+    Ok((command.for_loaded_repo(ui, workspace, repo)?, recovered))
 }
 
 #[instrument(skip_all)]
@@ -307,8 +357,16 @@ fn cmd_workspace_update_stale(
     // merged repo wouldn't change because the old one wins, but it's probably
     // fine if we picked the new wc_commit_id.
     let known_wc_commit = {
-        let mut workspace_command = for_stale_working_copy(ui, command)?;
+        let (mut workspace_command, recovered) = for_stale_working_copy(ui, command)?;
         workspace_command.maybe_snapshot(ui)?;
+
+        if recovered {
+            // We have already recovered from the situation that prompted the user to run
+            // this command, and it is known that the workspace is not stale
+            // (since we just updated it), so we can return early.
+            return Ok(());
+        }
+
         let wc_commit_id = workspace_command.get_wc_commit_id().unwrap();
         workspace_command.repo().store().get_commit(wc_commit_id)?
     };
@@ -318,10 +376,12 @@ fn cmd_workspace_update_stale(
     let (mut locked_ws, desired_wc_commit) =
         workspace_command.unchecked_start_working_copy_mutation()?;
     match check_stale_working_copy(locked_ws.locked_wc(), &desired_wc_commit, &repo)? {
-        WorkingCopyFreshness::Fresh | WorkingCopyFreshness::Updated(_) => writeln!(
-            ui.stderr(),
-            "Nothing to do (the working copy is not stale)."
-        )?,
+        WorkingCopyFreshness::Fresh | WorkingCopyFreshness::Updated(_) => {
+            writeln!(
+                ui.stderr(),
+                "Nothing to do (the working copy is not stale)."
+            )?;
+        }
         WorkingCopyFreshness::WorkingCopyStale | WorkingCopyFreshness::SiblingOperation => {
             // The same check as start_working_copy_mutation(), but with the stale
             // working-copy commit.
@@ -332,13 +392,13 @@ fn cmd_workspace_update_stale(
                 .locked_wc()
                 .check_out(&desired_wc_commit)
                 .map_err(|err| {
-                internal_error_with_message(
-                    format!(
-                        "Failed to check out commit {}",
-                        desired_wc_commit.id().hex()
-                    ),
-                    err,
-                )
+                    internal_error_with_message(
+                        format!(
+                            "Failed to check out commit {}",
+                            desired_wc_commit.id().hex()
+                        ),
+                        err,
+                    )
                 })?;
             locked_ws.finish(repo.op_id().clone())?;
             write!(ui.stderr(), "Working copy now at: ")?;

cli/tests/test_workspaces.rs

@@ -382,6 +382,98 @@ fn test_workspaces_updated_by_other() {
     "###);
 }
 
+#[test]
+fn test_workspaces_current_op_discarded_by_other() {
+    let test_env = TestEnvironment::default();
+    // Use the local backend because GitBackend::gc() depends on the git CLI.
+    test_env.jj_cmd_ok(
+        test_env.env_root(),
+        &["init", "main", "--config-toml=ui.allow-init-native=true"],
+    );
+    let main_path = test_env.env_root().join("main");
+    let secondary_path = test_env.env_root().join("secondary");
+
+    std::fs::write(main_path.join("file"), "contents\n").unwrap();
+    test_env.jj_cmd_ok(&main_path, &["new"]);
+
+    test_env.jj_cmd_ok(&main_path, &["workspace", "add", "../secondary"]);
+
+    // Create an op by abandoning the parent commit. Importantly, that commit also
+    // changes the target tree in the secondary workspace.
+    test_env.jj_cmd_ok(&main_path, &["abandon", "@-"]);
+
+    let stdout = test_env.jj_cmd_success(
+        &main_path,
+        &[
+            "operation",
+            "log",
+            "--template",
+            r#"id.short(10) ++ " " ++ description"#,
+        ],
+    );
+    insta::assert_snapshot!(stdout, @r###"
+    @  6c22dbc43c abandon commit 479653b3216ad2af6a8ffcbe5885203b7bb3d0d49175ffd2be58a454da95493b0e8e44885ac4f7caacbeb79b407c66d68991f5a01666e976687c6732b3311780
+    ◉  72b8975c75 Create initial working-copy commit in workspace secondary
+    ◉  b816f120a4 add workspace 'secondary'
+    ◉  332e901b2a new empty commit
+    ◉  c07b7d7796 snapshot working copy
+    ◉  c5295044c8 add workspace 'default'
+    ◉  0c73edb541 initialize repo
+    ◉  0000000000
+    "###);
+
+    // Abandon ops, including the one the secondary workspace is currently on.
+    test_env.jj_cmd_ok(&main_path, &["operation", "abandon", "..@-"]);
+    test_env.jj_cmd_ok(&main_path, &["util", "gc", "--expire=now"]);
+
+    insta::assert_snapshot!(get_log_output(&test_env, &main_path), @r###"
+    @  3bebcd0f8335f735c00063290247d6c3332106261d0dd4810105906cdf29878990ee717690750ec203a7c4763a36968a0d9de799f46bc37fcf62dde13a52b73f default@
+    │ ◉  1d558d136f4d90ec2fe9beac8eb638eb509d0e378c4546f7b2634f93d0c86cfdd08b7efe4fd3aeb383f41fb7ebb2311a14754f617e71389c3aa9519c05c17dd8 secondary@
+    ├─╯
+    ◉  00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
+    "###);
+
+    let stderr = test_env.jj_cmd_failure(&secondary_path, &["st"]);
+    insta::assert_snapshot!(stderr, @r###"
+    Error: Could not read working copy's operation.
+    Hint: Run `jj workspace update-stale` to recover.
+    See https://github.com/martinvonz/jj/blob/main/docs/working-copy.md#stale-working-copy for more information.
+    "###);
+
+    let (stdout, stderr) = test_env.jj_cmd_ok(&secondary_path, &["workspace", "update-stale"]);
+    insta::assert_snapshot!(stderr, @r###"
+    Failed to read working copy's current operation; attempting recovery. Error message from read attempt: Object 72b8975c7573303a258c292ad69f93dad1922532a2839000c217102da736ed7c6235872bdfa8c747a723df78547666686722d0d301da2d70d815798634c8eedc of type operation not found
+    Created and checked out recovery commit 84e67dd875e0
+    "###);
+    insta::assert_snapshot!(stdout, @"");
+
+    insta::assert_snapshot!(get_log_output(&test_env, &main_path), @r###"
+    ◉  be7e513bfb77cde552e363f947ecc5757557d0d24caf5c627eb9631f38215334722c31b4266c3627111210b1c5eb00f4cd7af4299510cb8c2fe39b405d31b45b secondary@
+    ◉  1d558d136f4d90ec2fe9beac8eb638eb509d0e378c4546f7b2634f93d0c86cfdd08b7efe4fd3aeb383f41fb7ebb2311a14754f617e71389c3aa9519c05c17dd8
+    │ @  3bebcd0f8335f735c00063290247d6c3332106261d0dd4810105906cdf29878990ee717690750ec203a7c4763a36968a0d9de799f46bc37fcf62dde13a52b73f default@
+    ├─╯
+    ◉  00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
+    "###);
+
+    let (stdout, stderr) = test_env.jj_cmd_ok(&secondary_path, &["st"]);
+    insta::assert_snapshot!(stderr, @"");
+    insta::assert_snapshot!(stdout, @r###"
+    Working copy changes:
+    A file
+    Working copy : znkkpsqq be7e513b (no description set)
+    Parent commit: pmmvwywv 1d558d13 (empty) (no description set)
+    "###);
+
+    let (stdout, stderr) = test_env.jj_cmd_ok(&secondary_path, &["obslog"]);
+    insta::assert_snapshot!(stderr, @"");
+    insta::assert_snapshot!(stdout, @r###"
+    @  znkkpsqq [email protected] 2001-02-03 04:05:16.000 +07:00 secondary@ be7e513b
+    │  (no description set)
+    ◉  znkkpsqq hidden [email protected] 2001-02-03 04:05:16.000 +07:00 84e67dd8
+       (empty) (no description set)
+    "###);
+}
+
 #[test]
 fn test_workspaces_update_stale_noop() {
     let test_env = TestEnvironment::default();

lib/src/local_working_copy.rs

@@ -197,6 +197,10 @@ impl FileStatesMap {
         .collect();
     }
 
+    fn clear(&mut self) {
+        self.data.clear();
+    }
+
     /// Returns read-only map containing all file states.
     fn all(&self) -> FileStates<'_> {
         FileStates::from_sorted(&self.data)
@@ -1436,6 +1440,11 @@ impl TreeState {
         self.tree_id = new_tree.id();
         Ok(())
     }
+
+    pub fn reset_to_empty(&mut self) {
+        self.file_states.clear();
+        self.tree_id = self.store.empty_merged_tree_id();
+    }
 }
 
 fn checkout_error_for_stat_error(err: std::io::Error, path: &Path) -> CheckoutError {
@@ -1753,6 +1762,18 @@ impl LockedWorkingCopy for LockedLocalWorkingCopy {
         Ok(())
     }
 
+    fn reset_to_empty(&mut self) -> Result<(), ResetError> {
+        self.wc
+            .tree_state_mut()
+            .map_err(|err| ResetError::Other {
+                message: "Failed to read the working copy state".to_string(),
+                err: err.into(),
+            })?
+            .reset_to_empty();
+        self.tree_state_dirty = true;
+        Ok(())
+    }
+
     fn sparse_patterns(&self) -> Result<&[RepoPathBuf], WorkingCopyStateError> {
         self.wc.sparse_patterns()
     }

lib/src/working_copy.rs

@@ -107,6 +107,9 @@ pub trait LockedWorkingCopy {
     /// Update to another commit without touching the files in the working copy.
     fn reset(&mut self, commit: &Commit) -> Result<(), ResetError>;
 
+    /// Update to the empty tree without touching the files in the working copy.
+    fn reset_to_empty(&mut self) -> Result<(), ResetError>;
+
     /// See `WorkingCopy::sparse_patterns()`
     fn sparse_patterns(&self) -> Result<&[RepoPathBuf], WorkingCopyStateError>;