workspace: recover from missing operation

If the operation corresponding to a workspace is missing for some reason
(the specific situation in the test in this commit is that an operation
was abandoned and garbage-collected from another workspace), currently,
jj fails with a 255 error code. Teach jj a way to recover from this
situation.

When jj detects such a situation, it prints a message and stops
operation, similar to when a workspace is stale. The message tells the
user what command to run.

When that command is run, jj loads the repo at the @ operation (instead
of the operation of the workspace), creates a new commit on the @
commit with an empty tree, and then proceeds as usual - in particular,
including the auto-snapshotting of the working tree, which creates
another commit that obsoletes the newly created commit.

There are several design points I considered.

1) Whether the recovery should be automatic, or (as in this commit)
manual in that the user should be prompted to run a command. The user
might prefer to recover in another way (e.g. by simply deleting the
workspace) and this situation is (hopefully) rare enough that I think
it's better to prompt the user.

2) Which command the user should be prompted to run (and thus, which
command should be taught to perform the recovery). I chose "workspace
update-stale" because the circumstances are very similar to it: it's
symptom is that the regular jj operation is blocked somewhere at the
beginning, and "workspace update-stale" already does some special work
before the blockage (this commit adds more of such special work). But it
might be better for something more explicitly named, or even a sequence
of commands (e.g. "create a new operation that becomes @ that no
workspace points to", "low-level command that makes a workspace point to
the operation @") but I can see how this can be unnecessarily confusing
for the user.

3) How we recover. I can think of several ways:
a) Always create a commit, and allow the automatic snapshotting to
create another commit that obsoletes this commit.
b) Create a commit but somehow teach the automatic snapshotting to
replace the created commit in-place (so it has no predecessor, as viewed
in "obslog").
c) Do either a) or b), with the added improvement that if there is no
diff between the newly created commit and the former @, to behave as if
no new commit was created (@ remains as the former @).
I chose a) since it was the simplest and most easily reasoned about,
which I think is the best way to go when recovering from a rare
situation.

cli/examples/custom-working-copy/main.rs

@@ -241,6 +241,10 @@ impl LockedWorkingCopy for LockedConflictsWorkingCopy {
         self.inner.reset(new_tree)
     }
 
+    fn reset_to_empty(&mut self) -> Result<(), ResetError> {
+        self.inner.reset_to_empty()
+    }
+
     fn sparse_patterns(&self) -> Result<&[RepoPathBuf], WorkingCopyStateError> {
         self.inner.sparse_patterns()
     }

cli/src/cli_util.rs

@@ -1396,6 +1396,14 @@ See https://github.com/martinvonz/jj/blob/main/docs/working-copy.md#stale-workin
                         short_operation_hash(&old_op_id)
                     )));
                 }
+                WorkingCopyFreshness::MissingOperation => {
+                    return Err(user_error_with_hint(
+                        "Could not read working copy's operation.",
+                        "Run `jj workspace update-stale` to recover.
+See https://github.com/martinvonz/jj/blob/main/docs/working-copy.md#stale-working-copy \
+                         for more information.",
+                    ));
+                }
             };
         self.user_repo = ReadonlyUserRepo::new(repo);
         let progress = crate::progress::snapshot_progress(ui);
@@ -1901,6 +1909,8 @@ pub enum WorkingCopyFreshness {
     WorkingCopyStale,
     /// The working copy is a sibling of the latest operation.
     SiblingOperation,
+    /// The working copy's operation could not be loaded.
+    MissingOperation,
 }
 
 #[instrument(skip_all)]
@@ -1915,9 +1925,13 @@ pub fn check_stale_working_copy(
         // The working copy isn't stale, and no need to reload the repo.
         Ok(WorkingCopyFreshness::Fresh)
     } else {
-        let wc_operation_data = repo
-            .op_store()
-            .read_operation(locked_wc.old_operation_id())?;
+        let wc_operation_data = match repo.op_store().read_operation(locked_wc.old_operation_id()) {
+            Ok(o) => o,
+            Err(OpStoreError::ObjectNotFound { .. }) => {
+                return Ok(WorkingCopyFreshness::MissingOperation)
+            }
+            Err(e) => return Err(e),
+        };
         let wc_operation = Operation::new(
             repo.op_store().clone(),
             locked_wc.old_operation_id().clone(),

cli/src/commands/workspace.rs

@@ -15,21 +15,22 @@
 use std::fmt::Debug;
 use std::fs;
 use std::io::Write;
+use std::sync::Arc;
 
 use clap::Subcommand;
 use itertools::Itertools;
 use jj_lib::file_util;
 use jj_lib::object_id::ObjectId;
 use jj_lib::op_store::WorkspaceId;
 use jj_lib::operation::Operation;
-use jj_lib::repo::Repo;
+use jj_lib::repo::{ReadonlyRepo, Repo};
 use jj_lib::rewrite::merge_commit_trees;
 use jj_lib::workspace::Workspace;
 use tracing::instrument;
 
 use crate::cli_util::{
-    self, check_stale_working_copy, print_checkout_stats, user_error, CommandError, CommandHelper,
-    RevisionArg, WorkingCopyFreshness, WorkspaceCommandHelper,
+    self, check_stale_working_copy, print_checkout_stats, short_commit_hash, user_error,
+    CommandError, CommandHelper, RevisionArg, WorkingCopyFreshness, WorkspaceCommandHelper,
 };
 use crate::ui::Ui;
 
@@ -280,20 +281,73 @@ fn cmd_workspace_root(
     Ok(())
 }
 
+fn create_and_check_out_recovery_commit(
+    command: &CommandHelper,
+    ui: &mut Ui,
+    workspace: Workspace,
+) -> Result<Arc<ReadonlyRepo>, CommandError> {
+    let operation = command.resolve_operation(ui, workspace.repo_loader())?;
+    let repo = workspace.repo_loader().load_at(&operation)?;
+    let workspace_id = workspace.workspace_id().clone();
+    let tree_id = repo.store().empty_merged_tree_id();
+
+    let mut workspace_command = command.for_loaded_repo(ui, workspace, repo.clone())?;
+    let (mut locked_workspace, commit) =
+        workspace_command.unchecked_start_working_copy_mutation()?;
+    let commit_id = commit.id();
+
+    let mut tx = repo.start_transaction(command.settings());
+    let mut_repo = tx.mut_repo();
+    let new_commit = mut_repo
+        .new_commit(command.settings(), vec![commit_id.clone()], tree_id.clone())
+        .write()?;
+    mut_repo.set_wc_commit(workspace_id, new_commit.id().clone())?;
+    let ret = Ok(tx.commit("recovery commit"));
+
+    locked_workspace.locked_wc().reset_to_empty()?;
+    locked_workspace.finish(operation.id().clone())?;
+
+    writeln!(
+        ui.stderr(),
+        "Created and checked out recovery commit {}",
+        short_commit_hash(new_commit.id())
+    )?;
+
+    ret
+}
+
 /// Loads workspace that will diverge from the last working-copy operation.
 fn for_stale_working_copy(
     command: &CommandHelper,
     ui: &mut Ui,
-) -> Result<WorkspaceCommandHelper, CommandError> {
+) -> Result<(WorkspaceCommandHelper, bool), CommandError> {
     let workspace = command.load_workspace()?;
     let op_store = workspace.repo_loader().op_store();
-    let op_id = workspace.working_copy().operation_id();
-    let op_data = op_store
-        .read_operation(op_id)
-        .map_err(|e| CommandError::InternalError(format!("Failed to read operation: {e}")))?;
-    let operation = Operation::new(op_store.clone(), op_id.clone(), op_data);
-    let repo = workspace.repo_loader().load_at(&operation)?;
-    command.for_loaded_repo(ui, workspace, repo)
+    let (repo, recovered) = {
+        let op_id = workspace.working_copy().operation_id();
+        match op_store.read_operation(op_id) {
+            Ok(op_data) => (
+                workspace.repo_loader().load_at(&Operation::new(
+                    op_store.clone(),
+                    op_id.clone(),
+                    op_data,
+                ))?,
+                false,
+            ),
+            Err(e) => {
+                writeln!(
+                    ui.stderr(),
+                    "Failed to read working copy's current operation; attempting recovery. Error \
+                     message from read attempt: {e}"
+                )?;
+                (
+                    create_and_check_out_recovery_commit(command, ui, command.load_workspace()?)?,
+                    true,
+                )
+            }
+        }
+    };
+    Ok((command.for_loaded_repo(ui, workspace, repo)?, recovered))
 }
 
 #[instrument(skip_all)]
@@ -306,22 +360,36 @@ fn cmd_workspace_update_stale(
     // operation, then merge the concurrent operations. The wc_commit_id of the
     // merged repo wouldn't change because the old one wins, but it's probably
     // fine if we picked the new wc_commit_id.
-    let known_wc_commit = {
-        let mut workspace_command = for_stale_working_copy(command, ui)?;
+    let (known_wc_commit, recovered) = {
+        let (mut workspace_command, recovered) = for_stale_working_copy(command, ui)?;
         workspace_command.maybe_snapshot(ui)?;
+
         let wc_commit_id = workspace_command.get_wc_commit_id().unwrap();
-        workspace_command.repo().store().get_commit(wc_commit_id)?
+        (
+            workspace_command.repo().store().get_commit(wc_commit_id)?,
+            recovered,
+        )
     };
     let mut workspace_command = command.workspace_helper_no_snapshot(ui)?;
 
     let repo = workspace_command.repo().clone();
     let (mut locked_ws, desired_wc_commit) =
         workspace_command.unchecked_start_working_copy_mutation()?;
     match check_stale_working_copy(locked_ws.locked_wc(), &desired_wc_commit, &repo)? {
-        WorkingCopyFreshness::Fresh | WorkingCopyFreshness::Updated(_) => writeln!(
-            ui.stderr(),
-            "Nothing to do (the working copy is not stale)."
-        )?,
+        WorkingCopyFreshness::Fresh | WorkingCopyFreshness::Updated(_) => {
+            if !recovered {
+                writeln!(
+                    ui.stderr(),
+                    "Nothing to do (the working copy is not stale)."
+                )?;
+            }
+        }
+        WorkingCopyFreshness::MissingOperation =>
+        // We should have already recovered from this situation above, so there must
+        // have been a concurrent operation.
+        {
+            return Err(user_error("Concurrent working copy operation. Try again."))
+        }
         WorkingCopyFreshness::WorkingCopyStale | WorkingCopyFreshness::SiblingOperation => {
             // The same check as start_working_copy_mutation(), but with the stale
             // working-copy commit.

cli/tests/test_workspaces.rs

@@ -384,6 +384,103 @@ fn test_workspaces_updated_by_other() {
     "###);
 }
 
+#[test]
+fn test_workspaces_current_op_discarded_by_other() {
+    let test_env = TestEnvironment::default();
+    // Use the local backend because GitBackend::gc() depends on the git CLI.
+    test_env.jj_cmd_ok(
+        test_env.env_root(),
+        &["init", "main", "--config-toml=ui.allow-init-native=true"],
+    );
+    let main_path = test_env.env_root().join("main");
+    let secondary_path = test_env.env_root().join("secondary");
+
+    std::fs::write(main_path.join("file"), "contents\n").unwrap();
+    test_env.jj_cmd_ok(&main_path, &["new"]);
+
+    test_env.jj_cmd_ok(&main_path, &["workspace", "add", "../secondary"]);
+
+    // Create an op by abandoning the parent commit. Importantly, that commit also
+    // changes the target tree in the secondary workspace.
+    test_env.jj_cmd_ok(&main_path, &["abandon", "@-"]);
+
+    let stdout = test_env.jj_cmd_success(
+        &main_path,
+        &[
+            "operation",
+            "log",
+            "--template",
+            r#"id.short(10) ++ " " ++ description"#,
+        ],
+    );
+    insta::assert_snapshot!(stdout, @r###"
+    @  6c22dbc43c abandon commit 479653b3216ad2af6a8ffcbe5885203b7bb3d0d49175ffd2be58a454da95493b0e8e44885ac4f7caacbeb79b407c66d68991f5a01666e976687c6732b3311780
+    ◉  72b8975c75 Create initial working-copy commit in workspace secondary
+    ◉  b816f120a4 add workspace 'secondary'
+    ◉  332e901b2a new empty commit
+    ◉  c07b7d7796 snapshot working copy
+    ◉  c5295044c8 add workspace 'default'
+    ◉  0c73edb541 initialize repo
+    ◉  0000000000
+    "###);
+
+    // Abandon ops, including the one the secondary workspace is currently on.
+    test_env.jj_cmd_ok(&main_path, &["operation", "abandon", "..@-"]);
+    test_env.jj_cmd_ok(&main_path, &["util", "gc", "--expire=now"]);
+
+    insta::assert_snapshot!(get_log_output(&test_env, &main_path), @r###"
+    @  3bebcd0f8335f735c00063290247d6c3332106261d0dd4810105906cdf29878990ee717690750ec203a7c4763a36968a0d9de799f46bc37fcf62dde13a52b73f default@
+    │ ◉  1d558d136f4d90ec2fe9beac8eb638eb509d0e378c4546f7b2634f93d0c86cfdd08b7efe4fd3aeb383f41fb7ebb2311a14754f617e71389c3aa9519c05c17dd8 secondary@
+    ├─╯
+    ◉  00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
+    "###);
+
+    let stderr = test_env.jj_cmd_failure(&secondary_path, &["st"]);
+    insta::assert_snapshot!(stderr, @r###"
+    Error: Could not read working copy's operation.
+    Hint: Run `jj workspace update-stale` to recover.
+    See https://github.com/martinvonz/jj/blob/main/docs/working-copy.md#stale-working-copy for more information.
+    "###);
+
+    let (stdout, stderr) = test_env.jj_cmd_ok(&secondary_path, &["workspace", "update-stale"]);
+    // Replace before asserting, since some platforms have different text to
+    // describe a missing file.
+    let stderr = regex::Regex::new(r"not found:.*\(")
+        .unwrap()
+        .replace(&stderr, "not found: REMOVED (");
+    insta::assert_snapshot!(stderr, @r###"
+    Failed to read working copy's current operation; attempting recovery. Error message from read attempt: Object 72b8975c7573303a258c292ad69f93dad1922532a2839000c217102da736ed7c6235872bdfa8c747a723df78547666686722d0d301da2d70d815798634c8eedc of type operation not found: REMOVED (os error 2)
+    Created and checked out recovery commit 84e67dd875e0
+    "###);
+    insta::assert_snapshot!(stdout, @"");
+
+    insta::assert_snapshot!(get_log_output(&test_env, &main_path), @r###"
+    ◉  be7e513bfb77cde552e363f947ecc5757557d0d24caf5c627eb9631f38215334722c31b4266c3627111210b1c5eb00f4cd7af4299510cb8c2fe39b405d31b45b secondary@
+    ◉  1d558d136f4d90ec2fe9beac8eb638eb509d0e378c4546f7b2634f93d0c86cfdd08b7efe4fd3aeb383f41fb7ebb2311a14754f617e71389c3aa9519c05c17dd8
+    │ @  3bebcd0f8335f735c00063290247d6c3332106261d0dd4810105906cdf29878990ee717690750ec203a7c4763a36968a0d9de799f46bc37fcf62dde13a52b73f default@
+    ├─╯
+    ◉  00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
+    "###);
+
+    let (stdout, stderr) = test_env.jj_cmd_ok(&secondary_path, &["st"]);
+    insta::assert_snapshot!(stderr, @"");
+    insta::assert_snapshot!(stdout, @r###"
+    Working copy changes:
+    A file
+    Working copy : znkkpsqq be7e513b (no description set)
+    Parent commit: pmmvwywv 1d558d13 (empty) (no description set)
+    "###);
+
+    let (stdout, stderr) = test_env.jj_cmd_ok(&secondary_path, &["obslog"]);
+    insta::assert_snapshot!(stderr, @"");
+    insta::assert_snapshot!(stdout, @r###"
+    @  znkkpsqq [email protected] 2001-02-03 04:05:16.000 +07:00 secondary@ be7e513b
+    │  (no description set)
+    ◉  znkkpsqq hidden [email protected] 2001-02-03 04:05:16.000 +07:00 84e67dd8
+       (empty) (no description set)
+    "###);
+}
+
 #[test]
 fn test_workspaces_update_stale_noop() {
     let test_env = TestEnvironment::default();

lib/src/local_working_copy.rs

@@ -197,6 +197,10 @@ impl FileStatesMap {
         .collect();
     }
 
+    fn clear(&mut self) {
+        self.data.clear();
+    }
+
     /// Returns read-only map containing all file states.
     fn all(&self) -> FileStates<'_> {
         FileStates::from_sorted(&self.data)
@@ -1436,6 +1440,12 @@ impl TreeState {
         self.tree_id = new_tree.id();
         Ok(())
     }
+
+    pub fn reset_to_empty(&mut self) -> () {
+        self.file_states.clear();
+        self.tree_id = self.store.empty_merged_tree_id();
+        ()
+    }
 }
 
 fn checkout_error_for_stat_error(err: std::io::Error, path: &Path) -> CheckoutError {
@@ -1752,6 +1762,18 @@ impl LockedWorkingCopy for LockedLocalWorkingCopy {
         Ok(())
     }
 
+    fn reset_to_empty(&mut self) -> Result<(), ResetError> {
+        self.wc
+            .tree_state_mut()
+            .map_err(|err| ResetError::Other {
+                message: "Failed to read the working copy state".to_string(),
+                err: err.into(),
+            })?
+            .reset_to_empty();
+        self.tree_state_dirty = true;
+        Ok(())
+    }
+
     fn sparse_patterns(&self) -> Result<&[RepoPathBuf], WorkingCopyStateError> {
         self.wc.sparse_patterns()
     }

lib/src/working_copy.rs

@@ -108,6 +108,9 @@ pub trait LockedWorkingCopy {
     /// Update to another tree without touching the files in the working copy.
     fn reset(&mut self, new_tree: &MergedTree) -> Result<(), ResetError>;
 
+    /// Update to the empty tree without touching the files in the working copy.
+    fn reset_to_empty(&mut self) -> Result<(), ResetError>;
+
     /// See `WorkingCopy::sparse_patterns()`
     fn sparse_patterns(&self) -> Result<&[RepoPathBuf], WorkingCopyStateError>;