feat: Sync from noir

[AztecBot]

Automated pull of development from the noir programming language, a dependency of Aztec.
BEGIN_COMMIT_OVERRIDE
chore: let add_definition_location take a Location (noir-lang/noir#7185)
fix(LSP): correct signature for assert and assert_eq (noir-lang/noir#7184)
chore(experimental): Prevent enum panics by returning Options where possible instead of panicking (noir-lang/noir#7180)
feat(experimental): Construct enum variants in expressions (noir-lang/noir#7174)
feat: add noir-inspector (noir-lang/noir#7136)
fix: ensure canonical bits decomposition (noir-lang/noir#7168)
fix: Keep inc_rc for array inputs during preprocessing (noir-lang/noir#7163)
fix(docs): Update broken links to EC lib (noir-lang/noir#7141)
feat: inline simple functions (noir-lang/noir#7160)
feat(ssa): Expand feature set of the Brillig constraint check (noir-lang/noir#7060)
fix(ssa): Resolve value before fetching from DFG in a couple cases (noir-lang/noir#7169)
fix: Function::is_no_predicates always returned false for brillig f… (noir-lang/noir#7167)
chore(refactor): Remove globals field on Ssa object and use only the shared globals graph (noir-lang/noir#7156)
chore: let Function::inlined take a should_inline_call function (noir-lang/noir#7149)
chore: add compile-time assertions on generic arguments of stdlib functions (noir-lang/noir#6981)
fix: LSP hover over function with &mut self (noir-lang/noir#7155)
feat(brillig): Set global memory size at program compile time (noir-lang/noir#7151)
feat: LSP autocomplete module declaration (noir-lang/noir#7154)
feat(ssa): Reuse constants from the globals graph when making constants in a function DFG (noir-lang/noir#7153)
feat: LSP chain inlay hints (noir-lang/noir#7152)
chore: turn on overflow checks in CI rust tests (noir-lang/noir#7145)
fix(ssa): Use post order when mapping instructions in loop invariant pass (noir-lang/noir#7140)
fix: preserve types when reading from calldata arrays (noir-lang/noir#7144)
feat: Resolve enums & prepare type system (noir-lang/noir#7115)
feat: loop must have at least one break (noir-lang/noir#7126)
feat: parse globals in SSA parser (noir-lang/noir#7112)
fix: allow calling trait impl method from struct if multiple impls exist (noir-lang/noir#7124)
fix: avoid creating unnecessary memory blocks (noir-lang/noir#7114)
chore: relax threshold for reporting regressions (noir-lang/noir#7130)
fix: proper cleanup when breaking from comptime loop on error (noir-lang/noir#7125)
fix: Prevent overlapping associated types impls (noir-lang/noir#7047)
feat: unconstrained optimizations for BoundedVec (noir-lang/noir#7119)
chore: mark libs good (noir-lang/noir#7123)
chore: remove comments for time/memory benchmarks (noir-lang/noir#7121)
fix: don't always use an exclusive lock in nargo check (noir-lang/noir#7120)
feat(ssa): Pass to preprocess functions (noir-lang/noir#7072)
chore: Formatting issues / minor errors in the docs (noir-lang/noir#7105)
fix: defunctionalize pass on the caller runtime to apply (noir-lang/noir#7100)
feat: Parser and formatter support for enums (noir-lang/noir#7110)
feat(brillig): SSA globals code gen (noir-lang/noir#7021)
feat: loop keyword in runtime and comptime code (noir-lang/noir#7096)
chore: Add benchmarking dashboard (noir-lang/noir#7068)
feat(experimental): try to infer lambda argument types inside calls (noir-lang/noir#7088)
feat(ssa): Add flag to DIE pass to be able to keep store instructions (noir-lang/noir#7106)
chore: Cookbook Onboard integration (noir-lang/noir#7044)
chore: lock to ubuntu 22.04 (noir-lang/noir#7098)
fix: Remove unused brillig functions (noir-lang/noir#7102)
chore(ssa): Use correct prefix when printing array values in global space (noir-lang/noir#7095)
END_COMMIT_OVERRIDE

[aakoshh]

Managed to reproduce this failure on the mainframe as follows:

Build the system:

./bootstrap.sh clean
git fetch origin
git checkout --track origin/sync-noir
./bootstrap.sh full
#npm install --global corepack --force
#yarn set version 4.5.2; yarn install

It changed since I last tried and takes a long time, but succeeded.

Then we need to build images and run the test that ci runs; luckily there is bootstrap entry point for it, which also takes care of building the necessary docker images:

./bootstrap.sh test-e2e -- e2e_fees_failures

Alternatively the images can be built as follows:

./bootstrap.sh image-e2e

This shows the same failure as on CI. After this when we change something in the Noir code under noir/noir-repo, we can run ./bootstrap.sh again to rebuild nargo and all the contract and protocol circuits, which unfortunately takes a really long time, without indicating what's happening (haven't found a way to enable verbose logging yet); it's not yet clear what is the minimum that needs to be rebuilt. Altogether it was about 15 minutes for another test run to start (noir-projects/joblog.txt contains timings for the 3 contract directories).

Now, when we run ./bootstrap.sh image-e2e again it says the image already exists, because it uses the git hash to determine the tag, which has not changed (we're using the image pulled from CI). To trigger a change we need to make a git commit, and then rebuild the images locally.

Unfortunately this doesn't work, complaining that the commit wasn't pushed:

Earthly error

================================== ❌ FAILURE ===================================

  +bootstrap-noir-bb *failed* | Repeating the failure error...
  +bootstrap-noir-bb *failed* | --> RUN --raw-output bash -c "$bootstrap_noir_bb"
  +bootstrap-noir-bb *failed* | The commit was not pushed, run aborted.
  +bootstrap-noir-bb *failed* | ERROR Earthfile:32:2
  +bootstrap-noir-bb *failed* |       The command
  +bootstrap-noir-bb *failed* |           RUN --raw-output bash -c "$bootstrap_noir_bb"
  +bootstrap-noir-bb *failed* |       did not complete successfully. Exit code 1
    +bootstrap-aztec | apply build +bootstrap-with-verifier: earthfile2llb for +bootstrap-with-verifier: Earthfile:68:2 copy artifact: apply build +rollup-verifier-contract-with-cache: earthfile2llb for +rollup-verifier-contract-with-cache: Earthfile:268:2 failed to expand LET value "rollup-verifier-contract-$(./noir-projects/bootstrap.sh hash).tar.gz": failed to process "rollup-verifier-contract-$(./noir-projects/bootstrap.sh hash).tar.gz": non constant build arg read request: process "/bin/sh -c CARGO_HOME=/opt/rust/cargo EARTHLY_GIT_HASH=ebb07743425a702b0213e4ccd389aa617a9cdf1a GITHUB_LOG=1 PATH=/opt/rust/cargo/bin:/opt/foundry/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin RUSTUP_HOME=/opt/rust/rustup bootstrap_noir_bb='rm -rf $(ls -A) &&\n    git init >/dev/null 2>&1 &&\n    git remote add origin https://github.com/aztecprotocol/aztec-packages >/dev/null 2>&1 &&\n    (git fetch --depth 1 origin $EARTHLY_GIT_HASH >/dev/null 2>&1 || (echo \"The commit was not pushed, run aborted.\" && exit 1)) &&\n    git reset --hard FETCH_HEAD >/dev/null 2>&1 &&\n    ./build-images/adhoc-installs.sh &&\n    DENOISE=1 CI=1 ./noir/bootstrap.sh fast &&\n    DENOISE=1 CI=1 ./barretenberg/bootstrap.sh fast &&\n    mv $(ls -A) /usr/src' AWS_ACCESS_KEY_ID=\"$(cat /run/secrets/AWS_ACCESS_KEY_ID)\" AWS_SECRET_ACCESS_KEY=\"$(cat /run/secrets/AWS_SECRET_ACCESS_KEY)\" /usr/bin/earth_debugger /bin/sh -c 'bash -c \"$bootstrap_noir_bb\"'" did not complete successfully: exit code: 1
    +bootstrap-aztec | in		github.com/AztecProtocol/aztec-packages:sync-noir+rollup-verifier-contract-with-cache
    +bootstrap-aztec | in		github.com/AztecProtocol/aztec-packages:sync-noir+bootstrap-with-verifier

Help: To debug your build, you can use the --interactive (-i) flag to drop into a shell of the failing RUN step: "earthly -i --secret AWS_ACCESS_KEY_ID=XXXXX --secret AWS_SECRET_ACCESS_KEY=XXXXX --secret S3_BUILD_CACHE_UPLOAD=XXXXX --secret S3_BUILD_CACHE_DOWNLOAD=XXXXX --secret AZTEC_BOT_COMMENTER_GITHUB_TOKEN=XXXXX --artifact +bootstrap-aztec/usr/src /tmp/tmp.LWZns6APgS/usr/src"

🛰️ Reuse cache between CI runs with Earthly Satellites! 2-20X faster than without cache. Generous free tier https://cloud.earthly.dev

Trying again after creating a branch for my investigations:

git add noir
git commit -m "DEBUG"
git checkout -b af/sync-noir-debug
git push --set-upstream origin af/sync-noir-debug

This time it does the build, but it looks like it's running ./bootstrap.sh full inside the Docker build process again 😣

Okay so either I figure out a different way of running the test script directly against a running container, or use this workflow, but it's less likely that I can limit the build if it has to rebuild an e2e image from scratch. FWIW according to e2e_test_config.yml I have to run e2e_fees/failures.test.ts; the way the docker image does it is here.

EDIT: It's been building the image for over 30 minutes now, at the moment creating the VK for the second protocol contract, which seem to take ages. I used to be able to run these; maybe at the time it ran natively using more CPUs. I'll check it in the morning.

EDIT2: After about 10 more minutes the test ran. I can confirm that disabling the Ssa::preprocess_fns pass the tests do not fail.

[aakoshh]

Found the way to execute the test without involving the image-e2e step, which saves the 30 minute rebuild of the images after each change:

cd yarn-project/end-to-end
test_name=e2e_fees_failures
test_path=$(yq e ".tests.${test_name}.test_path" "scripts/e2e_test_config.yml")
export LOG_LEVEL=debug
yarn test:e2e $test_path -t "fails transaction that error in setup"  

This runs the failure tests, which uses the FeesTest fixture to set up the test, including the SnapshotManager that starts Anvil and deploys L1 contracts, so this particular test doesn't have external dependencies such as docker compose (unlike some others).

This way we can rebuild after a change with the following commands:

./noir/bootstrap.sh
./noir-projects/bootstrap.sh
./yarn-project/bootstrap.sh

These take 5, 10 and 2 minutes, respectively, so ideally we should still narrow it down to the contract that fails.

[TomAFrench]

I've disabled the sync workflow so that this branch is stable while we debug.

[aakoshh]

With export LOG_LEVEL=debug I saw that the error isn't coming from the execution of the user contract:

[12:08:11.407] DEBUG: simulator:acvm Oracle callback callPrivateFunction
[12:08:11.407] DEBUG: simulator:client_execution_context Calling private function 0x0000000000000000000000000000000000000000000000000000000000000004:0xcd9728af from 0x0000000000000000000000000000000000000000000000000000000000000004
[12:08:11.440] VERBOSE: simulator:private_execution Executing private function SchnorrAccount:constructor {"contract":"0x09e77cf1e2ad3aa1792c300eaf73615b6ba81543c87f3cff2fa0b8d27b643130"}
[12:08:11.461] DEBUG: simulator:acvm Oracle callback getContractInstance
[12:08:11.464] DEBUG: simulator:acvm Oracle callback notifyCreatedNullifier
[12:08:11.467] DEBUG: simulator:acvm Oracle callback notifyCreatedNote
[12:08:11.467] DEBUG: simulator:acvm Oracle callback debugLog
[12:08:11.474] DEBUG: simulator:acvm Oracle callback getRandomField
[12:08:11.475] DEBUG: simulator:acvm Oracle callback getRandomField
[12:08:11.475] DEBUG: simulator:acvm Oracle callback getRandomField
[12:08:11.475] DEBUG: simulator:acvm Oracle callback getRandomField
[12:08:11.475] DEBUG: simulator:acvm Oracle callback getRandomField
[12:08:11.476] DEBUG: simulator:acvm Oracle callback getRandomField
[12:08:11.476] DEBUG: simulator:acvm Oracle callback getRandomField
[12:08:11.476] DEBUG: simulator:acvm Oracle callback getRandomField
[12:08:11.476] DEBUG: simulator:acvm Oracle callback getRandomField
[12:08:11.476] DEBUG: simulator:acvm Oracle callback getRandomField
[12:08:11.477] DEBUG: simulator:acvm Oracle callback getRandomField
[12:08:11.477] DEBUG: simulator:acvm Oracle callback getRandomField
[12:08:11.477] DEBUG: simulator:acvm Oracle callback getRandomField
[12:08:11.477] DEBUG: simulator:acvm Oracle callback getRandomField
[12:08:11.478] DEBUG: simulator:acvm Oracle callback getRandomField
[12:08:11.502] DEBUG: simulator:acvm Oracle callback getIndexedTaggingSecretAsSender
[12:08:11.514] DEBUG: simulator:acvm Oracle callback notifyCreatedNullifier
[12:08:11.515] DEBUG: simulator:acvm Oracle callback notifyCreatedNullifier
[12:08:11.542] DEBUG: simulator:private_execution Ran external function 0x0b09fc3f35c38e157bc9f3ebe6bc2d5c69260b26a8d4766cefdaff48cd5579bf:0xcd9728af {"circuitName":"app-circuit","duration":883.6424932479858,"eventName":"circuit-witness-generation","inputSize":1344,"outputSize":17993,"appCircuitName":"SchnorrAccount:constructor"}
[12:08:11.545] DEBUG: simulator:private_execution Ran external function 0x123c3f390bfb61a1d831a90a39bdfcfdaf0d8879605deb0b1d67ff06dd8fc998:0xcd9728af {"circuitName":"app-circuit","duration":887.01926612854,"eventName":"circuit-witness-generation","inputSize":1344,"outputSize":17993,"appCircuitName":"SchnorrAccount:constructor"}
[12:08:11.546] DEBUG: simulator:private_execution Returning from call to 0x0b09fc3f35c38e157bc9f3ebe6bc2d5c69260b26a8d4766cefdaff48cd5579bf:0xcd9728af
[12:08:11.546] DEBUG: simulator:private_execution Returning from call to 0x123c3f390bfb61a1d831a90a39bdfcfdaf0d8879605deb0b1d67ff06dd8fc998:0xcd9728af
[12:08:11.585] DEBUG: simulator:private_execution Ran external function 0x0000000000000000000000000000000000000000000000000000000000000004:0xea92cc40 {"circuitName":"app-circuit","duration":1177.359474182129,"eventName":"circuit-witness-generation","inputSize":1952,"outputSize":17993,"appCircuitName":"MultiCallEntrypoint:entrypoint"}
[12:08:11.588] DEBUG: simulator:private_execution Ran external function 0x0000000000000000000000000000000000000000000000000000000000000004:0xea92cc40 {"circuitName":"app-circuit","duration":1185.167963027954,"eventName":"circuit-witness-generation","inputSize":1952,"outputSize":17993,"appCircuitName":"MultiCallEntrypoint:entrypoint"}
[12:08:11.588] DEBUG: simulator:private_execution Returning from call to 0x0000000000000000000000000000000000000000000000000000000000000004:0xea92cc40
[12:08:11.588] DEBUG: simulator:private_execution Returning from call to 0x0000000000000000000000000000000000000000000000000000000000000004:0xea92cc40
[12:08:11.589] DEBUG: pxe:service Private simulation completed for 0x0000000000000000000000000000000000000000000000000000000000000004:entrypoint
[12:08:11.589] DEBUG: pxe:service Private simulation completed for 0x0000000000000000000000000000000000000000000000000000000000000004:entrypoint

Instead we fail to simulate the kernel proof. With some extra logging added:

[12:08:11.591] DEBUG: pxe:service Executing kernel prover (simulate: true, profile: false, dryRun: true)...
[12:08:11.613] DEBUG: pxe:kernel-prover Calling private kernel init with isPrivateOnly true and firstNullifierHint 0x094d50d426951f38f96a1f237e60171ada982d3d759d3a5eda7f2be3ebdea3fc {"simulate":true}
[12:08:11.739] DEBUG: bb-prover:wasm:bundle Simulated PrivateKernelInitArtifact {"eventName":"circuit-simulation","circuitName":"private-kernel-init","duration":90.9289779663086,"inputSize":23650,"outputSize":55856}
[12:08:11.883] DEBUG: bb-prover:wasm:bundle Failed to simulate PrivateKernelInnerArtifact {"circuitName":"private-kernel-inner","witness":{}}
    error: {
      "callStack": [
        "0.2656",
        "0.7920",
        "0.16878"
      ],
      "rawAssertionPayload": {
        "selector": "2920182694213909827",
        "data": []
      },
      "brilligFunctionId": 0
    }
[12:08:11.892] ERROR: pxe:service Error: Error: Circuit execution failed (simulation_provider): attempt to subtract with overflow
    at Object.<anonymous>.module.exports.__wbg_constructor_a10f2b77c63b8d5e (/mnt/user-data/akosh/aztec-packages/noir/packages/acvm_js/nodejs/acvm_js.js:675:17)
    at acvm_js::js_execution_error::JsExecutionError::new::h8a9de55c9a82dc5a (wasm://wasm/00939eea:1:1193712)
    at acvm_js::execute::ProgramExecutor<B>::execute_circuit::{{closure}}::h9b9eb41499ff1a7b (wasm://wasm/00939eea:1:422002)
    at acvm_js::execute::execute_program_with_native_program_and_return::{{closure}}::h2efe214fe0bd0eb8 (wasm://wasm/00939eea:1:971037)
    at acvm_js::execute::execute_program_with_native_type_return::{{closure}}::h6f9a02ef14028ff7 (wasm://wasm/00939eea:1:1133753)
    at wasm_bindgen_futures::future_to_promise::{{closure}}::{{closure}}::hf6a5876861a1a114 (wasm://wasm/00939eea:1:884387)
    at wasm_bindgen_futures::queue::Queue::new::{{closure}}::hf077634dad17addb (wasm://wasm/00939eea:1:1235773)
    at wasm_bindgen::convert::closures::invoke1_mut::h7f1511a2915e72e7 (wasm://wasm/00939eea:1:1573526)
    at __wbg_adapter_22 (/mnt/user-data/akosh/aztec-packages/noir/packages/acvm_js/nodejs/acvm_js.js:220:10)
    at real (/mnt/user-data/akosh/aztec-packages/noir/packages/acvm_js/nodejs/acvm_js.js:205:20) {
  callStack: [ '0.2656', '0.7920', '0.16878' ],
  rawAssertionPayload: { selector: '2920182694213909827', data: [] },
  brilligFunctionId: 0
}

I tried to log the input witness in bb_private_kernel_prover like this:

    const outputWitness = await this.simulationProvider
      .executeProtocolCircuit(witnessMap, compiledCircuit)
      .catch((err: Error) => {
        this.log.debug(`Failed to simulate ${circuitType}`, {
          circuitName: mapProtocolArtifactNameToCircuitName(circuitType),
          witness: witnessMap,
          error: err,
        });
        throw err;
      });

Unfortunately it shows up as empty in the logs, not sure if that is the problem. It's called from the kernel_prover

(To iterate these it's enough to run ./yarn-project/bootstrap.sh which is relatively quick).

[aakoshh]

Logging the input witnesses with console.log shows it's not empty (attached).
inner-witness.txt

[aakoshh]

Here it is in JSON as well.
inner-witness.json

It was captured like so:

    console.log(
      'WITNESS INPUTS',
      JSON.stringify(inputs, (_, v) => (typeof v === 'bigint' ? v.toString() : v)),
    );

NB the data can be captured during the run if the AZTEC_GENERATE_TEST_DATA env var is true, and it could be saved in the test like this, but it's an ad-hoc thing, and because in our case the test fails, we might as well capture it where it's emitted.

I'll tweak nargo execute to accept JSON and try to run it.

[aakoshh]

Just found one reason I have been barking up the wrong tree: I never paid attention to the fact that bb_private_kernel_prover calls getSimulatedClientArtifactByName, because in the logs it appeared as {"circuitName":"private-kernel-inner"}, rather than private-kernel-inner-simulated which is another circuit. However if we look in bundle.ts that is exactly what is returned.

I got suspicious when I started logging the hash of the circuit which did not match what I see in private-kernel-inner.json.

What is strange it that it also doesn't fully match what's in private-kernel-inner-simulated.json:

[10:48:30.233] DEBUG: wasm-simulator execution failed {"hash":7496887466397766000,"message":"Circuit execution failed: attempt to subtract with overflow"}

The JSON file hash is almost the same:

{"noir_version":"1.0.0-beta.1+c88df77ea72b3018","hash":7496887466397765668,...

I suspect the number is too large for JavaScript and gets rounded, but this questions the use of numbers for hashes.

[TomAFrench]

I think it questions the use of javascript but good flag.

[aakoshh]

Fair enough 🤷
For the record in Noir this is ProgramArtifact::hash which is u64 while on the TypeScript side it's NoirCompiledCircuit.hash.

Just thought with the recent addition of printing hashes it can be frustrating if it doesn't match.

[aakoshh]

Finally 🎉 😌

❯ cargo run -q -p nargo_cli -- --program-dir ../../noir-projects/noir-protocol-circuits execute --silence-warnings --package private_kernel_inner_simulated
error: Assertion failed: attempt to subtract with overflow
   ┌─ /Users/aakoshh/Work/aztec/aztec-packages/noir-projects/noir-protocol-circuits/crates/private-kernel-lib/src/private_kernel_inner.nr:67:71
   │
67 │             self.previous_kernel.public_inputs.end.private_call_stack[private_call_stack_size - 1];
   │                                                                       ---------------------------
   │
   = Call stack:
     1. /Users/aakoshh/Work/aztec/aztec-packages/noir-projects/noir-protocol-circuits/crates/private-kernel-inner-simulated/src/main.nr:19:5
     2. /Users/aakoshh/Work/aztec/aztec-packages/noir-projects/noir-protocol-circuits/crates/private-kernel-lib/src/private_kernel_inner.nr:67:71

Failed assertion

The execution uses the JSON inputs from above ☝️

For the record the differences between the normal and the simulated circuits are:

• main is unconstrained in simulated
• previous_kernel_public_inputs and app_public_inputs are not using call_data in simulated

[TomAFrench]

Awesome! I can confirm that those inputs work correctly on master but fail on this branch.

[aakoshh]

So far I have not succeeded in creating an isolated test case (I tried by getting rid of most fields apart from the one involved in the exception, but kept the structs involved in processing it). Instead I'm cutting away code from the original circuit to shorten the SSA.

Here's one where execute has been reduced to this:

pub fn execute(self) -> PrivateKernelCircuitPublicInputs {
        let output = unsafe { self.generate_output() };
        let private_call_stack_size = array_length(self.previous_kernel.public_inputs.end.private_call_stack);
        assert_eq(private_call_stack_size, 1);
        output
}

My suspicion is that generate_output modifies the original array, rather than a copy.

Here's the SSA where it fails:
failure-11249.ssa.txt

And here it is where preprocess_fns returns immediately, and the whole thing succeeds:
success-11249.ssa.txt

I'll try to look for anything suspicious, and to reproduce this as a regression test.

[aakoshh]

Trying to figure out what goes on with the preprocessing pass.

Before it, we have this, with inc_rc clearly visible when we turn the call stack array into a BoundedVec. I assume because of these, when an item is popped off the vector later, it realises that the ref-count is larger than 1 and makes a copy:

brillig(inline) fn new_from_previous_kernel f19 {
  // Pass the previous call stack to the function
  b0(..., v51: [(Field, Field, u32, u1, Field, Field, u32, u32); 8], ...):
    // Allocate empty data structures
    ..., v121, v122, ... = call f20() -> (..., [(Field, Field, u32, u1, Field, Field, u32, u32); 8], u32, ...)
    ...
    // IncRC for the empty call stack
    inc_rc v121
    // Allocate a reference for the call stack
    v191 = allocate -> &mut [(Field, Field, u32, u1, Field, Field, u32, u32); 8]
    // Store the empty call stack at the reference
    store v121 at v191
    // Allocate reference for the length of the non-empty data
    v192 = allocate -> &mut u32
    store v122 at v192    
    ...
    // Call `array_to_bounded_vec` with the input array
    v272, v273 = call f21(v51) -> ([(Field, Field, u32, u1, Field, Field, u32, u32); 8], u32)
    // IncRC on the returned storage array
    inc_rc v272
    ...
    // Store the new array at the reference
    store v272 at v191 
    store v273 at v192
    ...
    // Load the reference
    v334 = load v191 -> [(Field, Field, u32, u1, Field, Field, u32, u32); 8]
    v335 = load v192 -> u32
    ...
    // One more IncRC on the array
    inc_rc v334
    // Return the bounded vec along the data
    return ..., v334, v335, ...
}
brillig(inline) fn array_length f4 {
  b0(v0: [(Field, Field, u32, u1, Field, Field, u32, u32); 8]):
    ...
}
brillig(inline) fn array_to_bounded_vec f21 {
  b0(v0: [(Field, Field, u32, u1, Field, Field, u32, u32); 8]):
    // Call `array_length` to figure out the first non-empty data
    v2 = call f4(v0) -> u32
    v4, v5 = call f22(v0, v2) -> ([(Field, Field, u32, u1, Field, Field, u32, u32); 8], u32)
    return v4, v5
}
brillig(inline) fn from_parts_unchecked f22 {
  b0(v0: [(Field, Field, u32, u1, Field, Field, u32, u32); 8], v1: u32):
    // Assert that `!(8 < length)`, ie. `length <= max_length`
    v3 = lt u32 8, v1
    v4 = not v3
    constrain v3 == u1 0
    // IncRC on the storage
    inc_rc v0
    return v0, v1
}
brillig(inline) fn empty f20 {
  b0():
    ...
}

In the preprocessed version, we have the following SSA, which already completely inlines the above functions. Notable, however, are the complete absence of inc_rc:

brillig(inline) fn new_from_previous_kernel f9 {
  b0(... v51: [(Field, Field, u32, u1, Field, Field, u32, u32); 8], ...):
    // Some empty data that never gets touched
    ...
    // Then the inlined version of this loop from `find_index_hint`
    // let mut index = N;
    // for i in 0..N {
    //     // We check `index == N` to ensure that we only update the index if we haven't found a match yet.
    //     if (index == N) & find(array[i]) {
    //         index = i;
    //     }
    // }
    // index
    ...
    // Allocate a reference for the payload length, starting with 8
    v74 = allocate -> &mut u32
    store u32 8 at v74
    // Start a loop from 0
    jmp b1(u32 0)
  b1(v60: u32):
    // If the loop index is less than 8, go to b3 else b2 (the return block)
    v76 = lt v60, u32 8
    jmpif v76 then: b3, else: b2
  b2():
    // Load the payload length
    v77 = load v74 -> u32
    // assert !(8 < length), ie. `length <= max_length`
    v78 = lt u32 8, v77
    constrain v78 == u1 0
    // Return the original array as is, along with the payload length we determined
    return ..., v51, v77, ...
  b3():
    // Load the payload length
    v80 = load v74 -> u32
    // End condition, reading the length itself
    v81 = eq v80, u32 8
    // Read the 0th, 1st, 2nd as we loop through v60 (parameter of b1)
    v82 = unchecked_mul v60, u32 8
    v83 = array_get v51, index v82 -> Field
    v85 = unchecked_add v82, u32 1
    v86 = array_get v51, index v85 -> Field
    v88 = unchecked_add v82, u32 2
    v89 = array_get v51, index v88 -> u32
    v91 = unchecked_add v82, u32 3
    v92 = array_get v51, index v91 -> u1
    v94 = unchecked_add v82, u32 4
    v95 = array_get v51, index v94 -> Field
    v97 = unchecked_add v82, u32 5
    v98 = array_get v51, index v97 -> Field
    v100 = unchecked_add v82, u32 6
    v101 = array_get v51, index v100 -> u32
    v103 = unchecked_add v82, u32 7
    v104 = array_get v51, index v103 -> u32
    // Check whether all fields are "is_empty"
    v105 = eq v83, Field 0
    v106 = eq v86, Field 0
    v107 = mul v105, v106
    v108 = eq u32 0, v89
    v109 = mul v107, v108
    v110 = not v92
    v111 = mul v109, v110
    v112 = eq v95, Field 0
    v113 = mul v111, v112
    v114 = eq v98, Field 0
    v115 = mul v113, v114
    v116 = eq v101, u32 0
    v117 = mul v115, v116
    v118 = eq v104, u32 0
    v119 = mul v117, v118
    // Only consider the `is_empty` result if the index is still 8
    v120 = mul v81, v119
    // If it's all empty (zero), then store the current loop index at the payload length in b4;
    // then increase the loop index and look at the next item in the array.
    jmpif v120 then: b4, else: b5
  b4():
    store v60 at v74
    jmp b5()
  b5():
    v121 = unchecked_add v60, u32 1
    jmp b1(v121)
}

I suspect that if we consider the return value a bounded vec, which can be popped, and we did not increment the ref-count of the underlying array, then it's understandable that the assertion that the data is still there will fail.

[aakoshh]

I also figured why I had difficulty reproducing this in a test: it did not process the pre-process the function because they were considered big, because main in the test was relatively small.

This is the circuit:

PREPROCESS new_from_previous_kernel: own=381 transitive=626 cutoff=74 skip=false
PREPROCESS main: own=15 transitive=1091 cutoff=74 skip=true

And this was the test:

PREPROCESS new_from_previous_kernel: own=19 transitive=123 cutoff=53 skip=true
PREPROCESS main: own=5 transitive=217 cutoff=53 skip=true

If I tweak the condition for preprocessing to be more permissive then the test can reproduce the issue as well.