Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Filesystem store breaking in error case #610

Closed
chrisstaite-menlo opened this issue Jan 17, 2024 · 6 comments
Closed

Filesystem store breaking in error case #610

chrisstaite-menlo opened this issue Jan 17, 2024 · 6 comments
Labels
bug Something isn't working

Comments

@chrisstaite-menlo
Copy link
Collaborator

I've reported this before as occurring when the disk had run out of space, but this time it was caused by the GrpcStore erroring:

  2024-01-17T12:57:31.564953Z ERROR nativelink_worker::local_worker: Error executing action: Error { code: Internal, messages: ["Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "Failed to receive data in filesystem store", "While processing with temp file \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/716a8cf840dc7db8307bfd544c26feb7bfcc5b5d6c0a1c7ef51b000000000000-104467\"", "---", "status: InvalidArgument, message: \"Chunk should never be EOF, expected None in this case : During first buf_channel::take()\", details: [], metadata: MetadataMap { headers: {} }", "While fetching message in GrpcStore::get_part()", "---", "Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "Failed to read data data buffer from slow store", "Failed to populate()", "---", "Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "for digest DigestInfo { size_bytes: 104467, hash: \"716a8cf840dc7db8307bfd544c26feb7bfcc5b5d6c0a1c7e4f71fb30f9d41feb\" }", "in download_to_directory : /root/.cache/turbo-cache/work/f8bc522601e6a694635dd7621859d5c27e5e0b58d50758edc62d83f8fc0e1a8b/set_by_reclient/a/gen/services/media_session/public/mojom", "in download_to_directory : /root/.cache/turbo-cache/work/f8bc522601e6a694635dd7621859d5c27e5e0b58d50758edc62d83f8fc0e1a8b/set_by_reclient/a/gen/services/media_session/public", "in download_to_directory : /root/.cache/turbo-cache/work/f8bc522601e6a694635dd7621859d5c27e5e0b58d50758edc62d83f8fc0e1a8b/set_by_reclient/a/gen/services/media_session", "in download_to_directory : /root/.cache/turbo-cache/work/f8bc522601e6a694635dd7621859d5c27e5e0b58d50758edc62d83f8fc0e1a8b/set_by_reclient/a/gen/services", "in download_to_directory : /root/.cache/turbo-cache/work/f8bc522601e6a694635dd7621859d5c27e5e0b58d50758edc62d83f8fc0e1a8b/set_by_reclient/a/gen", "in download_to_directory : /root/.cache/turbo-cache/work/f8bc522601e6a694635dd7621859d5c27e5e0b58d50758edc62d83f8fc0e1a8b/set_by_reclient/a", "in download_to_directory : /root/.cache/turbo-cache/work/f8bc522601e6a694635dd7621859d5c27e5e0b58d50758edc62d83f8fc0e1a8b/set_by_reclient"] }
    at nativelink-worker/src/local_worker.rs:295 on tokio-runtime-worker ThreadId(12)

  2024-01-17T12:57:32.117817Z ERROR nativelink_store::filesystem_store: Error { code: NotFound, messages: ["No such file or directory (os error 2)", "Failed to touch file in filesystem store \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/6afa9b9a2fc02b834bf5fa033d480e4413078b2e8e5208feaa1b000000000000-2866\""] }
    at nativelink-store/src/filesystem_store.rs:287 on tokio-runtime-worker ThreadId(25)

The tmp_path hashes don't match the error in the GrpcStore download, but something weird is going on.
@chrisstaite-menlo
Copy link
Collaborator Author

When this happens it totally breaks the store and the only way to fix it is to restart the service which is not ideal because it appears to be happening fairly regularly on my workers now.

@aaronmondal aaronmondal added the bug Something isn't working label Jan 17, 2024
@chrisstaite-menlo
Copy link
Collaborator Author

Something seems to be very badly wrong with the latest commits, I'm getting so many errors reading into my workers that they barely do any work.

  2024-01-17T16:52:09.007601Z ERROR nativelink_worker::local_worker: Error executing action: Error { code: Internal, messages: ["Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "Failed to receive data in filesystem store", "While processing with temp file \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/658785b562ec4159449feb5f642878771b5d7fbfb1c9b22ca508000000000000-145320\"", "---", "status: InvalidArgument, message: \"Chunk should never be EOF, expected None in this case : During first buf_channel::take()\", details: [], metadata: MetadataMap { headers: {} }", "While fetching message in GrpcStore::get_part()", "---", "Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "Failed to read data data buffer from slow store", "Failed to populate()", "---", "Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "for digest DigestInfo { size_bytes: 145320, hash: \"658785b562ec4159449feb5f642878771b5d7fbfb1c9b22c5bb2e18647fa6a2f\" }", "in download_to_directory : /root/.cache/turbo-cache/work/4589f9874b03183fc23cc9a24d87a996802749377e90c9629f24f9fd216e39d5/build/linux/debian_bullseye_amd64-sysroot/lib/x86_64-linux-gnu", "in download_to_directory : /root/.cache/turbo-cache/work/4589f9874b03183fc23cc9a24d87a996802749377e90c9629f24f9fd216e39d5/build/linux/debian_bullseye_amd64-sysroot/lib", "in download_to_directory : /root/.cache/turbo-cache/work/4589f9874b03183fc23cc9a24d87a996802749377e90c9629f24f9fd216e39d5/build/linux/debian_bullseye_amd64-sysroot", "in download_to_directory : /root/.cache/turbo-cache/work/4589f9874b03183fc23cc9a24d87a996802749377e90c9629f24f9fd216e39d5/build/linux", "in download_to_directory : /root/.cache/turbo-cache/work/4589f9874b03183fc23cc9a24d87a996802749377e90c9629f24f9fd216e39d5/build"] }
    at nativelink-worker/src/local_worker.rs:295 on tokio-runtime-worker ThreadId(27)

  2024-01-17T16:52:09.334453Z  INFO nativelink_worker::running_actions_manager: Worker Received Action: ActionInfo { command_digest: DigestInfo { size_bytes: 2936, hash: "12d22a42aa28b39403f3f2f4f9bb1019d31dd276a883a3b7deb688748dec630b" }, input_root_digest: DigestInfo { size_bytes: 243, hash: "6a18216580d4615263ea5c679ff40ba4073de78057adef63619a0eccb4196665" }, timeout: 120s, platform_properties: PlatformProperties { properties: {"label:action_large": Unknown("1"), "container-image": Unknown("docker://gcr.io/chops-public-images-prod/rbe/siso-chromium/linux@sha256:912808c295e578ccde53b0685bcd0d56c15d7a03e819dcce70694bfe3fdab35e"), "OSFamily": Unknown("Linux")} }, priority: 0, load_timestamp: SystemTime { tv_sec: 1705510329, tv_nsec: 333591395 }, insert_timestamp: SystemTime { tv_sec: 1705510329, tv_nsec: 336182562 }, unique_qualifier: ActionInfoHashKey { instance_name: "jenkins", digest: DigestInfo { size_bytes: 352, hash: "c89a9671bdf82c2123e719d03736b2b738f57a6e0766d6b6830f9664f2d1d7a2" }, salt: 0 }, skip_cache_lookup: true, digest_function: Sha256 }
    at nativelink-worker/src/running_actions_manager.rs:1597 on tokio-runtime-worker ThreadId(18)

  2024-01-17T16:52:09.388768Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/658785b562ec4159449feb5f642878771b5d7fbfb1c9b22cc108000000000000-145320"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(18)

  2024-01-17T16:52:09.388858Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/658785b562ec4159449feb5f642878771b5d7fbfb1c9b22cc208000000000000-145320"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(14)

  2024-01-17T16:52:09.391002Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/9525bb6f8bac6069b2758dd633542beafee15eebd91f4356c308000000000000-2664892"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(18)

  2024-01-17T16:52:09.391046Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e57352e2d3a3b3f0ed7a0423b101f59073d46781315be430c508000000000000-3001366"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(14)

  2024-01-17T16:52:09.391061Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/0e0d78de29a00e5e0edde7978ef320da82487697f9a6786fca08000000000000-5721104"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(25)

  2024-01-17T16:52:09.391152Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/fb2c57907b7b6874447b014fac08aee4203e01ab510f9bd6c408000000000000-9784456"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(25)

  2024-01-17T16:52:09.391171Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/809a7b253d64319d1728de5b6616afe1c95232afcb817aeed208000000000000-2350394"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(27)

  2024-01-17T16:52:09.391155Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/051b7e88a378c49956ce1fef4f3dd2ea1f14f713b9c125ffc608000000000000-895140"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(30)

  2024-01-17T16:52:09.391191Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/454b21758d93a0807502d829714ad87dcd975c273844383bd108000000000000-371354"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(14)

  2024-01-17T16:52:09.391500Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/0e83d02a0144c726f05de850b71165691e380bd90205d22dcf08000000000000-1870824"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(2)

  2024-01-17T16:52:09.391504Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/b3e84eb4df0963f3fa3a5b9b37691d10c9cd2dbb1ffd68f3ce08000000000000-959974"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(18)

  2024-01-17T16:52:09.391714Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/068b56f69f5fee6ae34e11a3ca6581b0ce245b637e513619c908000000000000-3257688"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(30)

  2024-01-17T16:52:09.391759Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/a08fd4ce4cd09767db71aa7063ad5ab8c5393182030becd8c808000000000000-9052480"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(30)

  2024-01-17T16:52:09.391787Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e69d7ec8d737a0bbabe3bbc4e107a5797737ebf78f04f149c708000000000000-3036384"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(8)

  2024-01-17T16:52:09.392081Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/74d32fa1cee4dbee1a9e58335037bfe78de7c45026d6b327b708000000000000-99350168"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(7)

  2024-01-17T16:52:09.392102Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/eb82871b59be5321513597bb47f1f2e811880d351e0c7643b608000000000000-1676776"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(26)

  2024-01-17T16:52:09.392131Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/42bf014c4b6583993ef2cd067ec908898463a103644f2bc6b808000000000000-191407912"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(30)

  2024-01-17T16:52:09.392342Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/5bdec358e0419d32a92502d3f4315ac333d71923d70a2ce6bb08000000000000-69842920"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(30)

  2024-01-17T16:52:09.392367Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/5bdec358e0419d32a92502d3f4315ac333d71923d70a2ce6ba08000000000000-69842920"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(26)

  2024-01-17T16:52:09.392455Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/5bdec358e0419d32a92502d3f4315ac333d71923d70a2ce6b908000000000000-69842920"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(7)

  2024-01-17T16:52:09.392571Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/23ea58fe8ac6087bdfcbda8bf326e467fc80b57a2dc28242bf08000000000000-4246962"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(26)

  2024-01-17T16:52:09.392599Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/f8e5a9795670b80b5a5c2dd26afcbb93b896622953dfc157cb08000000000000-1378184"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(30)

  2024-01-17T16:52:09.392618Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/95873fa2f7d97918b01c36b68b438d42728eee2e99fe0ef8bd08000000000000-4642258"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(30)

  2024-01-17T16:52:09.392829Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/0be228510b887f6b322d6e31d66d5b5c72b5c374a62cc44abc08000000000000-5732310"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(24)

  2024-01-17T16:52:09.392900Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/01ba92d9e40180cff8b2c9acf3fb62efdc8c3767aeca45efd008000000000000-2032782"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(26)

  2024-01-17T16:52:09.392902Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/cc0984db6769b70fa24b1f3016ce1c3a997b4c5426864983c008000000000000-1751162"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(24)

  2024-01-17T16:52:09.392925Z  INFO nativelink_worker::running_actions_manager: Worker Cleanup
    at nativelink-worker/src/running_actions_manager.rs:1100 on tokio-runtime-worker ThreadId(21)

  2024-01-17T16:52:09.393020Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/94221894ec061137827f163707f6df462eb25e91f467431dd308000000000000-967866"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(8)

  2024-01-17T16:52:09.393081Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/354e9319fa64bb1c440883918d0052c387e7683363ab3516cc08000000000000-1336184"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(8)

  2024-01-17T16:52:09.392928Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/2250bb46fabe1b3a54f85deec9d1499a30626a1a1c760d83be08000000000000-5405210"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(24)

  2024-01-17T16:52:09.393119Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/034e99dc22681f58ccd0ec3d32cadb4a98c0fbf4588b13decd08000000000000-882420"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(8)

  2024-01-17T16:52:09.395852Z ERROR nativelink_worker::local_worker: Error executing action: Error { code: Internal, messages: ["Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "Failed to receive data in filesystem store", "While processing with temp file \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/658785b562ec4159449feb5f642878771b5d7fbfb1c9b22cc108000000000000-145320\"", "---", "status: InvalidArgument, message: \"Chunk should never be EOF, expected None in this case : During first buf_channel::take()\", details: [], metadata: MetadataMap { headers: {} }", "While fetching message in GrpcStore::get_part()", "---", "Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "Failed to read data data buffer from slow store", "Failed to populate()", "---", "Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "for digest DigestInfo { size_bytes: 145320, hash: \"658785b562ec4159449feb5f642878771b5d7fbfb1c9b22c5bb2e18647fa6a2f\" }", "in download_to_directory : /root/.cache/turbo-cache/work/4589f9874b03183fc23cc9a24d87a996802749377e90c9629f24f9fd216e39d5/build/linux/debian_bullseye_amd64-sysroot/lib/x86_64-linux-gnu", "in download_to_directory : /root/.cache/turbo-cache/work/4589f9874b03183fc23cc9a24d87a996802749377e90c9629f24f9fd216e39d5/build/linux/debian_bullseye_amd64-sysroot/lib", "in download_to_directory : /root/.cache/turbo-cache/work/4589f9874b03183fc23cc9a24d87a996802749377e90c9629f24f9fd216e39d5/build/linux/debian_bullseye_amd64-sysroot", "in download_to_directory : /root/.cache/turbo-cache/work/4589f9874b03183fc23cc9a24d87a996802749377e90c9629f24f9fd216e39d5/build/linux", "in download_to_directory : /root/.cache/turbo-cache/work/4589f9874b03183fc23cc9a24d87a996802749377e90c9629f24f9fd216e39d5/build"] }
    at nativelink-worker/src/local_worker.rs:295 on tokio-runtime-worker ThreadId(21)

  2024-01-17T16:52:09.636101Z  INFO nativelink_worker::running_actions_manager: Worker Received Action: ActionInfo { command_digest: DigestInfo { size_bytes: 2583, hash: "817ea3a81cfc854e1f0ab6f2fadad77dcf066abdc97c7f9991cc25cfb0fa38ab" }, input_root_digest: DigestInfo { size_bytes: 243, hash: "5b2b20d2fc5d8d0ba67ae5e2a1f1e10a06494475b88cde15b61ef24e82c157f8" }, timeout: 120s, platform_properties: PlatformProperties { properties: {"OSFamily": Unknown("Linux"), "label:action_large": Unknown("1"), "container-image": Unknown("docker://gcr.io/chops-public-images-prod/rbe/siso-chromium/linux@sha256:912808c295e578ccde53b0685bcd0d56c15d7a03e819dcce70694bfe3fdab35e")} }, priority: 0, load_timestamp: SystemTime { tv_sec: 1705510329, tv_nsec: 635235314 }, insert_timestamp: SystemTime { tv_sec: 1705510329, tv_nsec: 516098578 }, unique_qualifier: ActionInfoHashKey { instance_name: "jenkins", digest: DigestInfo { size_bytes: 352, hash: "d75304840766722fa0e4c74c36868b1d380d9c783fb54b772de3c1d7979e65be" }, salt: 0 }, skip_cache_lookup: true, digest_function: Sha256 }
    at nativelink-worker/src/running_actions_manager.rs:1597 on tokio-runtime-worker ThreadId(27)

  2024-01-17T16:52:09.676312Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/658785b562ec4159449feb5f642878771b5d7fbfb1c9b22cdd08000000000000-145320"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(24)

  2024-01-17T16:52:09.677086Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/658785b562ec4159449feb5f642878771b5d7fbfb1c9b22cde08000000000000-145320"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(26)

  2024-01-17T16:52:09.678369Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/fb2c57907b7b6874447b014fac08aee4203e01ab510f9bd6db08000000000000-9784456"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(21)

  2024-01-17T16:52:09.678420Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e57352e2d3a3b3f0ed7a0423b101f59073d46781315be430dc08000000000000-3001366"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(8)

  2024-01-17T16:52:09.678487Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/9525bb6f8bac6069b2758dd633542beafee15eebd91f4356da08000000000000-2664892"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(24)

  2024-01-17T16:52:09.678635Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/051b7e88a378c49956ce1fef4f3dd2ea1f14f713b9c125ffe708000000000000-895140"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(24)

  2024-01-17T16:52:09.678660Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/a08fd4ce4cd09767db71aa7063ad5ab8c5393182030becd8e608000000000000-9052480"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(24)

  2024-01-17T16:52:09.678715Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/454b21758d93a0807502d829714ad87dcd975c273844383be508000000000000-371354"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(14)

  2024-01-17T16:52:09.678908Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/b3e84eb4df0963f3fa3a5b9b37691d10c9cd2dbb1ffd68f3e208000000000000-959974"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(21)

  2024-01-17T16:52:09.678925Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/809a7b253d64319d1728de5b6616afe1c95232afcb817aeee408000000000000-2350394"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(8)

  2024-01-17T16:52:09.678954Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e69d7ec8d737a0bbabe3bbc4e107a5797737ebf78f04f149e308000000000000-3036384"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(29)

  2024-01-17T16:52:09.679003Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/0e0d78de29a00e5e0edde7978ef320da82487697f9a6786fe108000000000000-5721104"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(26)

  2024-01-17T16:52:09.679009Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/0e83d02a0144c726f05de850b71165691e380bd90205d22de008000000000000-1870824"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(24)

  2024-01-17T16:52:09.679123Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/068b56f69f5fee6ae34e11a3ca6581b0ce245b637e513619df08000000000000-3257688"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(25)

  2024-01-17T16:52:09.679406Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/5bdec358e0419d32a92502d3f4315ac333d71923d70a2ce6d708000000000000-69842920"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(27)

  2024-01-17T16:52:09.679571Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/5bdec358e0419d32a92502d3f4315ac333d71923d70a2ce6d908000000000000-69842920"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(29)

  2024-01-17T16:52:09.679609Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/5bdec358e0419d32a92502d3f4315ac333d71923d70a2ce6d808000000000000-69842920"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(27)

  2024-01-17T16:52:09.679838Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/42bf014c4b6583993ef2cd067ec908898463a103644f2bc6d408000000000000-191407912"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(21)

  2024-01-17T16:52:09.679954Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/eb82871b59be5321513597bb47f1f2e811880d351e0c7643d508000000000000-1676776"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(27)

  2024-01-17T16:52:09.679964Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/74d32fa1cee4dbee1a9e58335037bfe78de7c45026d6b327d608000000000000-99350168"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(25)

  2024-01-17T16:52:09.679975Z  INFO nativelink_worker::running_actions_manager: Worker Cleanup
    at nativelink-worker/src/running_actions_manager.rs:1100 on tokio-runtime-worker ThreadId(7)

  2024-01-17T16:52:09.683663Z ERROR nativelink_worker::local_worker: Error executing action: Error { code: Internal, messages: ["Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "Failed to receive data in filesystem store", "While processing with temp file \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/658785b562ec4159449feb5f642878771b5d7fbfb1c9b22cdd08000000000000-145320\"", "---", "status: InvalidArgument, message: \"Chunk should never be EOF, expected None in this case : During first buf_channel::take()\", details: [], metadata: MetadataMap { headers: {} }", "While fetching message in GrpcStore::get_part()", "---", "Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "Failed to read data data buffer from slow store", "Failed to populate()", "---", "Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "for digest DigestInfo { size_bytes: 145320, hash: \"658785b562ec4159449feb5f642878771b5d7fbfb1c9b22c5bb2e18647fa6a2f\" }", "in download_to_directory : /root/.cache/turbo-cache/work/fa6829be5edda144594066b31ccfd74dabd5e09e1c518e23a06e417d33723c73/build/linux/debian_bullseye_amd64-sysroot/lib/x86_64-linux-gnu", "in download_to_directory : /root/.cache/turbo-cache/work/fa6829be5edda144594066b31ccfd74dabd5e09e1c518e23a06e417d33723c73/build/linux/debian_bullseye_amd64-sysroot/lib", "in download_to_directory : /root/.cache/turbo-cache/work/fa6829be5edda144594066b31ccfd74dabd5e09e1c518e23a06e417d33723c73/build/linux/debian_bullseye_amd64-sysroot", "in download_to_directory : /root/.cache/turbo-cache/work/fa6829be5edda144594066b31ccfd74dabd5e09e1c518e23a06e417d33723c73/build/linux", "in download_to_directory : /root/.cache/turbo-cache/work/fa6829be5edda144594066b31ccfd74dabd5e09e1c518e23a06e417d33723c73/build"] }
    at nativelink-worker/src/local_worker.rs:295 on tokio-runtime-worker ThreadId(29)

  2024-01-17T16:52:10.218782Z  INFO nativelink_worker::running_actions_manager: Worker Received Action: ActionInfo { command_digest: DigestInfo { size_bytes: 2936, hash: "12d22a42aa28b39403f3f2f4f9bb1019d31dd276a883a3b7deb688748dec630b" }, input_root_digest: DigestInfo { size_bytes: 243, hash: "6a18216580d4615263ea5c679ff40ba4073de78057adef63619a0eccb4196665" }, timeout: 120s, platform_properties: PlatformProperties { properties: {"OSFamily": Unknown("Linux"), "label:action_large": Unknown("1"), "container-image": Unknown("docker://gcr.io/chops-public-images-prod/rbe/siso-chromium/linux@sha256:912808c295e578ccde53b0685bcd0d56c15d7a03e819dcce70694bfe3fdab35e")} }, priority: 0, load_timestamp: SystemTime { tv_sec: 1705510330, tv_nsec: 217361806 }, insert_timestamp: SystemTime { tv_sec: 1705510329, tv_nsec: 336182562 }, unique_qualifier: ActionInfoHashKey { instance_name: "jenkins", digest: DigestInfo { size_bytes: 352, hash: "c89a9671bdf82c2123e719d03736b2b738f57a6e0766d6b6830f9664f2d1d7a2" }, salt: 0 }, skip_cache_lookup: true, digest_function: Sha256 }
    at nativelink-worker/src/running_actions_manager.rs:1597 on tokio-runtime-worker ThreadId(7)

  2024-01-17T16:52:10.260324Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/eb82871b59be5321513597bb47f1f2e811880d351e0c7643e808000000000000-1676776"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(21)

  2024-01-17T16:52:10.263816Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/42bf014c4b6583993ef2cd067ec908898463a103644f2bc6e908000000000000-191407912"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(29)

  2024-01-17T16:52:10.263976Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/74d32fa1cee4dbee1a9e58335037bfe78de7c45026d6b327ea08000000000000-99350168"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(27)

  2024-01-17T16:52:10.264025Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/5bdec358e0419d32a92502d3f4315ac333d71923d70a2ce6eb08000000000000-69842920"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(27)

  2024-01-17T16:52:10.264245Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/5bdec358e0419d32a92502d3f4315ac333d71923d70a2ce6ed08000000000000-69842920"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(8)

  2024-01-17T16:52:10.264291Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/5bdec358e0419d32a92502d3f4315ac333d71923d70a2ce6ec08000000000000-69842920"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(27)

  2024-01-17T16:52:10.264453Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/23ea58fe8ac6087bdfcbda8bf326e467fc80b57a2dc28242f208000000000000-4246962"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(29)

  2024-01-17T16:52:10.264643Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/f8e5a9795670b80b5a5c2dd26afcbb93b896622953dfc157f308000000000000-1378184"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(29)

  2024-01-17T16:52:10.264653Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/94221894ec061137827f163707f6df462eb25e91f467431d0509000000000000-967866"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(18)

  2024-01-17T16:52:10.264714Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/01ba92d9e40180cff8b2c9acf3fb62efdc8c3767aeca45effc08000000000000-2032782"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(18)

  2024-01-17T16:52:10.264759Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/034e99dc22681f58ccd0ec3d32cadb4a98c0fbf4588b13defb08000000000000-882420"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(8)

  2024-01-17T16:52:10.264809Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/354e9319fa64bb1c440883918d0052c387e7683363ab3516fa08000000000000-1336184"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(29)

  2024-01-17T16:52:10.264870Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/0be228510b887f6b322d6e31d66d5b5c72b5c374a62cc44af008000000000000-5732310"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(18)

  2024-01-17T16:52:10.264991Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/95873fa2f7d97918b01c36b68b438d42728eee2e99fe0ef8f108000000000000-4642258"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(18)

  2024-01-17T16:52:10.265030Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/cc0984db6769b70fa24b1f3016ce1c3a997b4c5426864983fe08000000000000-1751162"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(18)

  2024-01-17T16:52:10.265070Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/2250bb46fabe1b3a54f85deec9d1499a30626a1a1c760d83fd08000000000000-5405210"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(7)

  2024-01-17T16:52:10.267176Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e57352e2d3a3b3f0ed7a0423b101f59073d46781315be430ef08000000000000-3001366"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(14)

  2024-01-17T16:52:10.267279Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/0e83d02a0144c726f05de850b71165691e380bd90205d22d0009000000000000-1870824"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(14)

  2024-01-17T16:52:10.267321Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/0e0d78de29a00e5e0edde7978ef320da82487697f9a6786f0209000000000000-5721104"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(25)

  2024-01-17T16:52:10.267479Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/809a7b253d64319d1728de5b6616afe1c95232afcb817aee0109000000000000-2350394"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(25)

  2024-01-17T16:52:10.267497Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e69d7ec8d737a0bbabe3bbc4e107a5797737ebf78f04f149f808000000000000-3036384"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(14)

  2024-01-17T16:52:10.267529Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/454b21758d93a0807502d829714ad87dcd975c273844383bf908000000000000-371354"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(2)

  2024-01-17T16:52:10.268015Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/9525bb6f8bac6069b2758dd633542beafee15eebd91f4356ee08000000000000-2664892"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(2)

  2024-01-17T16:52:10.268039Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/b3e84eb4df0963f3fa3a5b9b37691d10c9cd2dbb1ffd68f3f508000000000000-959974"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(14)

  2024-01-17T16:52:10.268121Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/051b7e88a378c49956ce1fef4f3dd2ea1f14f713b9c125ffff08000000000000-895140"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(14)

  2024-01-17T16:52:10.268147Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/a08fd4ce4cd09767db71aa7063ad5ab8c5393182030becd80409000000000000-9052480"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(2)

  2024-01-17T16:52:10.268205Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/068b56f69f5fee6ae34e11a3ca6581b0ce245b637e5136190309000000000000-3257688"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(14)

  2024-01-17T16:52:10.268376Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/fb2c57907b7b6874447b014fac08aee4203e01ab510f9bd6f408000000000000-9784456"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(18)

  2024-01-17T16:52:10.268421Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/658785b562ec4159449feb5f642878771b5d7fbfb1c9b22cf708000000000000-145320"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(18)

  2024-01-17T16:52:10.268441Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/658785b562ec4159449feb5f642878771b5d7fbfb1c9b22cf608000000000000-145320"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(18)

  2024-01-17T16:52:10.268540Z  INFO nativelink_worker::running_actions_manager: Worker Cleanup
    at nativelink-worker/src/running_actions_manager.rs:1100 on tokio-runtime-worker ThreadId(21)

  2024-01-17T16:52:10.273491Z ERROR nativelink_worker::local_worker: Error executing action: Error { code: Internal, messages: ["Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "Failed to receive data in filesystem store", "While processing with temp file \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/eb82871b59be5321513597bb47f1f2e811880d351e0c7643e808000000000000-1676776\"", "---", "status: InvalidArgument, message: \"Chunk should never be EOF, expected None in this case : During first buf_channel::take()\", details: [], metadata: MetadataMap { headers: {} }", "While fetching message in GrpcStore::get_part()", "---", "Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "Failed to read data data buffer from slow store", "Failed to populate()", "---", "Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "for digest DigestInfo { size_bytes: 1676776, hash: \"eb82871b59be5321513597bb47f1f2e811880d351e0c7643fd07b284636086d2\" }", "in download_to_directory : /root/.cache/turbo-cache/work/4589f9874b03183fc23cc9a24d87a996802749377e90c9629f24f9fd216e39d5/third_party/rust-toolchain/lib", "in download_to_directory : /root/.cache/turbo-cache/work/4589f9874b03183fc23cc9a24d87a996802749377e90c9629f24f9fd216e39d5/third_party/rust-toolchain", "in download_to_directory : /root/.cache/turbo-cache/work/4589f9874b03183fc23cc9a24d87a996802749377e90c9629f24f9fd216e39d5/third_party"] }
    at nativelink-worker/src/local_worker.rs:295 on tokio-runtime-worker ThreadId(8)

  2024-01-17T16:52:11.791800Z  INFO nativelink_worker::running_actions_manager: Worker Received Action: ActionInfo { command_digest: DigestInfo { size_bytes: 2936, hash: "12d22a42aa28b39403f3f2f4f9bb1019d31dd276a883a3b7deb688748dec630b" }, input_root_digest: DigestInfo { size_bytes: 243, hash: "6a18216580d4615263ea5c679ff40ba4073de78057adef63619a0eccb4196665" }, timeout: 120s, platform_properties: PlatformProperties { properties: {"container-image": Unknown("docker://gcr.io/chops-public-images-prod/rbe/siso-chromium/linux@sha256:912808c295e578ccde53b0685bcd0d56c15d7a03e819dcce70694bfe3fdab35e"), "OSFamily": Unknown("Linux"), "label:action_large": Unknown("1")} }, priority: 0, load_timestamp: SystemTime { tv_sec: 1705510331, tv_nsec: 790889956 }, insert_timestamp: SystemTime { tv_sec: 1705510330, tv_nsec: 816137570 }, unique_qualifier: ActionInfoHashKey { instance_name: "jenkins", digest: DigestInfo { size_bytes: 352, hash: "c89a9671bdf82c2123e719d03736b2b738f57a6e0766d6b6830f9664f2d1d7a2" }, salt: 0 }, skip_cache_lookup: true, digest_function: Sha256 }
    at nativelink-worker/src/running_actions_manager.rs:1597 on tokio-runtime-worker ThreadId(18)

  2024-01-17T16:52:11.835164Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/eb82871b59be5321513597bb47f1f2e811880d351e0c76430709000000000000-1676776"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(7)

  2024-01-17T16:52:11.837534Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/74d32fa1cee4dbee1a9e58335037bfe78de7c45026d6b3270609000000000000-99350168"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(7)

  2024-01-17T16:52:11.837612Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/42bf014c4b6583993ef2cd067ec908898463a103644f2bc60809000000000000-191407912"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(7)

  2024-01-17T16:52:11.837717Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/5bdec358e0419d32a92502d3f4315ac333d71923d70a2ce60909000000000000-69842920"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(2)

  2024-01-17T16:52:11.837732Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/5bdec358e0419d32a92502d3f4315ac333d71923d70a2ce60b09000000000000-69842920"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(21)

  2024-01-17T16:52:11.837867Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/5bdec358e0419d32a92502d3f4315ac333d71923d70a2ce60a09000000000000-69842920"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(26)

  2024-01-17T16:52:11.838137Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/f8e5a9795670b80b5a5c2dd26afcbb93b896622953dfc1571c09000000000000-1378184"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(7)

  2024-01-17T16:52:11.838185Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/23ea58fe8ac6087bdfcbda8bf326e467fc80b57a2dc282421d09000000000000-4246962"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(7)

  2024-01-17T16:52:11.838212Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/01ba92d9e40180cff8b2c9acf3fb62efdc8c3767aeca45ef2109000000000000-2032782"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(7)

  2024-01-17T16:52:11.838296Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/034e99dc22681f58ccd0ec3d32cadb4a98c0fbf4588b13de1f09000000000000-882420"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(7)

  2024-01-17T16:52:11.838609Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/94221894ec061137827f163707f6df462eb25e91f467431d2309000000000000-967866"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(7)

  2024-01-17T16:52:11.838654Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/354e9319fa64bb1c440883918d0052c387e7683363ab35161e09000000000000-1336184"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(7)

  2024-01-17T16:52:11.838689Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/0be228510b887f6b322d6e31d66d5b5c72b5c374a62cc44a1409000000000000-5732310"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(7)

  2024-01-17T16:52:11.838747Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/2250bb46fabe1b3a54f85deec9d1499a30626a1a1c760d832009000000000000-5405210"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(18)

  2024-01-17T16:52:11.838765Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/95873fa2f7d97918b01c36b68b438d42728eee2e99fe0ef81509000000000000-4642258"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(21)

  2024-01-17T16:52:11.838778Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/cc0984db6769b70fa24b1f3016ce1c3a997b4c54268649832209000000000000-1751162"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(25)

  2024-01-17T16:52:11.841213Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/0e0d78de29a00e5e0edde7978ef320da82487697f9a6786f1309000000000000-5721104"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(2)

  2024-01-17T16:52:11.841217Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e57352e2d3a3b3f0ed7a0423b101f59073d46781315be4301109000000000000-3001366"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(14)

  2024-01-17T16:52:11.841250Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/809a7b253d64319d1728de5b6616afe1c95232afcb817aee1b09000000000000-2350394"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(25)

  2024-01-17T16:52:11.841275Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/b3e84eb4df0963f3fa3a5b9b37691d10c9cd2dbb1ffd68f31209000000000000-959974"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(2)

  2024-01-17T16:52:11.841306Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e69d7ec8d737a0bbabe3bbc4e107a5797737ebf78f04f1491809000000000000-3036384"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(18)

  2024-01-17T16:52:11.841335Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/454b21758d93a0807502d829714ad87dcd975c273844383b1609000000000000-371354"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(2)

  2024-01-17T16:52:11.841317Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/a08fd4ce4cd09767db71aa7063ad5ab8c5393182030becd81909000000000000-9052480"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(26)

  2024-01-17T16:52:11.841372Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/fb2c57907b7b6874447b014fac08aee4203e01ab510f9bd60e09000000000000-9784456"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(2)

  2024-01-17T16:52:11.841370Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/068b56f69f5fee6ae34e11a3ca6581b0ce245b637e5136191009000000000000-3257688"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(8)

  2024-01-17T16:52:11.841313Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/0e83d02a0144c726f05de850b71165691e380bd90205d22d1a09000000000000-1870824"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(25)

  2024-01-17T16:52:11.841394Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/051b7e88a378c49956ce1fef4f3dd2ea1f14f713b9c125ff1709000000000000-895140"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(2)

  2024-01-17T16:52:11.841410Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/658785b562ec4159449feb5f642878771b5d7fbfb1c9b22c0d09000000000000-145320"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(25)

  2024-01-17T16:52:11.841391Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/9525bb6f8bac6069b2758dd633542beafee15eebd91f43560f09000000000000-2664892"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(18)

  2024-01-17T16:52:11.841417Z  INFO nativelink_store::filesystem_store: Filesystem Store: Deleting: "/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/658785b562ec4159449feb5f642878771b5d7fbfb1c9b22c0c09000000000000-145320"
    at nativelink-store/src/filesystem_store.rs:102 on tokio-runtime-worker ThreadId(21)

  2024-01-17T16:52:11.841356Z  INFO nativelink_worker::running_actions_manager: Worker Cleanup
    at nativelink-worker/src/running_actions_manager.rs:1100 on tokio-runtime-worker ThreadId(24)

  2024-01-17T16:52:11.845457Z ERROR nativelink_worker::local_worker: Error executing action: Error { code: Internal, messages: ["Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "Failed to receive data in filesystem store", "While processing with temp file \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/eb82871b59be5321513597bb47f1f2e811880d351e0c76430709000000000000-1676776\"", "---", "status: InvalidArgument, message: \"Chunk should never be EOF, expected None in this case : During first buf_channel::take()\", details: [], metadata: MetadataMap { headers: {} }", "While fetching message in GrpcStore::get_part()", "---", "Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "Failed to read data data buffer from slow store", "Failed to populate()", "---", "Received erroneous partial chunk: Error { code: Internal, messages: [\"Writer was dropped before EOF was sent\"] }", "for digest DigestInfo { size_bytes: 1676776, hash: \"eb82871b59be5321513597bb47f1f2e811880d351e0c7643fd07b284636086d2\" }", "in download_to_directory : /root/.cache/turbo-cache/work/4589f9874b03183fc23cc9a24d87a996802749377e90c9629f24f9fd216e39d5/third_party/rust-toolchain/lib", "in download_to_directory : /root/.cache/turbo-cache/work/4589f9874b03183fc23cc9a24d87a996802749377e90c9629f24f9fd216e39d5/third_party/rust-toolchain", "in download_to_directory : /root/.cache/turbo-cache/work/4589f9874b03183fc23cc9a24d87a996802749377e90c9629f24f9fd216e39d5/third_party"] }
    at nativelink-worker/src/local_worker.rs:295 on tokio-runtime-worker ThreadId(8)

@chrisstaite-menlo chrisstaite-menlo mentioned this issue Jan 17, 2024
Merged
5 tasks
@allada
Copy link
Member

allada commented Jan 17, 2024

I think this might be fixed in:
#496

@chrisstaite-menlo
Copy link
Collaborator Author

I have taken the latest main branch and applied #496 and #616 and it appears to be behaving itself.

@allada
Copy link
Member

allada commented Jan 18, 2024

Awesome, lets keep this ticket open for a few days just to ensure it doesn't resurface.

@chrisstaite-menlo
Copy link
Collaborator Author

Running super solid now even with massive concurrency. It appears that bumping experimental_http2_max_pending_accept_reset_streams to 300 has completely eradicated errors on the storage server.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@allada @aaronmondal @chrisstaite-menlo