Farmer hangs after Ctrl-C when there are invalid chunk errors

[teor2345]

Setup

On macOS 13.7 on M1 Max, I ran the following commands:

./subspace-node run --chain devnet --base-path ~/subspace-nets/devnet-2024-10-24/ --blocks-pruning archive --state-pruning archive --sync full --bootstrap-nodes /dns/bootstrap-0.devnet.subspace.foundation/tcp/30333/p2p/12D3KooWKd4qisMjuBXU4YvpbJgepiX278iumqMjCC3GiUjjZEZS --dsn-bootstrap-nodes /dns/bootstrap-0.devnet.subspace.foundation/tcp/30533/p2p/12D3KooWSKAQm66N7jQme72obbPPquBSZuSqMyx5mH5ki1aZ4pGv --pot-external-entropy 0xddca5a66 --farmer

./subspace-farmer farm --reward-address 5Dsysn9Pu9fLwNEau17JpqJ1tDRtRRiXBKy9GWWxUezu1BFq path=~/subspace-nets/devnet-2024-10-24/farm-0,size=10G --node-rpc-url ws://127.0.0.1:RPC_PORT

This bug happens with the binaries from:

• autonomys/subspace/actions/runs/11418607109
• Exit on panics, rather than unwinding #3177

Root Cause

I understand the network is partly shut down, and the farmer's storage might be corrupted. But I'm not sure how the corruption happened, I was running the devnet binaries as part of the test network. Unfortunately I don't have logs of the corruption itself, because my terminal only saves a few thousand lines.

Whatever the root cause was, the farmer shouldn't hang when it's newly started, even if its storage is corrupt.

Errors

There were a bunch of "invalid chunk" errors, so I pressed Ctrl-C:

2024-10-28T02:30:46.377291Z  INFO {farm_index=0}:{sector_index=7}: subspace_farmer_components::segment_reconstruction: Recovering missing piece... missing_piece_index=4403
2024-10-28T02:30:50.237170Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=628 error=Invalid chunk at location 20532296 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:30:52.110828Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=151 error=Invalid chunk at location 20530131 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:30:56.068184Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=983 error=Invalid chunk at location 20530024 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
^C2024-10-28T02:31:09.391617Z  INFO subspace_farmer::utils: Received SIGINT, shutting down farmer...

But the node didn't exit after I pressed Ctrl-C, or when I used kill (SIGTERM) on its pid:

2024-10-28T02:31:13.727143Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=34 error=Invalid chunk at location 20531914 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:31:15.589845Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=402 error=Invalid chunk at location 20530282 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar

I had to use kill -KILL to get it to exit.

Where it's hanging

The farmer is hanging somewhere in the piece reading code, but I'm not sure exactly which function is blocking it exiting.

Here's what I got from flamegraph --pid PID after the Ctrl-C when the farmer didn't shut down:

Full Logs

The full logs are:

$ ./subspace-farmer farm --reward-address 5Dsysn9Pu9fLwNEau17JpqJ1tDRtRRiXBKy9GWWxUezu1BFq path=~/subspace-nets/devnet-2024-10-24/farm-0,size=10G --node-rpc-url ws://127.0.0.1:49918
Exit 137
2024-10-28T02:30:42.546960Z  INFO subspace_farmer::commands::farm: Connecting to node RPC url=ws://127.0.0.1:49918
2024-10-28T02:30:42.559884Z  INFO subspace_farmer::node_client::caching_proxy_node_client: Downloading all segment headers from node...
2024-10-28T02:30:42.560240Z  INFO subspace_farmer::node_client::caching_proxy_node_client: Downloaded all segment headers from node successfully
2024-10-28T02:30:42.563752Z  INFO subspace_networking::constructor: DSN instance configured. allow_non_global_addresses_in_dht=false peer_id=12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv protocol_version=/subspace/2/1666067b80762b374703f5267360f8c63c0a8c068579b63c7455c6b2302ec57d
2024-10-28T02:30:42.567371Z  INFO libp2p_swarm: local_peer_id=12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv
2024-10-28T02:30:42.998612Z  INFO subspace_farmer::commands::farm: Preparing plotting thread pools plotting_thread_pool_core_indices=[CpuCoreSet { cores: CpuSet(0-9), .. }] replotting_thread_pool_core_indices=[CpuCoreSet { cores: CpuSet(0-4), .. }]
2024-10-28T02:30:43.006612Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while
2024-10-28T02:30:43.006756Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents
2024-10-28T02:30:43.014136Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-10-28T02:30:45.547367Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-10-28T02:30:45.562124Z  INFO {farm_index=0}: subspace_farmer::commands::farm: Farm 0:
2024-10-28T02:30:45.562136Z  INFO {farm_index=0}: subspace_farmer::commands::farm:   ID: 01JARGR52SKF93EY07J9C6DJRA
2024-10-28T02:30:45.562141Z  INFO {farm_index=0}: subspace_farmer::commands::farm:   Genesis hash: 0x1666067b80762b374703f5267360f8c63c0a8c068579b63c7455c6b2302ec57d
2024-10-28T02:30:45.562143Z  INFO {farm_index=0}: subspace_farmer::commands::farm:   Public key: 0x189db863c1905d14e524456c7496766ddd80da2856b24dd21627b5d745d52344
2024-10-28T02:30:45.562149Z  INFO {farm_index=0}: subspace_farmer::commands::farm:   Allocated space: 9.3 GiB (10.0 GB)
2024-10-28T02:30:45.562174Z  INFO {farm_index=0}: subspace_farmer::commands::farm:   Directory: /Users/hyper/subspace-nets/devnet-2024-10-24/farm-0
2024-10-28T02:30:45.562381Z  INFO subspace_farmer::commands::farm: Collecting already plotted pieces (this will take some time)...
2024-10-28T02:30:45.562423Z  INFO subspace_farmer::farmer_cache: Initializing piece cache
2024-10-28T02:30:45.564209Z  INFO subspace_farmer::commands::farm: Finished collecting already plotted pieces successfully
2024-10-28T02:30:45.564706Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-10-28T02:30:45.564873Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-10-28T02:30:45.564883Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::reward_signing: Subscribing to reward signing notifications
2024-10-28T02:30:45.565309Z  INFO subspace_farmer::commands::shared::network: DSN listening on /ip4/127.0.0.1/tcp/30533/p2p/12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv
2024-10-28T02:30:45.565566Z  INFO subspace_farmer::commands::shared::network: DSN listening on /ip6/::1/tcp/30533/p2p/12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv
2024-10-28T02:30:45.565577Z  INFO subspace_farmer::commands::shared::network: DSN listening on /ip4/192.168.18.222/tcp/30533/p2p/12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv
2024-10-28T02:30:45.565586Z  INFO subspace_farmer::commands::shared::network: DSN listening on /ip4/192.168.18.2/tcp/30533/p2p/12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv
2024-10-28T02:30:46.071369Z  INFO subspace_farmer::farmer_cache: Synchronizing piece cache
2024-10-28T02:30:46.073043Z  INFO {farm_index=0}:{sector_index=7}: subspace_farmer::single_disk_farm::plotting: Replotting sector (0.00% complete)
2024-10-28T02:30:46.102827Z  INFO subspace_farmer::farmer_cache: Finished piece cache synchronization
2024-10-28T02:30:46.377291Z  INFO {farm_index=0}:{sector_index=7}: subspace_farmer_components::segment_reconstruction: Recovering missing piece... missing_piece_index=4403
2024-10-28T02:30:50.237170Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=628 error=Invalid chunk at location 20532296 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:30:52.110828Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=151 error=Invalid chunk at location 20530131 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:30:56.068184Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=983 error=Invalid chunk at location 20530024 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
^C2024-10-28T02:31:09.391617Z  INFO subspace_farmer::utils: Received SIGINT, shutting down farmer...
2024-10-28T02:31:13.727143Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=34 error=Invalid chunk at location 20531914 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:31:15.589845Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=402 error=Invalid chunk at location 20530282 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:31:27.168930Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=447 error=Invalid chunk at location 15399502 s-bucket 24581 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:31:29.018152Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=860 error=Invalid chunk at location 20529955 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:31:40.799672Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=605 error=Invalid chunk at location 20529791 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:31:48.523526Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=298 error=Invalid chunk at location 20529593 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:32:32.737403Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=531 error=Invalid chunk at location 20530366 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:32:44.742552Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=574 error=Invalid chunk at location 15397684 s-bucket 24578 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:32:50.752305Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=304 error=Invalid chunk at location 20530222 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:32:56.726401Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=652 error=Invalid chunk at location 20530447 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:33:02.759542Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=936 error=Invalid chunk at location 15399186 s-bucket 24580 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:33:12.671331Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=9 error=Invalid chunk at location 20529393 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:33:18.631802Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=717 error=Invalid chunk at location 20532353 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:34:04.455366Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=316 error=Invalid chunk at location 15398798 s-bucket 24580 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:34:08.345956Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=794 error=Invalid chunk at location 15396527 s-bucket 24576 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:34:17.977849Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=84 error=Invalid chunk at location 20530090 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:34:25.793158Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=57 error=Invalid chunk at location 20531930 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:34:31.909144Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=979 error=Invalid chunk at location 20530020 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:35:03.696272Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=934 error=Invalid chunk at location 15397911 s-bucket 24578 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:35:13.484399Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=863 error=Invalid chunk at location 20531191 s-bucket 32770 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:35:45.555993Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=559 error=Invalid chunk at location 20529758 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:35:46.076149Z  WARN libp2p_kad::behaviour: Failed to trigger bootstrap: No known peers.
2024-10-28T02:35:47.469890Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=400 error=Invalid chunk at location 20532152 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:36:07.091913Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=669 error=Invalid chunk at location 20531688 s-bucket 32771 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:36:55.045260Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=141 error=Invalid chunk at location 20530125 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:36:56.909655Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=18 error=Invalid chunk at location 20529399 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:36:58.861221Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=117 error=Invalid chunk at location 15398021 s-bucket 24579 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:37:04.721454Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=635 error=Invalid chunk at location 20531062 s-bucket 32770 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:37:10.664913Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=219 error=Invalid chunk at location 20532664 s-bucket 32773 encoded true, possible disk corruption: Invalid scalar

[teor2345]

Attempted Fixes

This issue still happens after the code changes in:

• Exit on panics, rather than unwinding #3177
• my shutdown-timeout branch, which runs Runtime::shutdown_timeout on the tokio runtime after the main farmer function returns
• Add a shutdown watchdog to subspace binaries #3170, which didn't quite work, because the watchdog is spawned too late

Further Analysis

I added some logs to see where the hang was:

2024-10-28T03:04:35.739207Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=651 error=Invalid chunk at location 20531677 s-bucket 32771 encoded true, possible disk corruption: Invalid scalar
^C2024-10-28T03:04:39.384097Z  INFO subspace_farmer::utils: Received SIGINT, shutting down farmer...
2024-10-28T03:04:39.384111Z  INFO subspace_farmer::commands::farm: signal select branch
2024-10-28T03:04:39.384113Z  INFO subspace_farmer::commands::farm: end of async fn farm()
2024-10-28T03:05:10.236106Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=671 error=Invalid chunk at location 15396453 s-bucket 24576 encoded true, possible disk corruption: Invalid scalar

The farmer is blocked at the end of the farm() function. None of the logs I added in main() got logged.

[teor2345]

Aha, it did exit, it just took 17 minutes.

Full logs (including logs that show where in the code it hung):

$ ./subspace-farmer farm --reward-address 5Dsysn9Pu9fLwNEau17JpqJ1tDRtRRiXBKy9GWWxUezu1BFq path=~/subspace-nets/devnet-2024-10-24/farm-0,size=10G --node-rpc-url ws://127.0.0.1:49918
Exit 137
2024-10-28T03:04:27.869201Z  INFO subspace_farmer::commands::farm: Connecting to node RPC url=ws://127.0.0.1:49918
2024-10-28T03:04:27.879595Z  INFO subspace_farmer::node_client::caching_proxy_node_client: Downloading all segment headers from node...
2024-10-28T03:04:27.879993Z  INFO subspace_farmer::node_client::caching_proxy_node_client: Downloaded all segment headers from node successfully
2024-10-28T03:04:27.883274Z  INFO subspace_networking::constructor: DSN instance configured. allow_non_global_addresses_in_dht=false peer_id=12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv protocol_version=/subspace/2/1666067b80762b374703f5267360f8c63c0a8c068579b63c7455c6b2302ec57d
2024-10-28T03:04:27.886818Z  INFO libp2p_swarm: local_peer_id=12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv
2024-10-28T03:04:28.311078Z  INFO subspace_farmer::commands::farm: Preparing plotting thread pools plotting_thread_pool_core_indices=[CpuCoreSet { cores: CpuSet(0-9), .. }] replotting_thread_pool_core_indices=[CpuCoreSet { cores: CpuSet(0-4), .. }]
2024-10-28T03:04:28.318777Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while
2024-10-28T03:04:28.318931Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents
2024-10-28T03:04:28.326807Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-10-28T03:04:30.687798Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-10-28T03:04:30.701671Z  INFO {farm_index=0}: subspace_farmer::commands::farm: Farm 0:
2024-10-28T03:04:30.701682Z  INFO {farm_index=0}: subspace_farmer::commands::farm:   ID: 01JARGR52SKF93EY07J9C6DJRA
2024-10-28T03:04:30.701698Z  INFO {farm_index=0}: subspace_farmer::commands::farm:   Genesis hash: 0x1666067b80762b374703f5267360f8c63c0a8c068579b63c7455c6b2302ec57d
2024-10-28T03:04:30.701701Z  INFO {farm_index=0}: subspace_farmer::commands::farm:   Public key: 0x189db863c1905d14e524456c7496766ddd80da2856b24dd21627b5d745d52344
2024-10-28T03:04:30.701727Z  INFO {farm_index=0}: subspace_farmer::commands::farm:   Allocated space: 9.3 GiB (10.0 GB)
2024-10-28T03:04:30.701744Z  INFO {farm_index=0}: subspace_farmer::commands::farm:   Directory: /Users/hyper/subspace-nets/devnet-2024-10-24/farm-0
2024-10-28T03:04:30.701868Z  INFO subspace_farmer::commands::farm: Collecting already plotted pieces (this will take some time)...
2024-10-28T03:04:30.701924Z  INFO subspace_farmer::farmer_cache: Initializing piece cache
2024-10-28T03:04:30.703654Z  INFO subspace_farmer::commands::farm: Finished collecting already plotted pieces successfully
2024-10-28T03:04:30.703933Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-10-28T03:04:30.703949Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-10-28T03:04:30.703956Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::reward_signing: Subscribing to reward signing notifications
2024-10-28T03:04:30.705087Z  INFO subspace_farmer::commands::shared::network: DSN listening on /ip4/127.0.0.1/tcp/30533/p2p/12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv
2024-10-28T03:04:30.705350Z  INFO subspace_farmer::commands::shared::network: DSN listening on /ip6/::1/tcp/30533/p2p/12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv
2024-10-28T03:04:30.705358Z  INFO subspace_farmer::commands::shared::network: DSN listening on /ip4/192.168.18.222/tcp/30533/p2p/12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv
2024-10-28T03:04:30.705370Z  INFO subspace_farmer::commands::shared::network: DSN listening on /ip4/192.168.18.2/tcp/30533/p2p/12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv
2024-10-28T03:04:31.209924Z  INFO subspace_farmer::farmer_cache: Synchronizing piece cache
2024-10-28T03:04:31.211675Z  INFO {farm_index=0}:{sector_index=7}: subspace_farmer::single_disk_farm::plotting: Replotting sector (0.00% complete)
2024-10-28T03:04:31.896383Z  WARN libp2p_kad::behaviour: Failed to trigger bootstrap: No known peers.
2024-10-28T03:04:31.922698Z  INFO subspace_farmer::farmer_cache: Finished piece cache synchronization
2024-10-28T03:04:32.047669Z  INFO {farm_index=0}:{sector_index=7}: subspace_farmer_components::segment_reconstruction: Recovering missing piece... missing_piece_index=1059
2024-10-28T03:04:35.739207Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=651 error=Invalid chunk at location 20531677 s-bucket 32771 encoded true, possible disk corruption: Invalid scalar
^C2024-10-28T03:04:39.384097Z  INFO subspace_farmer::utils: Received SIGINT, shutting down farmer...
2024-10-28T03:04:39.384111Z  INFO subspace_farmer::commands::farm: signal select branch
2024-10-28T03:04:39.384113Z  INFO subspace_farmer::commands::farm: end of async fn farm()
2024-10-28T03:05:10.236106Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=671 error=Invalid chunk at location 15396453 s-bucket 24576 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:05:15.744994Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=582 error=Invalid chunk at location 20530402 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:05:19.355233Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=442 error=Invalid chunk at location 15397603 s-bucket 24578 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:05:40.039400Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=14 error=Invalid chunk at location 15396673 s-bucket 24577 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:06:13.850488Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=574 error=Invalid chunk at location 15397684 s-bucket 24578 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:06:40.465975Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=714 error=Invalid chunk at location 20530482 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:07:01.897407Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=979 error=Invalid chunk at location 20530020 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:07:05.792917Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=814 error=Invalid chunk at location 20529929 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:07:09.533264Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=446 error=Invalid chunk at location 20529684 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:07:13.242498Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=501 error=Invalid chunk at location 20529720 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:07:20.760735Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=863 error=Invalid chunk at location 20531191 s-bucket 32770 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:07:45.220739Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=577 error=Invalid chunk at location 20529770 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:07:48.921442Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=605 error=Invalid chunk at location 20529791 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:08:00.354067Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=404 error=Invalid chunk at location 20529659 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:08:05.904692Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=874 error=Invalid chunk at location 20533071 s-bucket 32773 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:08:17.274189Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=447 error=Invalid chunk at location 20531550 s-bucket 32771 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:09:09.340978Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=45 error=Invalid chunk at location 20529418 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:09:11.128233Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=678 error=Invalid chunk at location 23096087 s-bucket 36864 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:09:31.906446Z  WARN libp2p_kad::behaviour: Failed to trigger bootstrap: No known peers.
2024-10-28T03:09:35.754257Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=141 error=Invalid chunk at location 20530125 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:09:45.278837Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=558 error=Invalid chunk at location 20531015 s-bucket 32770 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:09:58.439013Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=117 error=Invalid chunk at location 20532595 s-bucket 32773 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:10:17.214293Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=366 error=Invalid chunk at location 15396908 s-bucket 24577 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:10:20.777816Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=852 error=Invalid chunk at location 20530572 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:11:03.752059Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=202 error=Invalid chunk at location 20532028 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:11:07.375453Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=316 error=Invalid chunk at location 20530868 s-bucket 32770 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:11:14.808446Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=95 error=Invalid chunk at location 15396723 s-bucket 24577 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:11:20.290224Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=936 error=Invalid chunk at location 15399186 s-bucket 24580 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:11:22.062909Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=298 error=Invalid chunk at location 20529593 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:11:31.383476Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=134 error=Invalid chunk at location 15398682 s-bucket 24580 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:11:51.795447Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=473 error=Invalid chunk at location 20534691 s-bucket 32776 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:12:04.896285Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=219 error=Invalid chunk at location 20532664 s-bucket 32773 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:12:21.747393Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=59 error=Invalid chunk at location 15396700 s-bucket 24577 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:12:25.429656Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=34 error=Invalid chunk at location 20531914 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:12:38.485213Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=377 error=Invalid chunk at location 20530265 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:12:40.243353Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=569 error=Invalid chunk at location 20531023 s-bucket 32770 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:13:10.285007Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=84 error=Invalid chunk at location 20530090 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:13:13.862207Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=403 error=Invalid chunk at location 20531523 s-bucket 32771 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:13:15.639197Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=848 error=Invalid chunk at location 20530569 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:13:26.886856Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=402 error=Invalid chunk at location 20530282 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:13:34.381042Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=860 error=Invalid chunk at location 20529955 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:13:41.992782Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=18 error=Invalid chunk at location 15396060 s-bucket 24576 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:13:47.416684Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=151 error=Invalid chunk at location 20530131 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:13:49.178926Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=57 error=Invalid chunk at location 20531930 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:13:50.972383Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=304 error=Invalid chunk at location 15396236 s-bucket 24576 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:14:31.803047Z  WARN libp2p_kad::behaviour: Failed to trigger bootstrap: No known peers.
2024-10-28T03:14:53.193743Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=73 error=Invalid chunk at location 20530082 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:15:02.531185Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=829 error=Invalid chunk at location 16680023 s-bucket 26624 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:15:04.306124Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=292 error=Invalid chunk at location 20530215 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:15:13.655777Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=144 error=Invalid chunk at location 20532611 s-bucket 32773 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:15:19.137320Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=2 error=Invalid chunk at location 20530039 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:15:35.892909Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=74 error=Invalid chunk at location 20530083 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:15:54.633768Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=717 error=Invalid chunk at location 20532353 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:16:05.829172Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=756 error=Invalid chunk at location 15398436 s-bucket 24579 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:16:07.632074Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=791 error=Invalid chunk at location 20529916 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:16:11.339153Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=150 error=Invalid chunk at location 20530130 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:16:18.701387Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=474 error=Invalid chunk at location 21812639 s-bucket 34816 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:16:28.044645Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=755 error=Invalid chunk at location 20530507 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:17:09.497806Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=452 error=Invalid chunk at location 20530318 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:17:13.174638Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=837 error=Invalid chunk at location 20533671 s-bucket 32774 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:17:24.471791Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=669 error=Invalid chunk at location 20531688 s-bucket 32771 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:17:31.978087Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=20 error=Invalid chunk at location 20530051 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:17:37.540352Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=9 error=Invalid chunk at location 20529393 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:18:17.353410Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=575 error=Invalid chunk at location 20531629 s-bucket 32771 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:18:45.386698Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=859 error=Invalid chunk at location 20531189 s-bucket 32770 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:19:17.863804Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=114 error=Invalid chunk at location 20529465 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:19:25.187593Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=648 error=Invalid chunk at location 20534151 s-bucket 32775 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:19:31.809430Z  WARN libp2p_kad::behaviour: Failed to trigger bootstrap: No known peers.
2024-10-28T03:19:34.725782Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=308 error=Invalid chunk at location 20529600 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:19:42.094973Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=120 error=Invalid chunk at location 15396121 s-bucket 24576 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:19:43.850573Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=133 error=Invalid chunk at location 20530120 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:19:47.487999Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=338 error=Invalid chunk at location 20529619 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:19:51.310132Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=234 error=Invalid chunk at location 20529545 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:19:53.129580Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=84 error=Invalid chunk at location 15396100 s-bucket 24576 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:19:58.533113Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=354 error=Invalid chunk at location 20530252 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:20:00.309980Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=837 error=Invalid chunk at location 20533671 s-bucket 32774 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:20:11.397075Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=355 error=Invalid chunk at location 20530253 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:20:47.089882Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=402 error=Invalid chunk at location 20530282 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:21:27.083450Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=311 error=Invalid chunk at location 15396239 s-bucket 24576 encoded true, possible disk corruption: Invalid scalar
2024-10-28T03:21:32.579313Z  INFO subspace_farmer: end of block with defer!() and farm() calls
2024-10-28T03:21:32.579328Z  INFO subspace_farmer: spawning watchdog
2024-10-28T03:21:32.579355Z  INFO subspace_farmer: end of tokio::main()