The log saved during kurtosis log collection is too large

[icesfeathers]

Background & motivation

I'm running a geth private network using kurtosis, and the kurtosis logs used 150G of the hard drive in one day, but the actual geth network only used 20G.
I checked the container kurtosis-logs-collection and found that it was continuously collecting debug logs, causing my hard disk space to be filled up with logs.

Desired behaviour

Currently, I stopped kurtosis's log collection container and cleared kurtosis-logs-storage to free up space.
For me, kurtosis logs are not more important than eth network.
It is recommended to increase the log collection level of kurtosis logs or disable it.

How important is this to you?

Critical; Kurtosis is unusable for me without it.

What area of the product does this pertain to?

CLI: the Command Line Interface

[icesfeathers]

Here is the command information and configuration information for running kurtosis

kurtosis run --cli-log-level "panic" --enclave my-testnet github.com/kurtosis-tech/[email protected] "$(cat /data/fleet/kurtosis_network_params.yaml)"

participants:
    - validator_count: 512
      beacon_extra_params:
          - "--reconstruct-historic-states"
      el_extra_params:
          - "--gcmode=archive"
      el_max_mem: 8192
      bn_max_mem: 4096
      v_max_mem: 4096
    - validator_count: 1
      el_max_mem: 8192
      bn_max_mem: 4096
      v_max_mem: 4096
network_params:
    seconds_per_slot: 6
    eth1_follow_distance: 2
    deneb_fork_epoch: 9999999999999
additional_services:
    - tx_spammer
    - blob_spammer
    - el_forkmon
    - blockscout
    - beacon_metrics_gazer
    - dora
    - prometheus_grafana
mev_type: "full"

[tedim52]

Thanks for filing this @icesfeathers ! As mentioned in Discord, it seems like what would make sense as a solution here is:

the ability the ability to turn off the collection of logs (in which case you won't be able to view logs after services have died)

OR

a byte based retention mechanism where historical logs are removed after a certain threshold is passed so it logs fill up disk space.(For context, currently, we have time based log retention where old logs are removed after 4 weeks but no byte based log retention)

[icesfeathers]

After I stopped the log container, the memory usage of the dockerd process soared and could not be released. Is this a bug?

[icesfeathers]

[icesfeathers]

@tedim52

[icesfeathers]

Since I have not turned on the debug mode, I cannot explain in detail. Since this docker only runs kurtosis, I would like to give you some feedback.

[tedim52]

Hey @icesfeathers ! I suspect what's causing the memory issues are the fact that logs are being sent to logs aggregator but aren't being processed/forwarded. Currently, we don't support a way to turn off logs collection safely in the product.