clock offset is huge

[huangxueliang]

{"log":"[2024/11/11 07:39:23.128 +00:00] [WARN] [tso/tso.go:178] ["clock offset is huge, check network latency and clock skew"] [jet-lag=235.002138ms] [prev-physical=2024/11/11 07:39:22.893 +00:00] [now=2024/11/11 07:39:23.128 +00:00]\n","stream":"stdout","time":"2024-11-11T07:39:23.129402722Z"}
{"log":"[2024/11/11 07:39:39.345 +00:00] [INFO] [writebuffer/write_buffer.go:263] ["checkpoint from latest consumed msg"] [collectionID=451860017339874767] [channel=by-dev-rootcoord-dml_6_451860017339874767v0]\n","stream":"stdout","time":"2024-11-11T07:39:39.34603714Z"}

how can i slove the WARN:clock offset is huge

[xiaofan-luan]

{"log":"[2024/11/11 07:39:23.128 +00:00] [WARN] [tso/tso.go:178] ["clock offset is huge, check network latency and clock skew"] [jet-lag=235.002138ms] [prev-physical=2024/11/11 07:39:22.893 +00:00] [now=2024/11/11 07:39:23.128 +00:00]\n","stream":"stdout","time":"2024-11-11T07:39:23.129402722Z"} {"log":"[2024/11/11 07:39:39.345 +00:00] [INFO] [writebuffer/write_buffer.go:263] ["checkpoint from latest consumed msg"] [collectionID=451860017339874767] [channel=by-dev-rootcoord-dml_6_451860017339874767v0]\n","stream":"stdout","time":"2024-11-11T07:39:39.34603714Z"}

how can i slove the WARN:clock offset is huge

install an ntp and make sure all the pod of milvus are almost sync at clock

[huangxueliang]

my milvus is Standalone,how can i do?

[xiaofan-luan]

if it's standalone, I guess what happens is your cpu usage in the too high and the pod is throttled by K8s.

[xiaofan-luan]

@huangxueliang
check your cpu usage of your pod and see if there are quota limitation and cause the pod stopped

[yhmo]

Observe the log, are there more warnings in the log?
Such as "rocksmq produce too slowly", that caused by low-performance disk.

Basic hardware requirements for the disk: https://milvus.io/docs/prerequisite-docker.md#Hardware-requirements
Requirement: SATA 3.0 SSD or higher
Recommendation: NVMe SSD or higher

[huangxueliang]

this is no the "rocksmq produce too slowly"

{"log":"[2024/11/11 08:07:32.635 +00:00] [WARN] [tso/tso.go:178] ["clock offset is huge, check network latency and clock skew"] [jet-lag=190.95458ms] [prev-physical=2024/11/11 08:07:32.444 +00:00] [now=2024/11/11 08:07:32.635 +00:00]\n","stream":"stdout","time":"2024-11-11T08:07:32.635291166Z"}
{"log":"[2024/11/11 08:07:42.546 +00:00] [INFO] [writebuffer/write_buffer.go:263] ["checkpoint from latest consumed msg"] [collectionID=451860017339874767] [channel=by-dev-rootcoord-dml_6_451860017339874767v0]\n","stream":"stdout","time":"2024-11-11T08:07:42.546891648Z"}
{"log":"[2024/11/11 08:07:43.346 +00:00] [INFO] [writebuffer/write_buffer.go:263] ["checkpoint from latest consumed msg"] [collectionID=453783373745773841] [channel=by-dev-rootcoord-dml_1_453783373745773841v0]\n","stream":"stdout","time":"2024-11-11T08:07:43.346708169Z"}
{"log":"[2024/11/11 08:07:43.946 +00:00] [INFO] [writebuffer/write_buffer.go:263] ["checkpoint from latest consumed msg"] [collectionID=451875630419352409] [channel=by-dev-rootcoord-dml_2_451875630419352409v0]\n","stream":"stdout","time":"2024-11-11T08:07:43.946549319Z"}
{"log":"[2024/11/11 08:07:43.947 +00:00] [INFO] [writebuffer/write_buffer.go:263] ["checkpoint from latest consumed msg"] [collectionID=451860017338870801] [channel=by-dev-rootcoord-dml_4_451860017338870801v0]\n","stream":"stdout","time":"2024-11-11T08:07:43.947237382Z"}
{"log":"[2024/11/11 08:07:43.947 +00:00] [INFO] [writebuffer/write_buffer.go:263] ["checkpoint from latest consumed msg"] [collectionID=451875630419356440] [channel=by-dev-rootcoord-dml_9_451875630419356440v0]\n","stream":"stdout","time":"2024-11-11T08:07:43.947421543Z"}
{"log":"[2024/11/11 08:07:43.947 +00:00] [INFO] [writebuffer/write_buffer.go:263] ["checkpoint from latest consumed msg"] [collectionID=451875630381535321] [channel=by-dev-rootcoord-dml_0_451875630381535321v0]\n","stream":"stdout","time":"2024-11-11T08:07:43.947691839Z"}
{"log":"[2024/11/11 08:07:44.546 +00:00] [INFO] [writebuffer/write_buffer.go:263] ["checkpoint from latest consumed msg"] [collectionID=451860017338870811] [channel=by-dev-rootcoord-dml_5_451860017338870811v0]\n","stream":"stdout","time":"2024-11-11T08:07:44.546959984Z"}
{"log":"[2024/11/11 08:07:44.746 +00:00] [INFO] [writebuffer/write_buffer.go:263] ["checkpoint from latest consumed msg"] [collectionID=453783373746980055] [channel=by-dev-rootcoord-dml_7_453783373746980055v0]\n","stream":"stdout","time":"2024-11-11T08:07:44.746920405Z"}

[yhmo]

Then it could an occasional case, doesn't affect function.