Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry pick for 0.24.2 #1657

Merged
merged 1 commit into from
May 2, 2022
Merged

Cherry pick for 0.24.2 #1657

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/markdown.links.config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@
},
{
"pattern": "^http://prometheus.k3d.localhost:50080"
},
{
"pattern": "^https://docs.github.com/.*$"
}
]
}
12 changes: 10 additions & 2 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,16 @@ This document contains a historical list of changes between releases. Only
changes that impact end-user behavior are listed; changes to documentation or
internal API changes are not present.

Main (unreleased)
-----------------
v0.24.2 (2022-05-02)
--------------------

### Bugfixes

- Added config watcher delay to prevent race condition in cases where scraping service mode has not gracefully exited. (@mattdurham)

### Other changes

- Update version of node_exporter to include additional metrics for osx. (@v-zhuravlev)

v0.24.1 (2022-04-14)
--------------------
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ require (
github.com/prometheus/consul_exporter v0.7.2-0.20210127095228-584c6de19f23
github.com/prometheus/memcached_exporter v0.9.0
github.com/prometheus/mysqld_exporter v0.13.0
github.com/prometheus/node_exporter v1.3.1
github.com/prometheus/node_exporter v1.3.1-0.20220127103407-4f27a4fd8efa
github.com/prometheus/procfs v0.7.4-0.20211011103944-1a7a2bd3279f
github.com/prometheus/prometheus v1.8.2-0.20220308163432-03831554a519
github.com/prometheus/statsd_exporter v0.22.2
Expand Down
4 changes: 4 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -442,6 +442,7 @@ github.com/cenkalti/backoff/v4 v4.1.2/go.mod h1:scbssz8iZGpm3xbr14ovlUdkxfGXNInq
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/census-instrumentation/opencensus-proto v0.3.0 h1:t/LhUZLVitR1Ow2YOnduCsavhwFUklBMoGVYUCqmCqk=
github.com/census-instrumentation/opencensus-proto v0.3.0/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/certifi/gocertifi v0.0.0-20180118203423-deb3ae2ef261/go.mod h1:GJKEexRPVJrBSOjoqN5VNOIKJ5Q3RViH6eu3puDRwx4=
github.com/certifi/gocertifi v0.0.0-20191021191039-0944d244cd40/go.mod h1:sGbDF6GwGcLpkNXPUTkMRoywsNa/ol15pxFe6ERfguA=
github.com/certifi/gocertifi v0.0.0-20200922220541-2c3bb06c6054/go.mod h1:sGbDF6GwGcLpkNXPUTkMRoywsNa/ol15pxFe6ERfguA=
github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
Expand Down Expand Up @@ -2025,6 +2026,7 @@ github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FI
github.com/pascaldekloe/goe v0.1.0 h1:cBOtyMzM9HTpWjXfbbunk26uA6nG3a8n06Wieeh0MwY=
github.com/pascaldekloe/goe v0.1.0/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
github.com/patrickmn/go-cache v0.0.0-20180527043350-9f6ff22cfff8/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ=
github.com/pborman/getopt v0.0.0-20180811024354-2b5b3bfb099b/go.mod h1:85jBQOZwpVEaDAr341tbn15RS4fCAsIst0qp7i8ex1o=
github.com/pborman/getopt v0.0.0-20190409184431-ee0cd42419d3/go.mod h1:85jBQOZwpVEaDAr341tbn15RS4fCAsIst0qp7i8ex1o=
github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=
github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
Expand Down Expand Up @@ -2134,6 +2136,8 @@ github.com/prometheus/exporter-toolkit v0.7.1/go.mod h1:ZUBIj498ePooX9t/2xtDjeQY
github.com/prometheus/memcached_exporter v0.9.0 h1:FnsCZBpNQJgPAnqNqqKtfWo9X+060V0UkGLRAzOfBes=
github.com/prometheus/memcached_exporter v0.9.0/go.mod h1:SDcRCAJ2r6DZft+IVHAD1BMNgNjxbpEaFVeCxX3FuXM=
github.com/prometheus/node_exporter v1.0.0-rc.0.0.20200428091818-01054558c289/go.mod h1:FGbBv5OPKjch+jNUJmEQpMZytIdyW0NdBtWFcfSKusc=
github.com/prometheus/node_exporter v1.3.1-0.20220127103407-4f27a4fd8efa h1:RD6gVRrJNA14XH/yfofwFGq305usV5t8tWVeS16nGSA=
github.com/prometheus/node_exporter v1.3.1-0.20220127103407-4f27a4fd8efa/go.mod h1:Y7LDbciZw+6s4dNRKtNA344Wltn8XPsDN4Jzm02RB8M=
github.com/prometheus/node_exporter v1.3.1 h1:pdJ5824noBu1O2a3a96uiWDH9Ns2ZTE9OxBlPy9Deik=
github.com/prometheus/node_exporter v1.3.1/go.mod h1:Y7LDbciZw+6s4dNRKtNA344Wltn8XPsDN4Jzm02RB8M=
github.com/prometheus/procfs v0.0.0-20180125133057-cb4147076ac7/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
Expand Down
10 changes: 7 additions & 3 deletions pkg/metrics/cluster/config_watcher.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,9 @@ func newConfigWatcher(log log.Logger, cfg Config, store configstore.Store, im in
if err := w.ApplyConfig(cfg); err != nil {
return nil, err
}
go w.run(ctx)
// Delay duration, this is to prevent a race condition, see method for details
delay := cfg.Lifecycler.HeartbeatPeriod * 3
go w.run(ctx, delay)
return w, nil
}

Expand All @@ -89,9 +91,11 @@ func (w *configWatcher) ApplyConfig(cfg Config) error {
return nil
}

func (w *configWatcher) run(ctx context.Context) {
func (w *configWatcher) run(ctx context.Context, delay time.Duration) {
defer level.Info(w.log).Log("msg", "config watcher run loop exiting")

// This is due to a race condition between the heartbeat and config ring in a very narrow set of circumstances
// https://gist.github.com/mattdurham/c15f27de17a6da97bf2e6a870991c7f2
time.Sleep(delay)
lastReshard := time.Now()

for {
Expand Down