Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test_Agent_Remote_ES_Output flaky due to race #4170

Open
michel-laterman opened this issue Dec 2, 2024 · 0 comments · May be fixed by #4171
Open

Test_Agent_Remote_ES_Output flaky due to race #4170

michel-laterman opened this issue Dec 2, 2024 · 0 comments · May be fixed by #4171
Assignees
@michel-laterman
Labels
flaky-test Unstable or unreliable test cases.

Comments

@michel-laterman
Copy link
Contributor

Flaky Test

https://buildkite.com/elastic/fleet-server/builds/5802#019388a2-40b1-42bf-8b6e-f1c2042fd80f

Stack Trace

=== RUN   Test_Agent_Remote_ES_Output
--
  | remote_es_output_integration_test.go:147: Test fleet server port=37795
  | remote_es_output_integration_test.go:150: Create policy with remote ES output
  | remote_es_output_integration_test.go:181: Create API key and enrollment key for new policy
  | remote_es_output_integration_test.go:214: Enroll agent
  | fleet_integration_test.go:727: map[action:created item:map[access_api_key:X0lPemlKTUJqdThBeUFmVF9iSlU6TW5vQW8tN1VRQ0NjcGxVN3FJVXd6dw== access_api_key_id:_IOziJMBju8AyAfT_bJU actions:<nil> active:true enrolled_at:2024-12-02T18:48:37Z id:84c90f2e-67c0-4179-acfc-32cb048abdbf local_metadata:map[] policy_id:337d887c-628a-42cd-b99b-0c0e14384e68 status:online tags:[] type:PERMANENT user_provided_metadata:<nil>]]
  | remote_es_output_integration_test.go:40: Fake a checkin for agent 84c90f2e-67c0-4179-acfc-32cb048abdbf
  | ==================
  | WARNING: DATA RACE
  | Write at 0x00c0003f4360 by goroutine 3151:
  | runtime.mapassign_faststr()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/runtime/map_faststr.go:223 +0x0
  | github.com/elastic/fleet-server/v7/internal/pkg/bulk.(*Bulker).updateBulkerMap()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/bulk/engine.go:148 +0xef
  | github.com/elastic/fleet-server/v7/internal/pkg/bulk.(*Bulker).CreateAndGetBulker()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/bulk/engine.go:177 +0x64c
  | github.com/elastic/fleet-server/v7/internal/pkg/policy.(*Output).Prepare()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/policy/policy_output.go:64 +0xaa6
  | github.com/elastic/fleet-server/v7/internal/pkg/api.processPolicy()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/handleCheckin.go:833 +0x1027
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*CheckinT).ProcessRequest()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/handleCheckin.go:364 +0x2024
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*CheckinT).handleCheckin()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/handleCheckin.go:145 +0x8de
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*apiServer).AgentCheckin()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/api.go:77 +0x4c4
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*ServerInterfaceWrapper).AgentCheckin.func1()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/openapi.gen.go:2179 +0x10a
  | net/http.HandlerFunc.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2220 +0x47
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*apiVersion).middleware-fm.(*apiVersion).middleware.func1()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/apiVersion.go:80 +0x418
  | net/http.HandlerFunc.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2220 +0x47
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*ServerInterfaceWrapper).AgentCheckin()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/openapi.gen.go:2186 +0x1341
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*ServerInterfaceWrapper).AgentCheckin-fm()
  | <autogenerated>:1 +0x51
  | net/http.HandlerFunc.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2220 +0x47
  | github.com/go-chi/chi/v5.(*ChainHandler).ServeHTTP()
  | /opt/buildkite-agent/go/pkg/mod/github.com/go-chi/chi/[email protected]/chain.go:31 +0x63
  | github.com/go-chi/chi/v5.(*Mux).routeHTTP()
  | /opt/buildkite-agent/go/pkg/mod/github.com/go-chi/chi/[email protected]/mux.go:459 +0x594
  | github.com/go-chi/chi/v5.(*Mux).routeHTTP-fm()
  | <autogenerated>:1 +0x51
  | net/http.HandlerFunc.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2220 +0x47
  | github.com/elastic/fleet-server/v7/internal/pkg/limit.(*Limiter).Wrap.func1.1()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/limit/limiter.go:94 +0x373
  | net/http.HandlerFunc.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2220 +0x47
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*limiter).middleware-fm.(*limiter).middleware.func1()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/router.go:124 +0x2d5
  | net/http.HandlerFunc.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2220 +0x47
  | github.com/go-chi/chi/v5/middleware.Recoverer.func1()
  | /opt/buildkite-agent/go/pkg/mod/github.com/go-chi/chi/[email protected]/middleware/recoverer.go:45 +0xf0
  | net/http.HandlerFunc.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2220 +0x47
  | github.com/elastic/fleet-server/v7/internal/pkg/logger.Middleware.func1()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/logger/http.go:254 +0xfe7
  | net/http.HandlerFunc.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2220 +0x47
  | github.com/go-chi/chi/v5.(*Mux).ServeHTTP()
  | /opt/buildkite-agent/go/pkg/mod/github.com/go-chi/chi/[email protected]/mux.go:90 +0x69e
  | net/http.serverHandler.ServeHTTP()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:3210 +0x2a1
  | net/http.(*conn).serve()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:2092 +0x12a4
  | net/http.(*Server).Serve.gowrap3()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:3360 +0x4f
  |  
  | Previous read at 0x00c0003f4360 by goroutine 3088:
  | runtime.mapiterinit()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/runtime/map.go:877 +0x0
  | github.com/elastic/fleet-server/v7/internal/pkg/policy.reportOutputHealth()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/policy/self.go:265 +0xa9
  | github.com/elastic/fleet-server/v7/internal/pkg/policy.(*selfMonitorT).updateState()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/policy/self.go:216 +0x5bc
  | github.com/elastic/fleet-server/v7/internal/pkg/policy.(*selfMonitorT).processPolicies()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/policy/self.go:183 +0x694
  | github.com/elastic/fleet-server/v7/internal/pkg/policy.(*selfMonitorT).Run()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/policy/self.go:122 +0x912
  | github.com/elastic/fleet-server/v7/internal/pkg/policy.SelfMonitor.Run-fm()
  | <autogenerated>:1 +0x61
  | github.com/elastic/fleet-server/v7/internal/pkg/server.loggedRunFunc.func1()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/server/fleet.go:303 +0xb8
  | golang.org/x/sync/errgroup.(*Group).Go.func1()
  | /opt/buildkite-agent/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x91
  |  
  | Goroutine 3151 (running) created at:
  | net/http.(*Server).Serve()
  | /opt/buildkite-agent/.gvm/versions/go1.23.3.linux.amd64/src/net/http/server.go:3360 +0x8ec
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*server).Run.func3()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/server.go:122 +0x110
  | github.com/elastic/fleet-server/v7/internal/pkg/api.(*server).Run.gowrap1()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/api/server.go:125 +0x82
  |  
  | Goroutine 3088 (running) created at:
  | golang.org/x/sync/errgroup.(*Group).Go()
  | /opt/buildkite-agent/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x124
  | github.com/elastic/fleet-server/v7/internal/pkg/server.(*Fleet).runSubsystems()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/server/fleet.go:504 +0x1898
  | github.com/elastic/fleet-server/v7/internal/pkg/server.(*Fleet).runServer()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/server/fleet.go:429 +0x6ae
  | github.com/elastic/fleet-server/v7/internal/pkg/server.(*Fleet).Run.func4()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/server/fleet.go:199 +0x48
  | github.com/elastic/fleet-server/v7/internal/pkg/server.(*Fleet).Run.func2.1()
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1733164329918929620/elastic/fleet-server/internal/pkg/server/fleet.go:128 +0x64
  | golang.org/x/sync/errgroup.(*Group).Go.func1()
  | /opt/buildkite-agent/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x91
  | ==================
  | remote_es_output_integration_test.go:50: Checkin successful, verify body
  | remote_es_output_integration_test.go:110: Fake an ack for action policy:337d887c-628a-42cd-b99b-0c0e14384e68:1 for agent 84c90f2e-67c0-4179-acfc-32cb048abdbf
  | remote_es_output_integration_test.go:129: Ack successful, verify body
  | remote_es_output_integration_test.go:234: Update policy to remove remote ES output
  | remote_es_output_integration_test.go:256: Checkin so that agent gets new policy revision
  | remote_es_output_integration_test.go:40: Fake a checkin for agent 84c90f2e-67c0-4179-acfc-32cb048abdbf
  | remote_es_output_integration_test.go:50: Checkin successful, verify body
  | remote_es_output_integration_test.go:259: Ack so that fleet triggers remote api key invalidate
  | remote_es_output_integration_test.go:110: Fake an ack for action policy:337d887c-628a-42cd-b99b-0c0e14384e68:2 for agent 84c90f2e-67c0-4179-acfc-32cb048abdbf
  | remote_es_output_integration_test.go:129: Ack successful, verify body
  | testing.go:1399: race detected during execution of test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@michel-laterman michel-laterman

Labels
flaky-test Unstable or unreliable test cases.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove race condition when accessing remote bulker map michel-laterman/fleet-server
1 participant
@michel-laterman