Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data race in internal/broadcasters.go #102

Closed
dlamotte opened this issue Feb 21, 2024 · 5 comments · Fixed by #151
Closed

Data race in internal/broadcasters.go #102

dlamotte opened this issue Feb 21, 2024 · 5 comments · Fixed by #151

Comments

@dlamotte
Copy link

Is this a support request?

No.

Describe the bug

https://github.com/launchdarkly/go-server-sdk/blob/v7/internal/broadcasters.go#L70 accesses b.subscribers without guarding the access with a mutex.

To reproduce
go test -race You might need -count=100. Our units running with -race -count=100 routinely fail here.

Expected behavior
No data races.

Logs
... redacted aspects of our codebase as they are not important.

==================
WARNING: DATA RACE
Read at 0x00c000496180 by goroutine 67:
  github.com/launchdarkly/go-server-sdk/v7/internal.(*Broadcaster[go.shape.struct { Key string }]).HasListeners()
      /Users/dlamotte/go/1.21.4/pkg/mod/github.com/launchdarkly/go-server-sdk/[email protected]/internal/broadcasters.go:70 +0x5c
  github.com/launchdarkly/go-server-sdk/v7/internal/datasource.(*DataSourceUpdateSinkImpl).Init()
      /Users/dlamotte/go/1.21.4/pkg/mod/github.com/launchdarkly/go-server-sdk/[email protected]/internal/datasource/data_source_update_sink_impl.go:60 +0x38
  github.com/launchdarkly/go-server-sdk/v7/ldfiledata.(*fileDataSource).reload()
      /Users/dlamotte/go/1.21.4/pkg/mod/github.com/launchdarkly/go-server-sdk/[email protected]/ldfiledata/file_data_source_impl.go:111 +0x610
  github.com/launchdarkly/go-server-sdk/v7/ldfiledata.(*fileDataSource).reload-fm()
      <autogenerated>:1 +0x34
  github.com/launchdarkly/go-server-sdk/v7/ldfilewatch.(*fileWatcher).run()
      /Users/dlamotte/go/1.21.4/pkg/mod/github.com/launchdarkly/go-server-sdk/[email protected]/ldfilewatch/watched_file_data_source.go:67 +0x1b4
  github.com/launchdarkly/go-server-sdk/v7/ldfilewatch.WatchFiles.func1()
      /Users/dlamotte/go/1.21.4/pkg/mod/github.com/launchdarkly/go-server-sdk/[email protected]/ldfilewatch/watched_file_data_source.go:44 +0x40

Previous write at 0x00c000496180 by goroutine 38:
  github.com/launchdarkly/go-server-sdk/v7/internal.(*Broadcaster[go.shape.struct { Key string }]).Close()
      /Users/dlamotte/go/1.21.4/pkg/mod/github.com/launchdarkly/go-server-sdk/[email protected]/internal/broadcasters.go:92 +0xf4
  github.com/launchdarkly/go-server-sdk/v7.(*LDClient).Close()
      /Users/dlamotte/go/1.21.4/pkg/mod/github.com/launchdarkly/go-server-sdk/[email protected]/ldclient.go:539 +0x220
...
  testing.(*common).Cleanup.func1()
      /Users/dlamotte/.goenv/versions/1.21.4/src/testing/testing.go:1169 +0x134
  testing.(*common).runCleanup()
      /Users/dlamotte/.goenv/versions/1.21.4/src/testing/testing.go:1347 +0x148
  testing.tRunner.func2()
      /Users/dlamotte/.goenv/versions/1.21.4/src/testing/testing.go:1589 +0x48
  runtime.deferreturn()
      /Users/dlamotte/.goenv/versions/1.21.4/src/runtime/panic.go:477 +0x34
  testing.(*T).Run.func1()
      /Users/dlamotte/.goenv/versions/1.21.4/src/testing/testing.go:1648 +0x40

Goroutine 67 (running) created at:
  github.com/launchdarkly/go-server-sdk/v7/ldfilewatch.WatchFiles()
      /Users/dlamotte/go/1.21.4/pkg/mod/github.com/launchdarkly/go-server-sdk/[email protected]/ldfilewatch/watched_file_data_source.go:44 +0x2c0
  github.com/launchdarkly/go-server-sdk/v7/ldfiledata.(*fileDataSource).Start()
      /Users/dlamotte/go/1.21.4/pkg/mod/github.com/launchdarkly/go-server-sdk/[email protected]/ldfiledata/file_data_source_impl.go:80 +0x200
  github.com/launchdarkly/go-server-sdk/v7.MakeCustomClient()
      /Users/dlamotte/go/1.21.4/pkg/mod/github.com/launchdarkly/go-server-sdk/[email protected]/ldclient.go:292 +0x1aa4
...
  testing.tRunner()
      /Users/dlamotte/.goenv/versions/1.21.4/src/testing/testing.go:1595 +0x194
  testing.(*T).Run.func1()
      /Users/dlamotte/.goenv/versions/1.21.4/src/testing/testing.go:1648 +0x40

Goroutine 38 (finished) created at:
  testing.(*T).Run()
      /Users/dlamotte/.goenv/versions/1.21.4/src/testing/testing.go:1648 +0x5d8
  testing.runTests.func1()
      /Users/dlamotte/.goenv/versions/1.21.4/src/testing/testing.go:2054 +0x80
  testing.tRunner()
      /Users/dlamotte/.goenv/versions/1.21.4/src/testing/testing.go:1595 +0x194
  testing.runTests()
      /Users/dlamotte/.goenv/versions/1.21.4/src/testing/testing.go:2052 +0x6d8
  testing.(*M).Run()
      /Users/dlamotte/.goenv/versions/1.21.4/src/testing/testing.go:1925 +0x908
  main.main()
      _testmain.go:57 +0x294
==================

SDK version

github.com/launchdarkly/go-server-sdk/v7 v7.0.0

Language version, developer tools
go version go1.21.4 darwin/arm64

OS/platform
OSX.

Additional context
Impact is that we cannot use -race when running our unit tests to find races. So we have to have two unit test runs until this is fixed which is not ideal as we actually had a data race in our code on top of this code. So, I'd prefer to eliminate this race so we can ensure we don't have races in our code.
@cwaldren-ld
Copy link
Contributor

Hi @dlamotte, thank you for the report. I'll investigate.

@cwaldren-ld
Copy link
Contributor

Can you post your SDK configuration?

@dlamotte
Copy link
Author

dlamotte commented Feb 21, 2024 •
edited
Loading

@cwaldren-ld I think you want this?

import (
...
ld "github.com/launchdarkly/go-server-sdk/v7"
)

	var config ld.Config
	{
		if conf := flags.options.Config; conf != nil {
			config = ld.Config{
				DataSource: ldfiledata.DataSource().
					FilePaths(*conf).
					Reloader(ldfilewatch.WatchFiles),
				Events: ldcomponents.NoEvents(),
			}
		} else {
			config = ld.Config{
				DataSource: ldcomponents.StreamingDataSource().InitialReconnectDelay(5 * time.Second),
				Offline:    false,
			}
		}

		config.DiagnosticOptOut = true
	}

	config.Logging = ldcomponents.Logging().MinLevel(ldlog.Error)

	client, err := ld.MakeCustomClient(apikey, config, 5*time.Second)

It's a partial. Hopefully its enough.

@JohnStarich JohnStarich mentioned this issue May 16, 2024
Merged
3 tasks
cwaldren-ld pushed a commit that referenced this issue May 28, 2024
@JohnStarich
Fix broadcast data races (#151)
9b387be 
**Requirements**

- [x] I have added test coverage for new or changed functionality
- [x] I have followed the repository's [pull request submission
guidelines](../blob/v5/CONTRIBUTING.md#submitting-pull-requests)
- [ ] I have validated my changes against all supported platform
versions
* What does "platform" mean here? Go version? OS version? Happy to help
test, just not sure what I'm testing!

**Related issues**

I hit data races when running tests in parallel, which effectively
obscures other data races my application may have. It looks like others
did too, judging by #102.
Fixes #102

**Describe the solution you've provided**

I've addressed 2 data races with improved locking:
* `HasListeners()` had a data race due to any mutation on
`b.subscribers`
* `Close()` had a data race where closing a send channel triggers a
panic in `Broadcast()`

I also saw an easy opportunity to use more fine-grained locks with an
RWMutex, although I'm happy to back that out if you would prefer.

**Describe alternatives you've considered**

I also considered using an atomic data type for subscribers, but I
figured that change would be less of a surgical fix. It also may be more
difficult to mutate the slice, since a compare-and-swap can fail and
would need a loop (it's not as simple as `atomic.Add`).

Another idea which may be simpler is using a channel to manage shutdown.
Typically I'd use a `context.Context` here to manage cancellation.
That'd also prevent `Broadcast()`ing on a full send channel from
blocking `Close()`.

**Additional context**

Add any other context about the pull request here.
@dlamotte
Copy link
Author

dlamotte commented May 29, 2024 •
edited
Loading

Appears fixed by #151 (looks like there is another follow-on PR, but best follow those PRs)

@cwaldren-ld
Copy link
Contributor

This has been released as part of https://github.com/launchdarkly/go-server-sdk/releases/tag/v7.4.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix broadcast data races JohnStarich/launchdarkly-go-server-sdk
2 participants
@dlamotte @cwaldren-ld