Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error syncing wallet via RPC client on macOS system #749

Open
notmandatory opened this issue Sep 10, 2022 · 2 comments
Open

Error syncing wallet via RPC client on macOS system #749

notmandatory opened this issue Sep 10, 2022 · 2 comments
Labels
bug Something isn't working

Comments

@notmandatory
Copy link
Member

Describe the bug

Unable to sync a regtest wallet with a local bitcoind node via the rpc client on a macOS system using bdk-cli.

To Reproduce

From the bdk-cli repo:

RUST_LOG=debug cargo test --features regtest-bitcoin -- --nocapture

Error is:

thread 'test::test_basic_wallet_op_bitcoind' panicked at 'called `Result::unwrap()` on an `Err` value: CmdExec("[2022-09-10T03:19:40Z ERROR bdk_cli] Rpc(JsonRpc(Transport(SocketError(Os { code: 35, kind: WouldBlock, message: \"Resource temporarily unavailable\" }))))\n")', tests/integration.rs:224:40

Expected behavior

The above test passes on GitHub CI action hosted on Ubuntu 20.04 and should also pass for a macOS system.

Build environment

  • BDK tag/commit: v0.22.0
  • OS+version: macOS 12.5.1, M1
  • Rust/Cargo version: 1.63.0
  • Rust/Cargo target: aarch64-apple-darwin

Additional context

I think this issue is related to apoelstra/rust-jsonrpc#67.

Found this issue while testing bitcoindevkit/bdk-cli#102.
@notmandatory notmandatory added the bug Something isn't working label Sep 10, 2022
@afilini
Copy link
Member

afilini commented Sep 12, 2022

I think the only workaround we have for this is to run the tests with fewer parallel threads, this generally tends to fix the issue.

I think it doesn't happen in the CI because the workers don't have that many cores available, so only a few tests at a time are run in parallel.

@notmandatory
Copy link
Member Author

I tried rerunning cargo test with a single thread, but got the same error.

 RUST_TEST_THREADS=1 cargo test --features regtest-bitcoin

My new theory is that we're running into an issue with bitcoind on macOS where the sqlite DB is very slow using the static linked lib, but runs fine with the MacOS system provided lib (For details see: bitcoin/bitcoin#25724). I re-ran my broken test with a modified version of the utils::new_backend function that uses the env to find my homebrew install of bitcoind and the test is now passing.

BITCOIND_EXE=/opt/homebrew/bin/bitcoind cargo test --features regtest-bitcoin

@fanquake fanquake mentioned this issue Sep 16, 2022
Merged
achow101 added a commit to bitcoin-core/gui that referenced this issue Oct 17, 2022
@achow101
Merge bitcoin/bitcoin#25985: Revert "build: Use Homebrew's sqlite pac…
a52ff61 
…kage if it is available"

d216d71 Revert "build: Use Homebrew's sqlite package if it is available" (fanquake)

Pull request description:

  This reverts ee7b84e from #20527.

  That change was made without any rationale, maybe other than, a brew
  installed version might be newer, and that's "better". However when
  building from source on macOS, it just results in drastically worse
  performance, and issues / confusion like #25724.

  The difference in performance can be observed using the example from bitcoin/bitcoin#25724 (comment),
  but minified i.e:
  ```bash
  time src/bitcoin-cli createwallet speedy true
  time src/bitcoin-cli importdescriptors '[
    {"desc":"raw(00145846369f3d6ba366d6f5a903fb5cf4dca3763c0e)#k9wh6v62","timestamp":"now"},
    {"desc":"raw(001420800aabf13f3a4c4ce3ce4c66cecf1d17f21a6e)#6m0hlfh4","timestamp":"now"},
    {"desc":"raw(0014c6bf9715e06d73ebf9b3b02d5cc48d24d8bbabc1)#wyavh36r","timestamp":"now"},
    {"desc":"raw(00141ba7807b3f46af113beaea5c698428ce7138cd8a)#jctdsups","timestamp":"now"},
    {"desc":"raw(00140c1bd27f10fff01b36ddf3c1febaa1acff19b080)#9s6nc3pk","timestamp":"now"},
    {"desc":"raw(00141226e31987e4bc2e63c0ee12908f675e40464b20)#9pp7qm39","timestamp":"now"},
    {"desc":"raw(0014f73f149f7503960a5e849c6ee7a8a8c336f631cb)#qtkxv9fc","timestamp":"now"},
    {"desc":"raw(0014c8ccb4d81ffc769fc5fdd8d7eed69b0e0cae5749)#hn39qayv","timestamp":"now"},
    {"desc":"raw(001498565aead2d67a22a6021d55210f2a917fc22169)#6ar3vwsx","timestamp":"now"},
    {"desc":"raw(001403013248ac0cd9eabe176cad162cda2a19f771e1)#4m47mukd","timestamp":"now"}
  ]'
  ```

  Running master, when building from souce and using brew installed
  sqlite, this takes ~3.4s. With this PR, the same operation takes ~0.3s.

  Resolves the "build from source" portion of #25724. Building from
  depends is still not ideal, however I have some other changes that might
  help improve things in that case.

  Related performance issue reports:
  * bitcoindevkit/bdk#749
  * https://bitcoin.stackexchange.com/questions/113898/bitcoin-v23-is-10-times-slower-than-v22-on-macos-for-basic-regtest-tests
  * bitcoin/bitcoin#25724
  * bitcoin/bitcoin#25985 (comment)

ACKs for top commit:
  achow101:
    ACK d216d71
  jarolrod:
    ACK d216d71
  hebasto:
    ACK d216d71, I have reviewed the code and it looks OK, I agree it can be merged. No conflicts with our build [docs](https://github.com/bitcoin/bitcoin/blob/d216d714aae36e6f1c95f82aef81a0be74dee2f3/doc/build-osx.md#descriptor-wallet-support).

Tree-SHA512: 1bb4b44385b11fa9fe66edd7449278f9e47a6cc679b7111f9adf17db94c34e29c9cceafc917454e134420db40b24b56da29226af6f43e6dbeff822b79b77ed60
sidhujag pushed a commit to syscoin/syscoin that referenced this issue Oct 23, 2022
@achow101
Merge bitcoin#25985: Revert "build: Use Homebrew's sqlite package if …
ab43cfc 
…it is available"

d216d71 Revert "build: Use Homebrew's sqlite package if it is available" (fanquake)

Pull request description:

  This reverts ee7b84e from bitcoin#20527.

  That change was made without any rationale, maybe other than, a brew
  installed version might be newer, and that's "better". However when
  building from source on macOS, it just results in drastically worse
  performance, and issues / confusion like bitcoin#25724.

  The difference in performance can be observed using the example from bitcoin#25724 (comment),
  but minified i.e:
  ```bash
  time src/bitcoin-cli createwallet speedy true
  time src/bitcoin-cli importdescriptors '[
    {"desc":"raw(00145846369f3d6ba366d6f5a903fb5cf4dca3763c0e)#k9wh6v62","timestamp":"now"},
    {"desc":"raw(001420800aabf13f3a4c4ce3ce4c66cecf1d17f21a6e)#6m0hlfh4","timestamp":"now"},
    {"desc":"raw(0014c6bf9715e06d73ebf9b3b02d5cc48d24d8bbabc1)#wyavh36r","timestamp":"now"},
    {"desc":"raw(00141ba7807b3f46af113beaea5c698428ce7138cd8a)#jctdsups","timestamp":"now"},
    {"desc":"raw(00140c1bd27f10fff01b36ddf3c1febaa1acff19b080)#9s6nc3pk","timestamp":"now"},
    {"desc":"raw(00141226e31987e4bc2e63c0ee12908f675e40464b20)#9pp7qm39","timestamp":"now"},
    {"desc":"raw(0014f73f149f7503960a5e849c6ee7a8a8c336f631cb)#qtkxv9fc","timestamp":"now"},
    {"desc":"raw(0014c8ccb4d81ffc769fc5fdd8d7eed69b0e0cae5749)#hn39qayv","timestamp":"now"},
    {"desc":"raw(001498565aead2d67a22a6021d55210f2a917fc22169)#6ar3vwsx","timestamp":"now"},
    {"desc":"raw(001403013248ac0cd9eabe176cad162cda2a19f771e1)#4m47mukd","timestamp":"now"}
  ]'
  ```

  Running master, when building from souce and using brew installed
  sqlite, this takes ~3.4s. With this PR, the same operation takes ~0.3s.

  Resolves the "build from source" portion of bitcoin#25724. Building from
  depends is still not ideal, however I have some other changes that might
  help improve things in that case.

  Related performance issue reports:
  * bitcoindevkit/bdk#749
  * https://bitcoin.stackexchange.com/questions/113898/bitcoin-v23-is-10-times-slower-than-v22-on-macos-for-basic-regtest-tests
  * bitcoin#25724
  * bitcoin#25985 (comment)

ACKs for top commit:
  achow101:
    ACK d216d71
  jarolrod:
    ACK d216d71
  hebasto:
    ACK d216d71, I have reviewed the code and it looks OK, I agree it can be merged. No conflicts with our build [docs](https://github.com/bitcoin/bitcoin/blob/d216d714aae36e6f1c95f82aef81a0be74dee2f3/doc/build-osx.md#descriptor-wallet-support).

Tree-SHA512: 1bb4b44385b11fa9fe66edd7449278f9e47a6cc679b7111f9adf17db94c34e29c9cceafc917454e134420db40b24b56da29226af6f43e6dbeff822b79b77ed60
@rajarshimaitra rajarshimaitra mentioned this issue Feb 10, 2023
Open
@riverKanies riverKanies mentioned this issue Nov 15, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
BDK Maintenance
Status: No status
Milestone
No milestone
Development

No branches or pull requests
2 participants
@afilini @notmandatory