Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] "Unexpected fault address" when opening SQLite db under armv7 #2188

Closed
4 tasks done
m-kloeckner opened this issue Oct 11, 2024 · 11 comments · Fixed by #2270
Closed
4 tasks done

[Bug] "Unexpected fault address" when opening SQLite db under armv7 #2188

m-kloeckner opened this issue Oct 11, 2024 · 11 comments · Fixed by #2270
Labels
bug Something isn't working good first issue Good for newcomers help wanted Extra attention is needed
Milestone

Comments

@m-kloeckner
Copy link

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

I am running headscale on a armv7 SOC (Cubieboard2) in a docker container since quite a while now. After switching to version 0.23 I'm running in an error.

When trying to start headscale using the official headscale/headscale docker image for armv7 the process fails with the following error after trying to open the SQLite db.

headscale | 2024-10-11T08:13:30Z WRN
headscale | WARN: The "dns.use_username_in_magic_dns" configuration key is deprecated and has been removed. Please see the changelog for more details.
headscale |
headscale | 2024-10-11T08:13:30Z INF No private key file at path, creating... path=/var/lib/headscale/noise_private.key
headscale | 2024-10-11T08:13:30Z INF Opening database database=sqlite3 path=/etc/headscale/db.sqlite
headscale | unexpected fault address 0x769affff
headscale | fatal error: fault
headscale | [signal SIGBUS: bus error code=0x2 addr=0x769affff pc=0x979e58]
headscale |
headscale | goroutine 1 gp=0x3402128 m=4 mp=0x3461408 [running]:
headscale | runtime.throw({0x175ca21, 0x5})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/panic.go:1067 +0x34 fp=0x34d4e1c sp=0x34d4e08 pc=0x9292c
headscale | runtime.sigpanic()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/signal_unix.go:897 +0x104 fp=0x34d4e4c sp=0x34d4e1c pc=0x94cd0
headscale | modernc.org/libc.Xmemset(0x35ca360, 0x769a8088, 0x0, 0x7f78)
headscale | /home/runner/go/pkg/mod/modernc.org/[email protected]/ccgo_linux_arm.go:146120 +0x24 fp=0x34d4e50 sp=0x34d4e50 pc=0x979e58
headscale | modernc.org/sqlite/lib._walIndexAppend(0x35ca360, 0x76600718, 0x1, 0x1)
headscale | /home/runner/go/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_arm.go:43350 +0xe4 fp=0x34d4e88 sp=0x34d4e50 pc=0x9caf58
headscale | modernc.org/sqlite/lib._walFrames(0x35ca360, 0x76600718, 0x1000, 0x75b07038, 0x1, 0x1, 0xa)
headscale | /home/runner/go/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_arm.go:46136 +0xa48 fp=0x34d4f0c sp=0x34d4e88 pc=0x9d1404
headscale | modernc.org/sqlite/lib._sqlite3WalFrames(0x35ca360, 0x76600718, 0x1000, 0x75b07038, 0x1, 0x1, 0xa)
headscale | /home/runner/go/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_arm.go:46177 +0x4c fp=0x34d4f30 sp=0x34d4f0c pc=0x9d1728
headscale | modernc.org/sqlite/lib._pagerWalFrames(0x35ca360, 0x76000418, 0x75b07038, 0x1, 0x1)
headscale | /home/runner/go/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_arm.go:37777 +0x114 fp=0x34d4f70 sp=0x34d4f30 pc=0x9c1d88
headscale | modernc.org/sqlite/lib._sqlite3PagerCommitPhaseOne(0x35ca360, 0x76000418, 0x0, 0x0)
headscale | /home/runner/go/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_arm.go:40996 +0x1b0 fp=0x34d4fa4 sp=0x34d4f70 pc=0x9c8510
headscale | modernc.org/sqlite/lib._sqlite3BtreeCommitPhaseOne(0x35ca360, 0x75e00058, 0x0)
headscale | /home/runner/go/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_arm.go:51705 +0xbc fp=0x34d4fc4 sp=0x34d4fa4 pc=0x9db3a0
headscale | modernc.org/sqlite/lib._vdbeCommit(0x35ca360, 0x76000018, 0x75a08808)
headscale | /home/runner/go/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_arm.go:64640 +0xdcc fp=0x34d508c sp=0x34d4fc4 pc=0x9fbc6c
headscale | modernc.org/sqlite/lib._sqlite3VdbeHalt(0x35ca360, 0x75a08808)
headscale | /home/runner/go/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_arm.go:65053 +0x348 fp=0x34d50b4 sp=0x34d508c pc=0x9fc42c
headscale | modernc.org/sqlite/lib._sqlite3VdbeExec(0x35ca360, 0x75a08808)
headscale | /home/runner/go/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_arm.go:71196 +0x4920 fp=0x34d5594 sp=0x34d50b4 pc=0xa0c450
headscale | modernc.org/sqlite/lib._sqlite3Step(0x35ca360, 0x75a08808)
headscale | /home/runner/go/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_arm.go:67892 +0x8c fp=0x34d55b4 sp=0x34d5594 pc=0xa02ad0
headscale | modernc.org/sqlite/lib.Xsqlite3_step(0x35ca360, 0x75a08808)
headscale | /home/runner/go/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_arm.go:67959 +0xdc fp=0x34d55e4 sp=0x34d55b4 pc=0xa02f2c
headscale | github.com/glebarez/go-sqlite.(*conn).step(0x348b7e0, 0x75a08808)
headscale | /home/runner/go/pkg/mod/github.com/glebarez/[email protected]/sqlite.go:1002 +0x28 fp=0x34d55fc sp=0x34d55e4 pc=0xb68c90
headscale | github.com/glebarez/go-sqlite.(*stmt).exec.func1(0x34fa6c8, 0x34d566c, {0x27acfa8, 0x0, 0x0})
headscale | /home/runner/go/pkg/mod/github.com/glebarez/[email protected]/sqlite.go:536 +0x128 fp=0x34d5648 sp=0x34d55fc pc=0xb66aac
headscale | github.com/glebarez/go-sqlite.(*stmt).exec(0x34fa6c8, {0x1a6deb4, 0x27acfa8}, {0x27acfa8, 0x0, 0x0})
headscale | /home/runner/go/pkg/mod/github.com/glebarez/[email protected]/sqlite.go:549 +0x1a0 fp=0x34d5680 sp=0x34d5648 pc=0xb66820
headscale | github.com/glebarez/go-sqlite.(*stmt).ExecContext(0x34fa6c8, {0x1a6deb4, 0x27acfa8}, {0x27acfa8, 0x0, 0x0})
headscale | /home/runner/go/pkg/mod/github.com/glebarez/[email protected]/sqlite_go18.go:43 +0x44 fp=0x34d56ac sp=0x34d5680 pc=0xb6ba98
headscale | database/sql.ctxDriverStmtExec({0x1a6deb4, 0x27acfa8}, {0x1a6e584, 0x34fa6c8}, {0x27acfa8, 0x0, 0x0})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/ctxutil.go:65 +0xb0 fp=0x34d56fc sp=0x34d56ac pc=0x730da0
headscale | database/sql.resultFromStatement({0x1a6deb4, 0x27acfa8}, {0x1a6d0a0, 0x348b7e0}, 0x36e4800, {0x363a580, 0x0, 0x8})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:2672 +0xfc fp=0x34d5750 sp=0x34d56fc pc=0x73bce8
headscale | database/sql.(*Stmt).ExecContext.func1(0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:2646 +0xb8 fp=0x34d57a8 sp=0x34d5750 pc=0x73ba88
headscale | database/sql.(*DB).retry(0x3508dc8, 0x34d57f0)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:1568 +0x78 fp=0x34d57d0 sp=0x34d57a8 pc=0x736d5c
headscale | database/sql.(*Stmt).ExecContext(0x35d82a0, {0x1a6deb4, 0x27acfa8}, {0x363a580, 0x0, 0x8})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:2640 +0xe0 fp=0x34d5820 sp=0x34d57d0 pc=0x73b978
headscale | gorm.io/gorm.(*PreparedStmtDB).ExecContext(0x36e4760, {0x1a6deb4, 0x27acfa8}, {0x3506780, 0x7e}, {0x363a580, 0x0, 0x8})
headscale | /home/runner/go/pkg/mod/gorm.io/[email protected]/prepare_stmt.go:151 +0xfc fp=0x34d58a0 sp=0x34d5820 pc=0x7c234c
headscale | gorm.io/gorm/callbacks.RawExec(0x36e47c0)
headscale | /home/runner/go/pkg/mod/gorm.io/[email protected]/callbacks/raw.go:9 +0x90 fp=0x34d58d4 sp=0x34d58a0 pc=0x967b2c
headscale | gorm.io/gorm.(*processor).Execute(0x3718330, 0x36e47c0)
headscale | /home/runner/go/pkg/mod/gorm.io/[email protected]/callbacks.go:130 +0x3fc fp=0x34d5954 sp=0x34d58d4 pc=0x7b3194
headscale | gorm.io/gorm.(*DB).Exec(0x348b680, {0x17dd172, 0x7e}, {0x0, 0x0, 0x0})
headscale | /home/runner/go/pkg/mod/gorm.io/[email protected]/finisher_api.go:769 +0x188 fp=0x34d5994 sp=0x34d5954 pc=0x7be7ec
headscale | github.com/juanfont/headscale/hscontrol/db.openDB({{0x1760531, 0x7}, 0x0, {0x0, 0x3b9aca00, 0x1, 0x1, 0x1}, {{0x34cb4e8, 0x18}, ...}, ...})
headscale | /home/runner/work/headscale/headscale/hscontrol/db/db.go:462 +0xc3c fp=0x34d5a84 sp=0x34d5994 pc=0xd182ec
headscale | github.com/juanfont/headscale/hscontrol/db.NewHeadscaleDatabase({{0x1760531, 0x7}, 0x0, {0x0, 0x3b9aca00, 0x1, 0x1, 0x1}, {{0x34cb4e8, 0x18}, ...}, ...}, ...)
headscale | /home/runner/work/headscale/headscale/hscontrol/db/db.go:45 +0x38 fp=0x34d5b40 sp=0x34d5a84 pc=0xd14454
headscale | github.com/juanfont/headscale/hscontrol.NewHeadscale(0x37386c8)
headscale | /home/runner/work/headscale/headscale/hscontrol/app.go:139 +0x188 fp=0x34d5cc8 sp=0x34d5b40 pc=0x13eb474
headscale | github.com/juanfont/headscale/cmd/headscale/cli.newHeadscaleServerWithConfig()
headscale | /home/runner/work/headscale/headscale/cmd/headscale/cli/utils.go:35 +0x8c fp=0x34d5cf0 sp=0x34d5cc8 pc=0x141e14c
headscale | github.com/juanfont/headscale/cmd/headscale/cli.init.func28(0x2795238, {0x27acfa8, 0x0, 0x0})
headscale | /home/runner/work/headscale/headscale/cmd/headscale/cli/serve.go:22 +0x14 fp=0x34d5d14 sp=0x34d5cf0 pc=0x14171bc
headscale | github.com/spf13/cobra.(*Command).execute(0x2795238, {0x27acfa8, 0x0, 0x0})
headscale | /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:989 +0x9b0 fp=0x34d5dd4 sp=0x34d5d14 pc=0x867fb8
headscale | github.com/spf13/cobra.(*Command).ExecuteC(0x2794968)
headscale | /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117 +0x44c fp=0x34d5e4c sp=0x34d5dd4 pc=0x868890
headscale | github.com/spf13/cobra.(*Command).Execute(...)
headscale | /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041
headscale | github.com/juanfont/headscale/cmd/headscale/cli.Execute()
headscale | /home/runner/work/headscale/headscale/cmd/headscale/cli/root.go:97 +0x20 fp=0x34d5e78 sp=0x34d5e4c pc=0x141d438
headscale | main.main()
headscale | /home/runner/work/headscale/headscale/cmd/headscale/headscale.go:42 +0x1a4 fp=0x34d5fa8 sp=0x34d5e78 pc=0x141f93c
headscale | runtime.main()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:272 +0x2ec fp=0x34d5fec sp=0x34d5fa8 pc=0x54fe4
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x34d5fec sp=0x34d5fec pc=0x9a508
headscale |
headscale | goroutine 2 gp=0x3402488 m=nil [force gc (idle)]:
headscale | runtime.gopark(0x183e348, 0x279d4e0, 0x11, 0xa, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345cfd4 sp=0x345cfc0 pc=0x92a6c
headscale | runtime.goparkunlock(...)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:430
headscale | runtime.forcegchelper()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:337 +0xe4 fp=0x345cfec sp=0x345cfd4 pc=0x55450
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345cfec sp=0x345cfec pc=0x9a508
headscale | created by runtime.init.6 in goroutine 1
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:325 +0x1c
headscale |
headscale | goroutine 3 gp=0x34027e8 m=nil [GC sweep wait]:
headscale | runtime.gopark(0x183e348, 0x279e360, 0xc, 0x9, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345d7c4 sp=0x345d7b0 pc=0x92a6c
headscale | runtime.goparkunlock(...)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:430
headscale | runtime.bgsweep(0x347a000)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgcsweep.go:317 +0x11c fp=0x345d7e4 sp=0x345d7c4 pc=0x3af10
headscale | runtime.gcenable.gowrap1()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:203 +0x28 fp=0x345d7ec sp=0x345d7e4 pc=0x2ad90
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345d7ec sp=0x345d7ec pc=0x9a508
headscale | created by runtime.gcenable in goroutine 1
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:203 +0x74
headscale |
headscale | goroutine 4 gp=0x3402908 m=nil [GC scavenge wait]:
headscale | runtime.gopark(0x183e348, 0x27a03e8, 0xd, 0xa, 0x2)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345dfb4 sp=0x345dfa0 pc=0x92a6c
headscale | runtime.goparkunlock(...)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:430
headscale | runtime.(*scavengerState).park(0x27a03e8)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgcscavenge.go:425 +0x68 fp=0x345dfc8 sp=0x345dfb4 pc=0x3827c
headscale | runtime.bgscavenge(0x347a000)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgcscavenge.go:658 +0x60 fp=0x345dfe4 sp=0x345dfc8 pc=0x389a8
headscale | runtime.gcenable.gowrap2()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:204 +0x28 fp=0x345dfec sp=0x345dfe4 pc=0x2ad3c
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345dfec sp=0x345dfec pc=0x9a508
headscale | created by runtime.gcenable in goroutine 1
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:204 +0xbc
headscale |
headscale | goroutine 17 gp=0x34bc008 m=nil [finalizer wait]:
headscale | runtime.gopark(0x183e1f0, 0x27ad038, 0x10, 0xa, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345c78c sp=0x345c778 pc=0x92a6c
headscale | runtime.runfinq()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mfinal.go:193 +0x110 fp=0x345c7ec sp=0x345c78c pc=0x29bbc
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345c7ec sp=0x345c7ec pc=0x9a508
headscale | created by runtime.createfing in goroutine 1
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mfinal.go:163 +0x5c
headscale |
headscale | goroutine 18 gp=0x34bca28 m=nil [chan receive]:
headscale | runtime.gopark(0x183e1d0, 0x3489534, 0xe, 0x7, 0x2)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345878c sp=0x3458778 pc=0x92a6c
headscale | runtime.chanrecv(0x3489500, 0x0, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/chan.go:639 +0x4e0 fp=0x34587c8 sp=0x345878c pc=0x18648
headscale | runtime.chanrecv1(0x3489500, 0x0)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/chan.go:489 +0x20 fp=0x34587dc sp=0x34587c8 pc=0x18138
headscale | runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1732
headscale | runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1735 +0x40 fp=0x34587ec sp=0x34587dc pc=0x2edb4
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x34587ec sp=0x34587ec pc=0x9a508
headscale | created by unique.runtime_registerUniqueMapCleanup in goroutine 1
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1730 +0xa4
headscale |
headscale | goroutine 51 gp=0x35977a8 m=nil [select]:
headscale | runtime.gopark(0x183e378, 0x0, 0x9, 0x3, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x34596f4 sp=0x34596e0 pc=0x92a6c
headscale | runtime.selectgo(0x34597d0, 0x34597c4, 0x0, 0x0, 0x2, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/select.go:335 +0xb68 fp=0x34597a0 sp=0x34596f4 pc=0x6a574
headscale | github.com/patrickmn/go-cache.(*janitor).Run(0x38f5950, 0x37181b0)
headscale | /home/runner/go/pkg/mod/github.com/patrickmn/[email protected]+incompatible/cache.go:1079 +0x98 fp=0x34597e0 sp=0x34597a0 pc=0xcf9ee4
headscale | github.com/patrickmn/go-cache.runJanitor.gowrap1()
headscale | /home/runner/go/pkg/mod/github.com/patrickmn/[email protected]+incompatible/cache.go:1099 +0x30 fp=0x34597ec sp=0x34597e0 pc=0xcfa0b8
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x34597ec sp=0x34597ec pc=0x9a508
headscale | created by github.com/patrickmn/go-cache.runJanitor in goroutine 1
headscale | /home/runner/go/pkg/mod/github.com/patrickmn/[email protected]+incompatible/cache.go:1099 +0xfc
headscale |
headscale | goroutine 52 gp=0x373afc8 m=nil [select]:
headscale | runtime.gopark(0x183e378, 0x0, 0x9, 0x3, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x3459f00 sp=0x3459eec pc=0x92a6c
headscale | runtime.selectgo(0x3459fdc, 0x3459fd0, 0x0, 0x0, 0x2, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/select.go:335 +0xb68 fp=0x3459fac sp=0x3459f00 pc=0x6a574
headscale | github.com/juanfont/headscale/hscontrol/notifier.(*batcher).doWork(...)
headscale | /home/runner/work/headscale/headscale/hscontrol/notifier/notifier.go:419
headscale | github.com/juanfont/headscale/hscontrol/notifier.NewNotifier.gowrap1()
headscale | /home/runner/work/headscale/headscale/hscontrol/notifier/notifier.go:52 +0x8c fp=0x3459fec sp=0x3459fac pc=0xe51360
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x3459fec sp=0x3459fec pc=0x9a508
headscale | created by github.com/juanfont/headscale/hscontrol/notifier.NewNotifier in goroutine 1
headscale | /home/runner/work/headscale/headscale/hscontrol/notifier/notifier.go:52 +0x1b0
headscale |
headscale | goroutine 15 gp=0x373b0e8 m=nil [GC worker (idle)]:
headscale | runtime.gopark(0x183e200, 0x3442960, 0x1a, 0xa, 0x0)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345ef88 sp=0x345ef74 pc=0x92a6c
headscale | runtime.gcBgMarkWorker(0x37e4e40)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xf4 fp=0x345efe4 sp=0x345ef88 pc=0x2dbec
headscale | runtime.gcBgMarkStartWorkers.gowrap1()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x28 fp=0x345efec sp=0x345efe4 pc=0x2dacc
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345efec sp=0x345efec pc=0x9a508
headscale | created by runtime.gcBgMarkStartWorkers in goroutine 33
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x14c
headscale |
headscale | goroutine 49 gp=0x373b208 m=nil [IO wait]:
headscale | runtime.gopark(0x183e33c, 0x76c02e88, 0x2, 0x2, 0x5)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x36e3ae0 sp=0x36e3acc pc=0x92a6c
headscale | runtime.netpollblock(0x76c02e78, 0x72, 0x0)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/netpoll.go:575 +0x100 fp=0x36e3af8 sp=0x36e3ae0 pc=0x4d420
headscale | internal/poll.runtime_pollWait(0x76c02e78, 0x72)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/netpoll.go:351 +0x54 fp=0x36e3b0c sp=0x36e3af8 pc=0x91c04
headscale | internal/poll.(*pollDesc).wait(0x36d01f8, 0x72, 0x0)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/internal/poll/fd_poll_runtime.go:84 +0x30 fp=0x36e3b20 sp=0x36e3b0c pc=0xe3c20
headscale | internal/poll.(*pollDesc).waitRead(...)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/internal/poll/fd_poll_runtime.go:89
headscale | internal/poll.(*FD).Read(0x36d01e0, {0x37e6000, 0x1000, 0x1000})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/internal/poll/fd_unix.go:165 +0x22c fp=0x36e3b68 sp=0x36e3b20 pc=0xe4d9c
headscale | net.(*netFD).Read(0x36d01e0, {0x37e6000, 0x1000, 0x1000})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/fd_posix.go:55 +0x38 fp=0x36e3b94 sp=0x36e3b68 pc=0x360bec
headscale | net.(*conn).Read(0x3798860, {0x37e6000, 0x1000, 0x1000})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/net.go:189 +0x48 fp=0x36e3bc0 sp=0x36e3b94 pc=0x373be8
headscale | net.(*TCPConn).Read(0x3798860, {0x37e6000, 0x1000, 0x1000})
headscale | :1 +0x44 fp=0x36e3be0 sp=0x36e3bc0 pc=0x388f94
headscale | crypto/tls.(*atLeastReader).Read(0x38f5140, {0x37e6000, 0x1000, 0x1000})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/crypto/tls/conn.go:809 +0x78 fp=0x36e3c0c sp=0x36e3be0 pc=0x3d6208
headscale | bytes.(*Buffer).ReadFrom(0x37395d4, {0x1a62418, 0x38f5140})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/bytes/buffer.go:211 +0xa4 fp=0x36e3c48 sp=0x36e3c0c pc=0x135a08
headscale | crypto/tls.(*Conn).readFromUntil(0x3739448, {0x1a61db8, 0x3798860}, 0x5)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/crypto/tls/conn.go:831 +0xd4 fp=0x36e3c70 sp=0x36e3c48 pc=0x3d6464
headscale | crypto/tls.(*Conn).readRecordOrCCS(0x3739448, 0x0)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/crypto/tls/conn.go:629 +0x134 fp=0x36e3dd0 sp=0x36e3c70 pc=0x3d3b68
headscale | crypto/tls.(*Conn).readRecord(...)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/crypto/tls/conn.go:591
headscale | crypto/tls.(*Conn).Read(0x3739448, {0x35a9000, 0x1000, 0x1000})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/crypto/tls/conn.go:1385 +0x148 fp=0x36e3e00 sp=0x36e3dd0 pc=0x3d9940
headscale | bufio.(*Reader).Read(0x37fd440, {0x34da024, 0x9, 0x9})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/bufio/bufio.go:241 +0x214 fp=0x36e3e24 sp=0x36e3e00 pc=0x201338
headscale | io.ReadAtLeast({0x1a61d78, 0x37fd440}, {0x34da024, 0x9, 0x9}, 0x9)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/io/io.go:335 +0x90 fp=0x36e3e50 sp=0x36e3e24 pc=0xd8560
headscale | io.ReadFull(...)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/io/io.go:354
headscale | net/http.http2readFrameHeader({0x34da024, 0x9, 0x9}, {0x1a61d78, 0x37fd440})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:1642 +0x54 fp=0x36e3e78 sp=0x36e3e50 pc=0x455bb4
headscale | net/http.(*http2Framer).ReadFrame(0x34da000)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:1909 +0x88 fp=0x36e3ef4 sp=0x36e3e78 pc=0x45634c
headscale | net/http.(*http2clientConnReadLoop).run(0x36e3fdc)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:9496 +0x10c fp=0x36e3fa4 sp=0x36e3ef4 pc=0x47ae4c
headscale | net/http.(*http2ClientConn).readLoop(0x3638008)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:9392 +0x80 fp=0x36e3fe4 sp=0x36e3fa4 pc=0x47a3dc
headscale | net/http.(*http2Transport).newClientConn.gowrap1()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:8006 +0x28 fp=0x36e3fec sp=0x36e3fe4 pc=0x47334c
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x36e3fec sp=0x36e3fec pc=0x9a508
headscale | created by net/http.(*http2Transport).newClientConn in goroutine 16
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/http/h2_bundle.go:8006 +0xd3c
headscale |
headscale | goroutine 35 gp=0x373b568 m=nil [GC worker (idle)]:
headscale | runtime.gopark(0x183e200, 0x3442948, 0x1a, 0xa, 0x0)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345e788 sp=0x345e774 pc=0x92a6c
headscale | runtime.gcBgMarkWorker(0x37e4e40)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xf4 fp=0x345e7e4 sp=0x345e788 pc=0x2dbec
headscale | runtime.gcBgMarkStartWorkers.gowrap1()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x28 fp=0x345e7ec sp=0x345e7e4 pc=0x2dacc
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345e7ec sp=0x345e7ec pc=0x9a508
headscale | created by runtime.gcBgMarkStartWorkers in goroutine 33
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x14c
headscale |
headscale | goroutine 53 gp=0x373bb08 m=nil [select]:
headscale | runtime.gopark(0x183e378, 0x0, 0x9, 0x3, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0x104 fp=0x345a6f4 sp=0x345a6e0 pc=0x92a6c
headscale | runtime.selectgo(0x345a7cc, 0x345a7c4, 0x0, 0x0, 0x2, 0x1)
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/select.go:335 +0xb68 fp=0x345a7a0 sp=0x345a6f4 pc=0x6a574
headscale | database/sql.(*DB).connectionOpener(0x3508dc8, {0x1a6df98, 0x3718360})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:1253 +0x9c fp=0x345a7dc sp=0x345a7a0 pc=0x735300
headscale | database/sql.OpenDB.gowrap1()
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:833 +0x38 fp=0x345a7ec sp=0x345a7dc pc=0x7335ac
headscale | runtime.goexit({})
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_arm.s:884 +0x4 fp=0x345a7ec sp=0x345a7ec pc=0x9a508
headscale | created by database/sql.OpenDB in goroutine 1
headscale | /nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/database/sql/sql.go:833 +0x13c
headscale exited with code 0

Expected Behavior

Opening the SQLite db does not crash the startup process of headscale on armv7.

Steps To Reproduce

  1. Run headscale in docker container using the headscale/headscale image for linux/arm/v7 on an armv7 board
  2. Start the container
  3. headscale starts, tries to open the SQLite db and fails
  4. See error

Environment

- OS: Alpine Linux 3.20
- Headscale version: 0.23
- Tailscale version: xx

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Anything else?

I built the docker image using debian:slim as a base; it fails with the exact same error.

@m-kloeckner m-kloeckner added the bug Something isn't working label Oct 11, 2024
@maxlysenko
Copy link

Hello,
Ran into the same issue on a ARMv7 board (OrangePi).
Found a statement about fragile modernc.org/libc dependency in the documentation of modernc.org/sqlite.
Tried changing the versions in the go.mod file and managed to build working executable.
The exact steps were:

  1. git clone https://github.com/juanfont/headscale.git
  2. cd headscale
  3. git checkout v0.23.0
  4. Modified go.mod file
--- go.mod.old
+++ go.mod.new
@@ -203,8 +203,8 @@
        google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
        gopkg.in/yaml.v2 v2.4.0 // indirect
        gvisor.dev/gvisor v0.0.0-20240722211153-64c016c92987 // indirect
-       modernc.org/libc v1.60.1 // indirect
+       modernc.org/libc v1.55.3 // indirect
        modernc.org/mathutil v1.6.0 // indirect
        modernc.org/memory v1.8.0 // indirect
-       modernc.org/sqlite v1.32.0 // indirect
+       modernc.org/sqlite v1.34.1 // indirect
 )
  1. go mod tidy
  2. GOOS="linux" GOARCH="arm" GOARM="7" go build -v -ldflags="-s -w -X github.com/juanfont/headscale/cmd/headscale/cli.Version=v0.23.0" github.com/juanfont/headscale/cmd/headscale

@jumper047
Copy link

@maxlysenko thank you for your fix, has same issue on my rpi 1b (armv6). Your patch helps me too.

@kradalby
Copy link
Collaborator

kradalby commented Dec 2, 2024

I wonder if this would trigger when doing armv6 builds on other machinery, if that is the case, the github action for just running the build could just compile for all ARCHes so we get feedback when something like that breaks.

If someone is up for it, I'll accept a PR doing:

  • Keep the latest sqlite package, but revert libc to one that works for all ARCHes
  • changes the CI job to build all CPU arches so we detect this in the future

@kradalby kradalby added help wanted Extra attention is needed good first issue Good for newcomers labels Dec 2, 2024
@kradalby
Copy link
Collaborator

kradalby commented Dec 5, 2024

I've set up the build job in #2270, they all currently build, which means that the dependency might be in sync on master?

I dont have a Arm6 board at hand, but I've added so all the binaries are added to the build job, can someone have a look at trying the relevant binary from https://github.com/juanfont/headscale/actions/runs/12160775450?pr=2270 to see if they start on the relevant board?

Please dont do in production, running the CLI or something should be fine.

@maxlysenko
Copy link

maxlysenko commented Dec 5, 2024

Hello,
Tested arm7 build from https://github.com/juanfont/headscale/actions/runs/12160775450?pr=2270, nothing changed, same crash when opening database.
The problem is probably not in the build process or cross-compilation, but in the versions of dependencies in the go.mod file.
Most likely nothing will change until you change version of modernc.org/libc to corresponding with modernc.org/sqlite.
According to description in modernc.org/sqlite docs you shoud use in your go.mod the exact same version of modernc.org/libc as in the go.mod file of the modernc.org/sqlite itself.
So if in your go.mod file you have modernc.org/sqlite v1.32.0 then you should use modernc.org/libc v1.55.3, because go.mod file for v1.32.0 of modernc.org/sqlite contains this version.
Also it seems from commit cc087474 that from v1.33.0 modernc.org/sqlite uses internal/libc, so bumping version of modernc.org/sqlite to v1.33.0 or higher also may help.
Upgrading only modernc.org/sqlite version did't work for me. Tried versions from v1.33.0 to v1.34.1.
Out of interest tried

git clone https://github.com/juanfont/headscale
cd headscale
git checkout v0.23.0
mv go.mod go.mod.old
go mod init github.com/juanfont/headscale
go mod tidy

to see which versions of modules go mod will pull automatically. The versions are

modernc.org/libc v1.55.3 // indirect
modernc.org/sqlite v1.33.1 // indirect

So for now my workaround is:

git clone https://github.com/juanfont/headscale
cd headscale
git checkout v0.23.0
go get modernc.org/[email protected] modernc.org/[email protected]
go mod tidy
go build -v -ldflags="-s -w -X github.com/juanfont/headscale/cmd/headscale/cli.Version=v0.23.0" github.com/juanfont/headscale/cmd/headscale

@kradalby
Copy link
Collaborator

kradalby commented Dec 6, 2024

hmm, I need a way to detect this at build time, we dont have any machinery available to ensure we dont have this regression at runtime. Open for proposals.

@maxlysenko
Copy link

modernc.org/libc was upgraded to v1.60.1 in go.mod in #2098.
Why it was done, why this version was chosen?
As far as I understand (looking at the go mod graph and go mod why output) none of the imported packages/modules requires this version.
So just downgrading modernc.org/libc to v1.55.3 in go.mod should fix the problem.
Unfortunatly I do not know how to detect runtime problem at build time.

@kradalby
Copy link
Collaborator

kradalby commented Dec 6, 2024

In general they are just updated to latest, there is no other rational, so I am happy to have them adjusted so it works again, what I am concerned about is me/someone else upgrading it without ensuring the lockstep.

Typically I just run a go get -u to upgrade all. If we dont have a test/ci failing, it will be hard to detect/prevent this.

@kradalby
Copy link
Collaborator

kradalby commented Dec 6, 2024

I've updated my PR with the dependency version you mention, but I am hesitant calling it a "fix" as it doesnt protect for the future.

@maxlysenko
Copy link

maxlysenko commented Dec 6, 2024

Tested arm builds from https://github.com/juanfont/headscale/actions/runs/12199297972, they don't crash on database access, they start and work.

@kradalby
Copy link
Collaborator

kradalby commented Dec 9, 2024

Great, the best measure I think I can do for now to ensure we keep this in lock is documentation, I have added a section to our go.mod file in f1b97dd (#2270)

@nblock nblock added this to the v0.24.0 milestone Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants