Doppelgänger stuck when primary node isn't Teku

[stefa2k]

Description

VC is connected to 5 nodes (in order):

• Lodestar
• Prym
• Lighthouse
• Teku

Teku vc doesn't start any validator duties after performing doppelgänger check successfully.

Changing the order of the nodes, making Teku the primary node, seems to resolve the issue.

Steps to Reproduce (Bug)

Connect a vc to multiple bn, primary should be not Teku.

Expected behavior: Teku vc performs doppelgänger check and successfully starts performing validtor duties.

Actual behavior:

#150: 2023-10-15 08:23:11.718 INFO  - �[32mValidator   *** 500 out of 500 validator registration(s) were successfully sent to the builder network via the Beacon Node.�[0m

#149: 2023-10-15 08:16:48.766 INFO  - �[32mValidator   *** 500 out of 500 validator registration(s) were successfully sent to the builder network via the Beacon Node.�[0m

#148: 2023-10-15 08:10:25.607 INFO  - �[32mValidator   *** 500 out of 500 validator registration(s) were successfully sent to the builder network via the Beacon Node.�[0m

#147: 2023-10-15 08:04:00.273 INFO  - �[32mValidator   *** 500 out of 500 validator registration(s) were successfully sent to the builder network via the Beacon Node.�[0m

#146: 2023-10-15 07:57:37.410 INFO  - �[32mValidator   *** 500 out of 500 validator registration(s) were successfully sent to the builder network via the Beacon Node.�[0m

#145: 2023-10-15 07:57:32.315 INFO  - �[32mValidator   *** Successfully connected to beacon node event stream�[0m

#144: 2023-10-15 07:57:30.102 INFO  - �[32mValidator   *** 500 out of 500 validator registration(s) were successfully sent to the builder network via the Beacon Node.�[0m

#143: 2023-10-15 07:57:23.757 WARN  - Unable to retrieve status for 500 validators.

#142: 2023-10-15 07:57:23.392 INFO  - �[32mValidator   *** Slashing protection loaded for validators: 93f8af3, b9f1529, a454590, 8d98fdf, 97e3a0b, ac5c5fc, 831ac7a, 988665c, 8a185fe, ae11b1a, 87db177, 954d7df, 8f6a726, b081d4b, 96a2fe9, a7615b0, a34c6cf, aa7c7ae, a1af961, 952d573… (500 total)�[0m

#141: 2023-10-15 07:57:22.924 INFO  - No validators doppelganger detected.

#140: 2023-10-15 07:57:22.923 INFO  - Doppelganger detection check finished. Stopping doppelganger detection for public keys 93f8af3, b9f1529, a454590, 8d98fdf, 97e3a0b, ac5c5fc, 831ac7a, 988665c, 8a185fe, ae11b1a, 87db177, 954d7df, 8f6a726, b081d4b, 96a2fe9, a7615b0, a34c6cf, aa7c7ae, a1af961, 952d573, 94eac1c, 99c0adc, 855d450, b62adb1, b4d93c4, af7039e, ae3c60f, a91f39b, 8d6c699, 8792bfa, 84ace35, b83d758, 91d7f56, 946f276, 8f9de2d, 93106a9, 8e0495a, afb9b95, a4e8ec1, a983451, 976e67e, 8edc915, 8e4b351, ab237a4, b8466d6, 974a6a0, a320620, 85e1b84, a4e74d8, ae33b22, a525546, a70a434, a0e053e, a7b885f, 978b4ed, 958511f, b51521b, 85e6981, b27b578, 8bcb984, 82baae2, 8ffb060, b910946, 8f61b0e, 851edaf, b686c64, a547446, 83ccf7d, 8c9a4bc, ad73915, 8acee7b, b0b7c6d, 95519a3, 8fe7a3d, 89f194c, afecc25, a9ddcf6, ace5041, 96d60a3, 993aecb, b2057ce, ae57e37, b400f6d, a4ce847, b5c0907, 8ce930d, a27bca7, 848e8f4, b9d7151, 8dd1776, 81a7a62, 922120a, b550d40, a603384, 854972f, 82ae8ff, b04833e, b95ea52, b146d94, b09f4e2, b91a05d, b21e422, 8fae9d4, 882b3b2, af1d635, 977a0b9, 8afa65f, 996c0ca, 8d5e87f, 88601c5, 886db3e, 8f35ca3, ab33a9b, a1dda57, 98da325, b3238a4, 97f16ab, ad14dd7, 80109e1, a1ab382, aec6821, b2a024a, 97dbdcc, b72252e, b611f69, b6b3a1c, 859766b, 8fc4778, b467c3f, 92bd455, a430327, 90ec2e8, a1198ba, 8c4223d, a9e4065, 8ed3072, aa28a58, 93afcd3, 951fd23, 96185ff, a3b471e, b5785e6, ab3892f, 8694582, 80014e5, 9837774, a3795d1, 8170f02, a0416f3, 91eda48, b80f1a3, 99c77cf, 8b53693, b126fb2, b6a2452, a9d4250, b5b03fb, 85a3e06, af8d538, a8857b0, 86a197f, 97ba958, 8cef71a, 951ed4f, ab86ba0, b0c8e8d, b13dc63, 8c56108, 82ca91a, 8989832, a7c0d05, 98d5a5a, 828d85f, 9510e16, b92373a, b47d2f7, aeef9a6, a406eed, a524056, 8f14d37, a392b58, a9a569f, b285df8, a1528ac, 879656d, 8a19ad7, a0293c9, ae9665e, 9947272, b43fb7d, 97a3f53, 944ab52, 9780919, aa968ee, 8437e67, 8fd76e1, a02a745, a1c70ce, 90ea23a, 8031211, 8230f1d, a035f79, 86e62d8, b5a02e4, 998455f, 843a379, b9a554a, aabf6ad, b4e47d1, 954c3d5, 88c4ffa, 8ea9dab, b151127, 88e271f, 881bac4, a277001, a5a2dd1, a32513b, a081115, 8858450, 9296d38, 8738077, 89bceb7, b488313, 9838724, aa30d54, a98cc2f, a253a5f, b3cd271, 8ac1523, b98736f, 85cc982, 895cec4, 84619b6, 92585de, 8c66c92, 8fd2d02, 9511890, 824d39e, 911d09e, a0aab18, b695a44, 8af499e, b307f03, afcb616, b39554d, ae10caa, 8184fb9, b9260f1, ae969fd, b5b2932, b9c2527, a986b40, 8bebbac, 8f70d06, 91217a4, 8538fc9, 958279f, 85179d8, 85fbffd, a1fc37b, 925b06a, 97e6a8d, b9d60c8, 983e4c0, 835711f, b3b015e, 80805d9, a7a78cc, 81a4661, 8927fc4, a2821c2, 97c7436, 8cd343c, a6199aa, 8f633aa, 812d32f, 96c7bf5, 8c72415, 8f5e745, b73cc51, 83c5929, b3e3eb6, 90c8b12, b541ca4, b914759, 91b3de5, 8ca1f23, a0c3f68, 93c8969, 930ead5, b3dfcb7, b43c06c, 8714c35, aff4687, a9a1148, a23032d, a15d8da, 8cb273f, 91abd47, a855bb3, 9591c3f, 8cf23d7, a85bb8e, 83007b5, b738595, ac8bebf, 8ae0f6b, 8958720, a9edb31, abd7fc5, a307ca2, 8720aa3, 9881978, b01a486, b27da28, 8bc62a9, 9413ddc, a83a8a7, a1060f2, 8de090d, b235b50, a084698, abba168, b2214d6, 8347e99, af054f1, 8ed8043, 928a843, b438a55, 912cee3, a48fca3, 930d63b, 98d6bc3, b16e0b8, 906bf99, b2de4f8, b75efdf, a55817c, a06a932, b2abd60, 81f7d8d, b521043, 8c63630, a226df5, b2ec745, b9a0bf3, aec3ff3, 89edfb8, a94cfe5, a835286, a486443, 939be84, b6c4949, aa44aa3, 868c376, 8eef9b1, 9847803, b3958f2, b6c9b4c, 84f7200, 9683c42, 88f9d1d, 89504a0, 97dbc15, b771bab, b9d8bf9, 858ea04, 9234071, 86219f5, a4523e0, 94e4935, 8a8d110, 91bdc44, 94a90f6, a180aff, b38a8e2, 86ce7f3, 80a6b79, a7b19be, 8ee4644, 8a729d3, b3b400c, aeda41b, b51e89d, b169d5b, 9497132, ae6bf71, 8eb56e3, 99558ad, b21314c, ae0f992, b3dc9ac, a006092, 8229905, 84529d0, 84ceaed, 92d6d5f, a5c1273, a3cd543, a45a9eb, b902c32, 9727434, 8af8110, aac3cb0, 968527e, 848b372, 8a5e0a9, b1b49b8, 95128e3, 95c9593, a4212f9, b66a8d6, 8eed5d0, b192b25, 98fad5f, 9190efe, b1ceace, 95e09a7, 8bf0e58, a066728, 979719e, b4bef20, aa14f7f, a2bbb8f, 8d9bc86, afabb3f, 9445343, 88f5c7d, 9829120, 8fe5257, 8967c83, 85c53b1, b4332c1, 8fe2ff7, 8dbde03, 835a6fa, 8b7c4a7, a99bd16, 82d419f, b2e8eee, 8f3903a, a5cd2a4, a89a5df, b942851, 828dfa6, a6abf97, a41218c, 8867610, 9349130, 896e61d, 85ac21b, 910898f, a7d9cc3, b3514b1, 98de17b, 901b805, 91febce, a47241b, 97b4b7c, 84b106c, 9360057, a067bec, b249d91, 89281d0, b9c157a, a46c298, a054874, abcd082, 92f0d70, a666da8, abad174, 88e6407, 80faaab, 96efde6, 981852c, 97a775c, ad17314, a622195, 85ffa72, a9599e7, b663def, 846e812, a686de7, 882afa9, 842b4f9, b1b2ee3, afe6ab5, 869bd83, 96701a0, 8e904e3, a6a3fcb, 89c3da1, a4f0a92, 80ec6f9, 910c783, b3afb93, 870c3c3, aaa669f, ac997aa.

#139: 2023-10-15 07:57:10.921 INFO  - Skipping validators doppelgangers check for public keys 8a8d110, b51521b, b80f1a3, afcb616, b663def, 8fe7a3d, 895cec4, 8ffb060, 9838724, 977a0b9, ab33a9b, b1b2ee3, 8858450, b98736f, a4ce847, 93106a9, a253a5f, a430327, ae10caa, 8dd1776, 8792bfa, 881bac4, 8f5e745, a547446, a622195, 8eef9b1, 946f276, abad174, a9d4250, a8857b0, b13dc63, ae33b22, a7b19be, 8edc915, a054874, 8a729d3, 80a6b79, b4e47d1, 8989832, b9d7151, 910c783, b2057ce, a6a3fcb, a6abf97, 8d98fdf, 8927fc4, 8eed5d0, afe6ab5, 954c3d5, 859766b, b910946, 85c53b1, 99c77cf, 80805d9, 930d63b, 8c66c92, 8af8110, a067bec, b169d5b, b75efdf, a1ab382, 8fe5257, 98da325, a4523e0, 952d573, a835286, 8bf0e58, 8f9de2d, af1d635, 848e8f4, a4f0a92, a686de7, 855d450, 851edaf, 98d6bc3, afecc25, a85bb8e, 88e6407, ae9665e, 8fc4778, 80faaab, b3e3eb6, 8bebbac, aeef9a6, abcd082, b9f1529, 8a185fe, ae3c60f, 998455f, 86a197f, 9780919, a32513b, b3958f2, b95ea52, 8f3903a, 88c4ffa, 958511f, 8967c83, 828dfa6, 96efde6, 968527e, a7c0d05, b285df8, 8867610, 951ed4f, b47d2f7, a0e053e, ab86ba0, 835a6fa, 97ba958, 8bcb984, 96d60a3, 842b4f9, abba168, ad17314, a1af961, 81a4661, 8bc62a9, b91a05d, 8c9a4bc, 85fbffd, a55817c, b146d94, b3238a4, b51e89d, 812d32f, a48fca3, aec3ff3, 85ac21b, 9683c42, aa968ee, b695a44, aec6821, 97e3a0b, 9497132, afb9b95, 90ec2e8, a9599e7, b6c4949, ab237a4, 896e61d, b4bef20, b21314c, ac5c5fc, 824d39e, 8694582, a666da8, 91abd47, 870c3c3, 86219f5, 89f194c, 82ae8ff, a0416f3, a3b471e, 93afcd3, a0c3f68, 879656d, 843a379, 8ae0f6b, 8ca1f23, 922120a, 92f0d70, 930ead5, 80ec6f9, b4332c1, 85ffa72, b72252e, b550d40, b902c32, 8e0495a, 8b7c4a7, 93f8af3, a9a1148, a4e8ec1, 8538fc9, 8fd2d02, a9ddcf6, b01a486, 95519a3, 8cf23d7, 82d419f, 80109e1, 8d5e87f, a98cc2f, 983e4c0, 82baae2, 828d85f, 8f35ca3, 9349130, 9413ddc, a45a9eb, 91b3de5, a035f79, 93c8969, 89edfb8, a7a78cc, b686c64, afabb3f, 97b4b7c, 869bd83, b126fb2, 831ac7a, aac3cb0, b521043, b16e0b8, b9d8bf9, a307ca2, 8fae9d4, b09f4e2, af8d538, b38a8e2, 81a7a62, b235b50, a603384, b5c0907, 906bf99, 8229905, 9510e16, a2bbb8f, b3514b1, b8466d6, 882b3b2, b5b03fb, b9a0bf3, 976e67e, 8720aa3, 9234071, a320620, b3b400c, a06a932, ac997aa, b5b2932, b9d60c8, 84b106c, ace5041, a91f39b, b307f03, 86ce7f3, 939be84, 8f6a726, ad14dd7, a6199aa, 9881978, 8ce930d, b62adb1, 9190efe, a7b885f, 8c63630, 88601c5, a9e4065, 97dbc15, ab3892f, 9296d38, 8f70d06, ae0f992, a1dda57, a99bd16, 8acee7b, 8d9bc86, a7615b0, b249d91, 911d09e, 89c3da1, 974a6a0, 8031211, 912cee3, aa30d54, aff4687, 8cb273f, b2214d6, b2a024a, 9947272, b6a2452, 97dbdcc, 988665c, b1ceace, 928a843, 85cc982, 94eac1c, 84619b6, 85a3e06, 8738077, 846e812, a1060f2, a02a745, 993aecb, b914759, 9829120, a006092, 97c7436, 8e904e3, 981852c, b3dfcb7, b0b7c6d, b3b015e, 8437e67, 854972f, b0c8e8d, 8ed3072, a23032d, 95e09a7, 84529d0, b611f69, 89504a0, 85e6981, 886db3e, 96a2fe9, 8afa65f, a983451, 97f16ab, 8dbde03, 8e4b351, a406eed, b21e422, ad73915, b192b25, b2de4f8, a525546, 9727434, ae969fd, 8ee4644, b6b3a1c, aaa669f, b73cc51, 978b4ed, b43c06c, 83c5929, 98fad5f, 89bceb7, a226df5, a454590, 979719e, b83d758, 8f14d37, 8ac1523, 954d7df, 83ccf7d, 9445343, 98de17b, 98d5a5a, 9511890, a34c6cf, 8958720, 91eda48, ae57e37, 910898f, 86e62d8, a524056, 848b372, a486443, b2abd60, 958279f, b9c2527, 85e1b84, a27bca7, b04833e, b738595, a855bb3, a3795d1, b438a55, b9a554a, a5cd2a4, 87db177, 996c0ca, a2821c2, 858ea04, 835711f, 8c4223d, 8b53693, 91d7f56, 91bdc44, a081115, b5a02e4, a41218c, b39554d, a94cfe5, 8c72415, b5785e6, b467c3f, 8af499e, a4e74d8, a1c70ce, a066728, 8cef71a, 89281d0, a3cd543, aa28a58, 8714c35, 91febce, b9c157a, a4212f9, aa7c7ae, b4d93c4, aeda41b, 8c56108, 92585de, a9a569f, 9360057, b92373a, b2e8eee, b488313, 92bd455, a9edb31, a89a5df, 8f61b0e, a7d9cc3, 8ed8043, 96c7bf5, 84ceaed, 8de090d, 8ea9dab, a46c298, b27da28, b43fb7d, 92d6d5f, b081d4b, a47241b, 90c8b12, a277001, 80014e5, b3cd271, 8184fb9, ae11b1a, b771bab, 8230f1d, a392b58, 90ea23a, 97a3f53, 94e4935, 8cd343c, 8f633aa, a1198ba, 82ca91a, b942851, a1528ac, 84f7200, a5a2dd1, 8170f02, b151127, 96185ff, aa14f7f, 88f9d1d, ac8bebf, b3afb93, 8eb56e3, 8a5e0a9, 882afa9, 8d6c699, a15d8da, 99558ad, 95c9593, 9837774, 8fd76e1, 97e6a8d, 85179d8, b27b578, b6c9b4c, 81f7d8d, 84ace35, 94a90f6, b9260f1, af7039e, b66a8d6, 9847803, abd7fc5, 91217a4, a986b40, b400f6d, a5c1273, 95128e3, 8a19ad7, b3dc9ac, a0aab18, ae6bf71, a1fc37b, a70a434, aa44aa3, a83a8a7, 88e271f, af054f1, aabf6ad, 83007b5, a0293c9, 8fe2ff7, 88f5c7d, a180aff, b1b49b8, 951fd23, a084698, 97a775c, b541ca4, 99c0adc, b2ec745, 96701a0, 8347e99, 901b805, 944ab52, 868c376, 925b06a, 9591c3f. No associated indices found. Public keys are inactive

...

#35: 2023-10-15 07:45:49.011 INFO  - Starting doppelganger detection for public keys: 93f8af3, b9f1529, a454590, 8d98fdf, 97e3a0b, ac5c5fc, 831ac7a, 988665c, 8a185fe, ae11b1a, 87db177, 954d7df, 8f6a726, b081d4b, 96a2fe9, a7615b0, a34c6cf, aa7c7ae, a1af961, 952d573, 94eac1c, 99c0adc, 855d450, b62adb1, b4d93c4, af7039e, ae3c60f, a91f39b, 8d6c699, 8792bfa, 84ace35, b83d758, 91d7f56, 946f276, 8f9de2d, 93106a9, 8e0495a, afb9b95, a4e8ec1, a983451, 976e67e, 8edc915, 8e4b351, ab237a4, b8466d6, 974a6a0, a320620, 85e1b84, a4e74d8, ae33b22, a525546, a70a434, a0e053e, a7b885f, 978b4ed, 958511f, b51521b, 85e6981, b27b578, 8bcb984, 82baae2, 8ffb060, b910946, 8f61b0e, 851edaf, b686c64, a547446, 83ccf7d, 8c9a4bc, ad73915, 8acee7b, b0b7c6d, 95519a3, 8fe7a3d, 89f194c, afecc25, a9ddcf6, ace5041, 96d60a3, 993aecb, b2057ce, ae57e37, b400f6d, a4ce847, b5c0907, 8ce930d, a27bca7, 848e8f4, b9d7151, 8dd1776, 81a7a62, 922120a, b550d40, a603384, 854972f, 82ae8ff, b04833e, b95ea52, b146d94, b09f4e2, b91a05d, b21e422, 8fae9d4, 882b3b2, af1d635, 977a0b9, 8afa65f, 996c0ca, 8d5e87f, 88601c5, 886db3e, 8f35ca3, ab33a9b, a1dda57, 98da325, b3238a4, 97f16ab, ad14dd7, 80109e1, a1ab382, aec6821, b2a024a, 97dbdcc, b72252e, b611f69, b6b3a1c, 859766b, 8fc4778, b467c3f, 92bd455, a430327, 90ec2e8, a1198ba, 8c4223d, a9e4065, 8ed3072, aa28a58, 93afcd3, 951fd23, 96185ff, a3b471e, b5785e6, ab3892f, 8694582, 80014e5, 9837774, a3795d1, 8170f02, a0416f3, 91eda48, b80f1a3, 99c77cf, 8b53693, b126fb2, b6a2452, a9d4250, b5b03fb, 85a3e06, af8d538, a8857b0, 86a197f, 97ba958, 8cef71a, 951ed4f, ab86ba0, b0c8e8d, b13dc63, 8c56108, 82ca91a, 8989832, a7c0d05, 98d5a5a, 828d85f, 9510e16, b92373a, b47d2f7, aeef9a6, a406eed, a524056, 8f14d37, a392b58, a9a569f, b285df8, a1528ac, 879656d, 8a19ad7, a0293c9, ae9665e, 9947272, b43fb7d, 97a3f53, 944ab52, 9780919, aa968ee, 8437e67, 8fd76e1, a02a745, a1c70ce, 90ea23a, 8031211, 8230f1d, a035f79, 86e62d8, b5a02e4, 998455f, 843a379, b9a554a, aabf6ad, b4e47d1, 954c3d5, 88c4ffa, 8ea9dab, b151127, 88e271f, 881bac4, a277001, a5a2dd1, a32513b, a081115, 8858450, 9296d38, 8738077, 89bceb7, b488313, 9838724, aa30d54, a98cc2f, a253a5f, b3cd271, 8ac1523, b98736f, 85cc982, 895cec4, 84619b6, 92585de, 8c66c92, 8fd2d02, 9511890, 824d39e, 911d09e, a0aab18, b695a44, 8af499e, b307f03, afcb616, b39554d, ae10caa, 8184fb9, b9260f1, ae969fd, b5b2932, b9c2527, a986b40, 8bebbac, 8f70d06, 91217a4, 8538fc9, 958279f, 85179d8, 85fbffd, a1fc37b, 925b06a, 97e6a8d, b9d60c8, 983e4c0, 835711f, b3b015e, 80805d9, a7a78cc, 81a4661, 8927fc4, a2821c2, 97c7436, 8cd343c, a6199aa, 8f633aa, 812d32f, 96c7bf5, 8c72415, 8f5e745, b73cc51, 83c5929, b3e3eb6, 90c8b12, b541ca4, b914759, 91b3de5, 8ca1f23, a0c3f68, 93c8969, 930ead5, b3dfcb7, b43c06c, 8714c35, aff4687, a9a1148, a23032d, a15d8da, 8cb273f, 91abd47, a855bb3, 9591c3f, 8cf23d7, a85bb8e, 83007b5, b738595, ac8bebf, 8ae0f6b, 8958720, a9edb31, abd7fc5, a307ca2, 8720aa3, 9881978, b01a486, b27da28, 8bc62a9, 9413ddc, a83a8a7, a1060f2, 8de090d, b235b50, a084698, abba168, b2214d6, 8347e99, af054f1, 8ed8043, 928a843, b438a55, 912cee3, a48fca3, 930d63b, 98d6bc3, b16e0b8, 906bf99, b2de4f8, b75efdf, a55817c, a06a932, b2abd60, 81f7d8d, b521043, 8c63630, a226df5, b2ec745, b9a0bf3, aec3ff3, 89edfb8, a94cfe5, a835286, a486443, 939be84, b6c4949, aa44aa3, 868c376, 8eef9b1, 9847803, b3958f2, b6c9b4c, 84f7200, 9683c42, 88f9d1d, 89504a0, 97dbc15, b771bab, b9d8bf9, 858ea04, 9234071, 86219f5, a4523e0, 94e4935, 8a8d110, 91bdc44, 94a90f6, a180aff, b38a8e2, 86ce7f3, 80a6b79, a7b19be, 8ee4644, 8a729d3, b3b400c, aeda41b, b51e89d, b169d5b, 9497132, ae6bf71, 8eb56e3, 99558ad, b21314c, ae0f992, b3dc9ac, a006092, 8229905, 84529d0, 84ceaed, 92d6d5f, a5c1273, a3cd543, a45a9eb, b902c32, 9727434, 8af8110, aac3cb0, 968527e, 848b372, 8a5e0a9, b1b49b8, 95128e3, 95c9593, a4212f9, b66a8d6, 8eed5d0, b192b25, 98fad5f, 9190efe, b1ceace, 95e09a7, 8bf0e58, a066728, 979719e, b4bef20, aa14f7f, a2bbb8f, 8d9bc86, afabb3f, 9445343, 88f5c7d, 9829120, 8fe5257, 8967c83, 85c53b1, b4332c1, 8fe2ff7, 8dbde03, 835a6fa, 8b7c4a7, a99bd16, 82d419f, b2e8eee, 8f3903a, a5cd2a4, a89a5df, b942851, 828dfa6, a6abf97, a41218c, 8867610, 9349130, 896e61d, 85ac21b, 910898f, a7d9cc3, b3514b1, 98de17b, 901b805, 91febce, a47241b, 97b4b7c, 84b106c, 9360057, a067bec, b249d91, 89281d0, b9c157a, a46c298, a054874, abcd082, 92f0d70, a666da8, abad174, 88e6407, 80faaab, 96efde6, 981852c, 97a775c, ad17314, a622195, 85ffa72, a9599e7, b663def, 846e812, a686de7, 882afa9, 842b4f9, b1b2ee3, afe6ab5, 869bd83, 96701a0, 8e904e3, a6a3fcb, 89c3da1, a4f0a92, 80ec6f9, 910c783, b3afb93, 870c3c3, aaa669f, ac997aa

#34: 2023-10-15 07:45:49.002 INFO  - Listening on http://localhost:5052/

#33: 2023-10-15 07:45:48.985 INFO  - Started Server@296481f1{STARTING}[11.0.15,sto=0] @160103ms

#32: 2023-10-15 07:45:48.983 INFO  - Started ServerConnector@5e037b06{SSL, (ssl, http/1.1)}{0.0.0.0:5052}

#31: 2023-10-15 07:45:48.936 INFO  - Unable to get SSLContext instance for protocol [TLS] on provider [Conscrypt], using default

#30: 2023-10-15 07:45:48.935 INFO  - Unable to get TrustManagerFactory instance for algorithm [PKIX] on provider [Conscrypt], using default

#29: 2023-10-15 07:45:48.878 INFO  - Unable to get KeyManagerFactory instance for algorithm [SunX509] on provider [Conscrypt], using default

#28: 2023-10-15 07:45:48.878 INFO  - x509=X509@79c75cb8(mykey,h=[stereum-4b1bc06f-25d2-a98e-f643-47630bb162b8, teku],a=[],w=[]) for Server@776c362c[provider=Conscrypt,keyStore=file:///opt/app/data/teku_api_keystore,trustStore=null]

#27: 2023-10-15 07:45:48.633 INFO  - Started i.j.j.@a58bd23{/,null,AVAILABLE}

#26: 2023-10-15 07:45:48.622 INFO  - Session workerName=node0

#25: 2023-10-15 07:45:48.581 INFO  - jetty-11.0.15; built: 2023-04-11T18:37:53.775Z; git: 5bc5e562c8d05c5862505aebe5cf83a61bdbcb96; jvm 17.0.8.1+1

#24: 2023-10-15 07:45:48.340 INFO  - Loaded 500 validators

Frequency: Pretty often

Versions (Add all that apply)

• Software version: 23.10.0
• Docker Version: 24.0.2

[mehdi-aouadi]

Hi @stefa2k, according to the logs, the DG detection doesn't look stuck:
The #139: 2023-10-15 07:57:10.921 INFO - Skipping validators doppelgangers check for public keys... means that there were no indices allocated to these keys which means that they were not active. This check is made using the eth/v1/beacon/states/head/validators API.
And the Doppelganger detection check finished obviously marks the end of the DG check.
Could you please share more logs after the DG check finishes? What makes you say that the validators do not perform there duties?

[stefa2k]

Could you please share more logs after the DG check finishes?

There are no more logs, there is simply nothing happending in more than 25 minutes between No validators doppelganger detected. and the last message of the log.

What makes you say that the validators do not perform there duties?

I expect logs like this:

2023-10-18 11:05:36.772 INFO  - �[34mValidator   *** Published attestation        Count: 24, Slot: 7566926, Root: 0c5c902c9f4149fd1868dc18244b2ede84c243da0b2a646521a237df0a15ddda�[0m

but there's nothing in the logs provided in the op. The 500 keys were alive and ready to perform duties, after switching the Teku beacon node (previously 4th in the list) to the first in the list Teku vc started without an issue immediately with duties after No validators doppelganger detected..

So I guess the conclusion is that Unable to retrieve status for 500 validators. indicates the Teku validator client wasn't able to see that the keys are in fact active. Does Teku vc only ask one of the provided beacon nodes (I guess the first one)?

[mehdi-aouadi]

The Unable to retrieve status for 500 validators has nothing to do with the doppelganger check and happens afterward. It indicates that the VC wasn't able to get the validator statuses from the eth/v1/beacon/states/head/validators API. It might be the cause of them not performing their duties. Is the Rest API running correctly and reachable on the BN side?

[mehdi-aouadi]

Actually the validators statutes requests when registering the validators do not fallback to the configured failover BNs unfortunately. I guess what's happening here is that the eth/v1/beacon/states/head/validators API didn't respond for some reason (this might be checked in the BN logs) and hence the registration failed.
We recently merged a change to enhance the validator registration but it's not released yet: #7423. If you're interested you can try using this Teku image with the change https://hub.docker.com/layers/consensys/teku/develop/images/sha256-4c9526d0c10870f8d49bbc93e96ac16584f598f15202f9c61a95f4b9748595f4?context=explore

[stefa2k]

Thanks @mehdi-aouadi to look into this, I'll have a closer look and re-evaluate this after the next release including #7423 ! If you don't mind, I'll post an update after that.

[zilm13]

@stefa2k Could you also look into logs of first bn when it's not Teku, Prysm or whatever. If there are any errors in it, it could give us an idea, what's going wrong

[mehdi-aouadi]

I'm closing this issue for now. As discussed, the next release will include some validators registration improvements. Feel free to reopen it if despite of that you still have the same issue