feat: PRT - Degrade availability on node errors #1917
18 fail, 2 314 pass in 26m 4s
Annotations
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestConsumerStateMachineHappyFlow (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestConsumerStateMachineHappyFlow/happy (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Jan 20 12:29:07 INF setting log level loglevel=debug
Jan 20 12:29:07 DBG batched nodeUrls by services batch="map[|websocket|:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[ws://127.0.0.1:40231, addons: (websocket), internal-path: ]} ||:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[http://127.0.0.1:36819]}]"
Jan 20 12:29:07 ERR URL scheme should be (http/https), got: ws apiInterface=rest url=ws://127.0.0.1:40231
Jan 20 12:29:07 DBG router keys chainProxyRouter="map[|websocket|:[{ChainProxy:0xc001119560 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc001119320 addonsSupported:map[:{}] methodsRouted:map[]}]]"
Jan 20 12:29:07 DBG chainRouter created chainProxyRouter="map[|websocket|:[{ChainProxy:0xc001119560 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc001119320 addonsSupported:map[:{}] methodsRouted:map[]}]]"
Jan 20 12:29:07 DBG could not send relay to provider error=bad GUID=0 provider=lava@test
Jan 20 12:29:07 INF Relay received a node error Error=bad Request=/cosmos/base/tendermint/v1beta1/blocks/{height} provider="{ProviderAddress:lava2@test ProviderReputationSummery:<nil> ProviderStake:<nil>}"
Jan 20 12:29:07 INF Relay received a node error Error=bad Request=/cosmos/base/tendermint/v1beta1/blocks/{height} provider="{ProviderAddress:lava2@test ProviderReputationSummery:<nil> ProviderStake:<nil>}"
consumer_relay_state_machine_test.go:169:
Error Trace: /home/runner/work/lava/lava/protocol/rpcconsumer/consumer_relay_state_machine_test.go:169
Error: Not equal:
expected: "{}"
actual : "ok"
Diff:
--- Expected
+++ Actual
@@ -1 +1 @@
-{}
+ok
Test: TestConsumerStateMachineHappyFlow/happy
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestRelayProcessorHappyFlow (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestRelayProcessorHappyFlow/happy (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Jan 20 12:29:07 INF setting log level loglevel=debug
Jan 20 12:29:07 DBG batched nodeUrls by services batch="map[|websocket|:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[ws://127.0.0.1:36111, addons: (websocket), internal-path: ]} ||:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[http://127.0.0.1:33199]}]"
Jan 20 12:29:07 ERR URL scheme should be (http/https), got: ws apiInterface=rest url=ws://127.0.0.1:36111
Jan 20 12:29:07 DBG router keys chainProxyRouter="map[|websocket|:[{ChainProxy:0xc001513b00 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc0015138c0 addonsSupported:map[:{}] methodsRouted:map[]}]]"
Jan 20 12:29:07 DBG chainRouter created chainProxyRouter="map[|websocket|:[{ChainProxy:0xc001513b00 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc0015138c0 addonsSupported:map[:{}] methodsRouted:map[]}]]"
relay_processor_test.go:209:
Error Trace: /home/runner/work/lava/lava/protocol/rpcconsumer/relay_processor_test.go:209
Error: Not equal:
expected: "{}"
actual : "ok"
Diff:
--- Expected
+++ Actual
@@ -1 +1 @@
-{}
+ok
Test: TestRelayProcessorHappyFlow/happy
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestRelayProcessorTimeout (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestRelayProcessorTimeout/timeout (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Jan 20 12:29:08 INF setting log level loglevel=debug
Jan 20 12:29:08 DBG batched nodeUrls by services batch="map[|websocket|:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[ws://127.0.0.1:33947, addons: (websocket), internal-path: ]} ||:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[http://127.0.0.1:39477]}]"
Jan 20 12:29:08 ERR URL scheme should be (http/https), got: ws apiInterface=rest url=ws://127.0.0.1:33947
Jan 20 12:29:08 DBG router keys chainProxyRouter="map[|websocket|:[{ChainProxy:0xc000922240 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc000aa6240 addonsSupported:map[:{}] methodsRouted:map[]}]]"
Jan 20 12:29:08 DBG chainRouter created chainProxyRouter="map[|websocket|:[{ChainProxy:0xc000922240 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc000aa6240 addonsSupported:map[:{}] methodsRouted:map[]}]]"
relay_processor_test.go:487:
Error Trace: /home/runner/work/lava/lava/protocol/rpcconsumer/relay_processor_test.go:487
Error: Not equal:
expected: "{}"
actual : "ok"
Diff:
--- Expected
+++ Actual
@@ -1 +1 @@
-{}
+ok
Test: TestRelayProcessorTimeout/timeout
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestRelayProcessorRetry (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestRelayProcessorRetry/retry (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Jan 20 12:29:08 INF setting log level loglevel=debug
Jan 20 12:29:08 DBG batched nodeUrls by services batch="map[|websocket|:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[ws://127.0.0.1:39825, addons: (websocket), internal-path: ]} ||:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[http://127.0.0.1:41343]}]"
Jan 20 12:29:08 ERR URL scheme should be (http/https), got: ws apiInterface=rest url=ws://127.0.0.1:39825
Jan 20 12:29:08 DBG router keys chainProxyRouter="map[|websocket|:[{ChainProxy:0xc001119d40 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc001119e60 addonsSupported:map[:{}] methodsRouted:map[]}]]"
Jan 20 12:29:08 DBG chainRouter created chainProxyRouter="map[|websocket|:[{ChainProxy:0xc001119d40 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc001119e60 addonsSupported:map[:{}] methodsRouted:map[]}]]"
Jan 20 12:29:08 DBG could not send relay to provider error=bad GUID=0 provider=lava@test
relay_processor_test.go:541:
Error Trace: /home/runner/work/lava/lava/protocol/rpcconsumer/relay_processor_test.go:541
Error: Not equal:
expected: "{}"
actual : "ok"
Diff:
--- Expected
+++ Actual
@@ -1 +1 @@
-{}
+ok
Test: TestRelayProcessorRetry/retry
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestRelayProcessorRetryNodeError (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestRelayProcessorRetryNodeError/retry (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Jan 20 12:29:08 INF setting log level loglevel=debug
Jan 20 12:29:08 DBG batched nodeUrls by services batch="map[|websocket|:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[ws://127.0.0.1:44135, addons: (websocket), internal-path: ]} ||:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[http://127.0.0.1:44883]}]"
Jan 20 12:29:08 ERR URL scheme should be (http/https), got: ws apiInterface=rest url=ws://127.0.0.1:44135
Jan 20 12:29:08 DBG router keys chainProxyRouter="map[|websocket|:[{ChainProxy:0xc000e70a20 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc000e707e0 addonsSupported:map[:{}] methodsRouted:map[]}]]"
Jan 20 12:29:08 DBG chainRouter created chainProxyRouter="map[|websocket|:[{ChainProxy:0xc000e70a20 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc000e707e0 addonsSupported:map[:{}] methodsRouted:map[]}]]"
Jan 20 12:29:08 DBG could not send relay to provider error=bad GUID=0 provider=lava@test
Jan 20 12:29:08 INF Relay received a node error Error=bad Request=/cosmos/base/tendermint/v1beta1/blocks/{height} provider="{ProviderAddress:lava@test2 ProviderReputationSummery:<nil> ProviderStake:<nil>}"
relay_processor_test.go:595:
Error Trace: /home/runner/work/lava/lava/protocol/rpcconsumer/relay_processor_test.go:595
Error: Not equal:
expected: "{\"message\": \"bad\",\"code\":123}"
actual : "{\"message\":\"bad\",\"code\":123}"
Diff:
--- Expected
+++ Actual
@@ -1 +1 @@
-{"message": "bad","code":123}
+{"message":"bad","code":123}
Test: TestRelayProcessorRetryNodeError/retry
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestRelayProcessorStatefulApi (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestRelayProcessorStatefulApi/stateful (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Jan 20 12:29:08 INF setting log level loglevel=debug
Jan 20 12:29:08 DBG batched nodeUrls by services batch="map[|websocket|:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[ws://127.0.0.1:44743, addons: (websocket), internal-path: ]} ||:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[http://127.0.0.1:42475]}]"
Jan 20 12:29:08 ERR URL scheme should be (http/https), got: ws apiInterface=rest url=ws://127.0.0.1:44743
Jan 20 12:29:08 DBG router keys chainProxyRouter="map[|websocket|:[{ChainProxy:0xc0001ae000 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc000f939e0 addonsSupported:map[:{}] methodsRouted:map[]}]]"
Jan 20 12:29:08 DBG chainRouter created chainProxyRouter="map[|websocket|:[{ChainProxy:0xc0001ae000 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc000f939e0 addonsSupported:map[:{}] methodsRouted:map[]}]]"
Jan 20 12:29:08 DBG could not send relay to provider error=bad GUID=0 provider=lava@test
relay_processor_test.go:659:
Error Trace: /home/runner/work/lava/lava/protocol/rpcconsumer/relay_processor_test.go:659
Error: Not equal:
expected: "{}"
actual : "ok"
Diff:
--- Expected
+++ Actual
@@ -1 +1 @@
-{}
+ok
Test: TestRelayProcessorStatefulApi/stateful
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestRelayProcessorStatefulApiErr (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestRelayProcessorStatefulApiErr/stateful (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Jan 20 12:29:08 INF setting log level loglevel=debug
Jan 20 12:29:08 DBG batched nodeUrls by services batch="map[|websocket|:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[ws://127.0.0.1:38993, addons: (websocket), internal-path: ]} ||:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[http://127.0.0.1:39157]}]"
Jan 20 12:29:08 ERR URL scheme should be (http/https), got: ws apiInterface=rest url=ws://127.0.0.1:38993
Jan 20 12:29:08 DBG router keys chainProxyRouter="map[|websocket|:[{ChainProxy:0xc001097b00 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc0010967e0 addonsSupported:map[:{}] methodsRouted:map[]}]]"
Jan 20 12:29:08 DBG chainRouter created chainProxyRouter="map[|websocket|:[{ChainProxy:0xc001097b00 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc0010967e0 addonsSupported:map[:{}] methodsRouted:map[]}]]"
Jan 20 12:29:08 DBG could not send relay to provider error=bad GUID=0 provider=lava@test
Jan 20 12:29:08 DBG cancelled relay processor total responses=3
Jan 20 12:29:08 DBG cancelled relay processor total responses=0
relay_processor_test.go:714:
Error Trace: /home/runner/work/lava/lava/protocol/rpcconsumer/relay_processor_test.go:714
Error: Not equal:
expected: "{\"message\": \"bad\",\"code\":123}"
actual : "{\"message\":\"bad\",\"code\":123}"
Diff:
--- Expected
+++ Actual
@@ -1 +1 @@
-{"message": "bad","code":123}
+{"message":"bad","code":123}
Test: TestRelayProcessorStatefulApiErr/stateful
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestRelayProcessorLatest (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestRelayProcessorLatest/latest_req (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Jan 20 12:29:08 INF setting log level loglevel=debug
Jan 20 12:29:08 DBG batched nodeUrls by services batch="map[|websocket|:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[ws://127.0.0.1:40893, addons: (websocket), internal-path: ]} ||:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[http://127.0.0.1:40885]}]"
Jan 20 12:29:08 ERR URL scheme should be (http/https), got: ws apiInterface=rest url=ws://127.0.0.1:40893
Jan 20 12:29:08 DBG router keys chainProxyRouter="map[|websocket|:[{ChainProxy:0xc000f93440 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc000a46c60 addonsSupported:map[:{}] methodsRouted:map[]}]]"
Jan 20 12:29:08 DBG chainRouter created chainProxyRouter="map[|websocket|:[{ChainProxy:0xc000f93440 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc000a46c60 addonsSupported:map[:{}] methodsRouted:map[]}]]"
Jan 20 12:29:08 DBG could not send relay to provider error=bad GUID=0 provider=lava@test
relay_processor_test.go:769:
Error Trace: /home/runner/work/lava/lava/protocol/rpcconsumer/relay_processor_test.go:769
Error: Not equal:
expected: "{}"
actual : "ok"
Diff:
--- Expected
+++ Actual
@@ -1 +1 @@
-{}
+ok
Test: TestRelayProcessorLatest/latest_req
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestRelayProcessorNodeErrorToDegradeAvailability (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Check warning on line 0 in github.com/lavanet/lava/v4/protocol/rpcconsumer
github-actions / Test Results
TestRelayProcessorNodeErrorToDegradeAvailability/relay_is_default_api_-_should_not_degrade_availability (github.com/lavanet/lava/v4/protocol/rpcconsumer) failed
artifacts/Test Results (Protocol)/protocol-report.xml [took 0s]
Raw output
Failed
Jan 20 12:29:08 INF setting log level loglevel=debug
Jan 20 12:29:08 DBG batched nodeUrls by services batch="map[|websocket|:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[ws://127.0.0.1:38733, addons: (websocket), internal-path: ]} ||:{NetworkAddress:{Address: KeyPem: CertPem: DisableTLS:false} ChainID:LAV1 ApiInterface:rest Geolocation:1 NodeUrls:[http://127.0.0.1:42319]}]"
Jan 20 12:29:08 ERR URL scheme should be (http/https), got: ws apiInterface=rest url=ws://127.0.0.1:38733
Jan 20 12:29:09 DBG router keys chainProxyRouter="map[|websocket|:[{ChainProxy:0xc001513c20 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc0015139e0 addonsSupported:map[:{}] methodsRouted:map[]}]]"
Jan 20 12:29:09 DBG chainRouter created chainProxyRouter="map[|websocket|:[{ChainProxy:0xc001513c20 addonsSupported:map[:{}] methodsRouted:map[]}] ||:[{ChainProxy:0xc0015139e0 addonsSupported:map[:{}] methodsRouted:map[]}]]"
Jan 20 12:29:09 DBG api not supported apiKey="{Name:fooBar ConnectionType:POST InternalPath:}"
relay_processor_test.go:1186:
Error Trace: /home/runner/work/lava/lava/protocol/rpcconsumer/relay_processor_test.go:1186
Error: Not equal:
expected: 1
actual : 0
Test: TestRelayProcessorNodeErrorToDegradeAvailability/relay_is_default_api_-_should_not_degrade_availability