Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate unfrequent relayer submission failures #232

Open
Hyodar opened this issue Jun 5, 2024 · 1 comment
Open

Investigate unfrequent relayer submission failures #232

Hyodar opened this issue Jun 5, 2024 · 1 comment

Comments

@Hyodar
Copy link
Contributor

Hyodar commented Jun 5, 2024

The relayer sometimes can't submit blocks, which results in state root updates with no NEAR DA data. This is not common at all, but should be explored anyway (first by metrics, if this is done after #225, then by execution logs) to see if there's anything we can do to improve that - increasing retries, having a more reliable NEAR RPC...

An easy way to catch these periods is by isolating near_da: false submissions by operators in the aggregator grafana dashboards.

@Hyodar
Copy link
Contributor Author

Hyodar commented Aug 9, 2024

Some additional log data on when something of the sort happened:

2024-08-08T08:05:44.885Z        ERROR   relayer/relayer.go:131  Error submitting blocks to NEAR, resubmitting   {"err": "NEAR DA client handler error: [Timeout]"}
2024-08-08T08:05:44.886Z        ERROR   relayer/relayer.go:109  Error submitting encoded blocks {"err": "unknown error while submitting block to NEAR"}

Most likely we need to enable retries for timeout as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant