Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Health Checks] Provision RPC module before P2P module #972

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

okdas
Copy link
Member

@okdas okdas commented Aug 8, 2023

Description

Currently, the P2P module fails to connect to other nodes because they are marked as unhealthy. In Kubernetes, by default, when a Pod is marked as unhealthy, it won't be resolved or discovered via DNS. As a result, when the network is provisioned from scratch and all nodes start at the same time, none of them can become healthy.

Provisioning the RPC module ahead of P2P makes health checks pass, allowing P2P to discover other nodes.

Summary generated by Reviewpad on 08 Aug 23 00:51 UTC

This pull request includes two patches.

Patch 1/3 disables health checks for the localnet build. It modifies the Tiltfile and statefulset.yaml files to set the "healthchecks.enabled" property to false. This change will disable liveness and readiness probes for the Pocket application running in the localnet environment.

Patch 2/3 modifies the Tiltfile and node.go files to provision the RPC module before the P2P module, allowing healthchecks to pass. This change rearranges the order in which the different modules of the Pocket application start, ensuring that the RPC module starts before the P2P module.

Overall, these patches aim to improve the development and testing experience by managing the health checks in the localnet environment and ensuring the correct order of module startup.

Issue

This has been reported on discord a few times.

Type of change

Please mark the relevant option(s):

  • New feature, functionality or library
  • Bug fix
  • Code health or cleanup
  • Major breaking change
  • Documentation
  • Other

List of changes

  • Changed a sequence of module provisioning.
  • Added a parameter to the helm chart that allows to turn off the health check on Kubernetes level.

Testing

  • make develop_test; if any code changes were made
  • make test_e2e on k8s LocalNet; if any code changes were made
  • e2e-devnet-test passes tests on DevNet; if any code was changed
  • Docker Compose LocalNet; if any major functionality was changed or introduced
  • k8s LocalNet; if any infrastructure or configuration changes were made

Required Checklist

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas (README(s), docs, godoc comments, etc...)
  • I have tested my changes using the available tooling
  • I have added, or updated, godoc format comments on touched members (see: tip.golang.org/doc/comment)
  • I have added, or updated, mermaid.js diagrams in the corresponding README(s)

@okdas okdas added the infra Core infrastructure - not protocol related label Aug 8, 2023
@okdas okdas self-assigned this Aug 8, 2023
@reviewpad reviewpad bot added the small Pull request is small label Aug 8, 2023
@okdas okdas requested a review from bryanchriswhite August 8, 2023 00:50
@okdas okdas assigned okdas and unassigned okdas Aug 8, 2023
@okdas okdas added the e2e-devnet-test Runs E2E tests on devnet label Aug 8, 2023
@okdas okdas marked this pull request as ready for review August 8, 2023 02:25
@okdas okdas requested a review from 0xRampey August 8, 2023 02:27
@0xRampey
Copy link

0xRampey commented Aug 8, 2023

P2P module running successfully now. Thanks!

Copy link
Member

@Olshansk Olshansk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small comment but lgtm otherwise

@@ -83,19 +83,19 @@ func (node *Node) Start() error {
return err
}

if err := node.GetBus().GetP2PModule().Start(); err != nil {
if err := node.GetBus().GetRPCModule().Start(); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment that the order of the Starts is important and should not be modified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
e2e-devnet-test Runs E2E tests on devnet infra Core infrastructure - not protocol related small Pull request is small waiting-for-review
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

3 participants