Fix: Fallback to RPC in more cases when reconcile stream errors

[sanjayprabhu]

Why is this change needed?

In cases like this, we're not falling back to the RPC

{"jobName":"reconcile","jobId":"9273216","reason":"Unable to get all casts for FID 652934: server timeout","errorName":"Error","errorMessage":"Unable to get all casts for FID 652934: server timeout","errorStack":"Error: Unable to get all casts for FID 652934: server timeout\n    at
MessageReconciliation.getAllCastMessagesByFidInBatchesOf (file:///app/node_modules/@farcaster/shuttle/dist/index.mjs:1060:15)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    at process.processTimers (node:internal/timers:511:9)\n    at async MessageReconciliation.allHubMessagesOfTypeForFid (file:///app/node_modules/@farcaster/shuttle/dist/index.mjs:980:22)\n    at async MessageReconciliation.reconcileMessagesOfTypeForFid (file:///app/node_modules/@farcaster/shuttle/dist/index.mjs:918:22)\n    at async MessageReconciliation.reconcileMessagesForFid (file:///app/node_modules/@farcaster/shuttle/dist/index.mjs:913:7)\n    at App.reconcileFids (/app/src/shuttle.ts:132:13)\n    at Worker.Worker.autorun (/app/src/worker.ts:21:9)\n    at Worker.processJob (/app/node_modules/bullmq/src/classes/worker.ts:776:22)\n    at Worker.retryIfFailed (/app/node_modules/bullmq/src/classes/worker.ts:982:16)"}

Merge Checklist

Choose all relevant options below by adding an x now or at any time before submitting for review

• PR title adheres to the conventional commits standard
• PR has a changeset
• PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore)
• PR includes documentation if necessary.

---

PR-Codex overview

This PR focuses on improving error handling in the reconcile process by enhancing the fallback mechanism to RPC in more scenarios when stream errors occur, and temporarily skipping a test for unresponsive server requests.

Detailed summary

• Updated the shuttle.integration.test.ts file to skip the test for unresponsive server requests.
• Enhanced the messageReconciliation.ts file to log a warning when a stream fetch times out and fall back to RPC.
• Added a cancellation mechanism to prevent hanging connections.

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

[changeset-bot]

🦋 Changeset detected

Latest commit: 16cb8e3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@farcaster/shuttle	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
hub-monorepo	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 7, 2024 7:50pm

[codecov]

Codecov Report

Attention: Patch coverage is 12.50000% with 7 lines in your changes missing coverage. Please review.

Project coverage is 44.15%. Comparing base (1880070) to head (16cb8e3).
Report is 420 commits behind head on main.

Files with missing lines	Patch %	Lines
...kages/shuttle/src/shuttle/messageReconciliation.ts	12.50%	7 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2396       +/-   ##
===========================================
- Coverage   74.05%   44.15%   -29.91%     
===========================================
  Files          99      146       +47     
  Lines        9425    26287    +16862     
  Branches     2097     9334     +7237     
===========================================
+ Hits         6980    11606     +4626     
- Misses       2327    12846    +10519     
- Partials      118     1835     +1717     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.