improve: Enhance error handling in trace module

[GarmashAlex]

Are there existing issue(s) that this PR would close?

No

Describe your changes.

• Replace panic-inducing expect() calls with proper error propagation
• Add Result return types to functions that could fail
• Fix debug logging placement for unsupported instructions
• Improve code structure and error handling consistency
• Add InvalidMemoryState error variant for better error reporting

These changes make the code more robust and maintainable by properly
handling potential errors instead of panicking, while also fixing the
placement of debug logging to ensure it works as intended.

[sjudson]

Thanks for the PR.

Right now the Jolt integration is in an experimental state, and the API will likely be significantly redesigned as we look to more robustly integrate it as a first-class proving backend within our codebase. For the moment then we'll likely not accept a PR updating it, but that may change as we make future improvements so I'll leave this PR open accordingly.