Targeted Branched Execution and Context Control?

[protoys-webdev]

First I want to say thank you! Cannoli is a great thing to have!
I use it for generating articles from notes. Therefore, I need to generate paragraphs and chapters within a given article architecture. As results vary naturally by the generative nature, I need often need to re-generate certain parts without the whole document.
This leads me to my problem, which probably is caused by my unknowingness:

• I currently do not see a way to re-generate certain branches in an easy way
• I cannot pinpoint it but it seems like the context sometimes spills a bit too wide. Is there a convenient way to control which nodes the purple boxes take as context?

Below I made a little example how I intend to use it. In this scenario it is for creating texts for web out of a given article but it shows the architectural model I have in mind.

Cannoli seems to be more versatile than I am currently able to grasp. Maybe someone knows the solution.

[blindmansion]

Thanks for trying cannoli! Looks like an awesome use case.

For the portion re-writes, there are a few options that come to mind at the moment, let me know if these answer your question:

1. Use a re-write loop

• Check out the example in cannoli college 5.3
• I know you mentioned that you don't want to have to rewrite the whole article, but a strategy that I've found useful is having GPT-4 write just the section to rewrite, and have GPT-3.5 re-write the whole thing very quickly and cheaply.

2. Use a set of field arrows and a formatter node when you have a very specific structure for the article

• Check out field arrows in cannoli college 2.2
• Describe each section, then give the call node a field arrow for each section, and maybe even use a re-write node. Then use a formatter node (3.3) to put it all together.

3. If you'd like to manually edit portions, create a cannoli for editing particular sections using the "SELECTION" variable

• Check out cannoli college 4.5. This is a new addition so make sure you have the latest version and re-download the cannoli college.
• Make a cannoli that takes in the selection and the note as context, and then replaces the selection with the edit.
• Use that cannoli to manually select sections you'd like to be re-written.

As for the context issues in the purple nodes: The content nodes don't make LLM calls, so they don't really have "context" in the same way as the LLM nodes. If you'd like to check what context an LLM node is seeing, use a logging arrow (2.1) to debug.

The main tool for controlling context in call nodes are the ~ and | suffixes on arrow labels. If an arrow label ends with ~, it will override the default for that arrow type to not send any message history along. If it ends with |, it will override the default to send the message history along. (Check out the examples of this in 1.5, 2.2, and 2.3).

If you'd like to see some specific examples of any of these suggestions to clarify, I'd be happy to make some! Just let me know.

[protoys-webdev]

Thanks for your precise hints about

• Cannoli College 5.3: re-write loops
• Cannoli College 2.2: field arrows (in combination with re-write loops)
• Cannoli College 3.3: formatter nodes (in combination with field arrows)
• Cannoli College 4.5: note editor nodes (use "SELECTION" variable, make sure to download current college note)
• Cannoli College 2.1: debug nodes for verbose on context
• Cannoli College 1.5, 2.2. and 2.3: ~ and | suffixes on arrow labels for overriding the default to not sending prior context and sending prior context (note to me: default --> plugin options?)

I fell in love with a maybe a bit crazy idea, so I just made sure here that I understood your input and can work on it later.

I developed my own little architecture style using a potpourri of (customized) Flutter scaffolds, (micro-) use cases, and an entity-relationship model underneath. The idea is to create and test app prototypes that are easy to build in Flutter afterward.

Related ideas:

• generate Flutter code using note editor nodes (College: 4.5) in Obsidian-Flutter templates
• generate junior-ai issues (see https://github.com/maxonflutter/Junior-AI-Flutter-Pair-Programmer)
• integrate one of my GitHub repositories into my obsidian vault
• use Cannoli to simulate user behavior/interaction with the app prototype
• use Cannoli to simulate database reads and writes in the ERM section (maybe in combination with the Obsidian data view plugin and Cannoli note editor nodes).

While the grand vision might be unachievable for me (at least for now), I am hooked on the idea and cannot help myself.

In case you are interested, attached is the foundational blueprint (without Cannoli logic so far, it will be added inductively) and the first two layers of what I am currently creating. I will continue with the ERM and then look at integrating Cannoli into the graph (it certainly will need more spacing).

I also share this idea with you because I am a bit concerned if I won't be stopped by performance issues, either by the Obsidian Canvas or by Cannoli. What do you think?

[blindmansion]

Wow you're cooking up something very interesting for sure!

Some issues I think you might come across trying to build this with current cannoli capability:

1. I'm not super familiar with using code inside obsidian, but if a code file isn't considered a "note" by obsidian, it isn't accessible to cannoli. You may have to add the .md extension to all code files.

2. Cannolis can only interact with stuff in two ways at the moment: http fetch requests and editing notes. If you can figure out a way for cannoli to emulate user actions with those two, I think it's possible you could build an automated tester, but it would definitely be a challenge.

3. There currently isn't a public GPT-4Vision API out yet, so if you wanted to use vision to drive the tester agent that isn't available yet (but I'd love to add it)

4. It isn't ready yet, but I'm currently working on a dataview integration. Likely the extent of the integration will be (1) dataview queries render for the LLM to see, and (2) dataview queries integrated with the forthcoming forEach groups (i.e. performing the same actions for each note in a query result).

If you come across other limitations of cannoli as you build this, please let me know, I'd love to enable it if feasible.

[protoys-webdev]

Thanks for the useful hints again!
With HTTP fetch you mean like POST, DELETE, PUT, or GET as in this?
--> https://hygraph.com/docs/api-reference/basics/webhooks

[blindmansion]

Yep that's right! You could set up an action mode that interacts with these webhooks.

[protoys-webdev]

just fyi, i continued doing my architecture work in a canvas but I had to make a decision to create results quickly... so I changed my approach from as described above to using flutterflow.