Implement Grbl character-counting protocol

[jekhor]

Implement Grbl character-counting protocol to use grbl command buffering: https://github.com/gnea/grbl/wiki/Grbl-v1.1-Interface#streaming-a-g-code-program-to-grbl
This should be very useful on jobs with big amount of short moves including raster engraving.

[tatarize]

Close #77 but actually implement this. Posting the relevant explanation of this protocol.

Streaming Protocol: Character-Counting [Recommended with Reservation]
To get the best of both worlds, the simplicity and reliability of the send-response method and assurance of maximum performance with software flow control, we came up with a simple character-counting protocol for streaming a G-code program to Grbl. It works like the send-response method, where the host PC sends a line of G-code for Grbl to execute and waits for a response message, but, rather than needing special XON/XOFF characters for flow control, this protocol simply uses Grbl's responses as a way to reliably track how much room there is in Grbl's serial receive buffer. An example of this protocol is outlined in the stream.py streaming script in our repo. This protocol is particular useful for very fast machines like laser cutters.

The main difference between this protocol and the others is the host PC needs to maintain a standing count of how many characters it has sent to Grbl and then subtract the number of characters corresponding to the line executed with each Grbl response. Suppose there is a short G-code program that has 5 lines with 25, 40, 31, 58, and 20 characters (counting the line feed and carriage return characters too). We know Grbl has a 128 character serial receive buffer, and the host PC can send up to 128 characters without overflowing the buffer. If we let the host PC send as many complete lines as we can without over flowing Grbl's serial receive buffer, the first three lines of 25, 40, and 31 characters can be sent for a total of 96 characters. When Grbl sends a response message, we know the first line has been processed and is no longer in the serial read buffer. As it stands, the serial read buffer now has the 40 and 31 character lines in it for a total of 71 characters. The host PC needs to then determine if it's safe to send the next line without overflowing the buffer. With the next line at 58 characters and the serial buffer at 71 for a total of 129 characters, the host PC will need to wait until more room has cleared from the serial buffer. When the next Grbl response message comes in, the second line has been processed and only the third 31 character line remains in the serial buffer. At this point, it's safe to send the remaining last two 58 and 20 character lines of the g-code program for a total of 110.

While seemingly complicated, this character-counting streaming protocol is extremely effective in practice. It always ensures Grbl's serial read buffer is filled, while never overflowing it. It maximizes Grbl's performance by keeping the look-ahead planner buffer full by better utilizing the bi-directional data flow of the serial port, and it's fairly simple to implement as our stream.py script illustrates. We have stress-tested this character-counting protocol to extremes and it has not yet failed. Seemingly, only the speed of the serial connection is the limit.

RESERVATION:

If a g-code line is parsed and generates an error response message, a GUI should stop the stream immediately. However, since the character-counting method stuffs Grbl's RX buffer, Grbl will continue reading from the RX buffer and parse and execute the commands inside it. A GUI won't be able to control this. The interim solution is to check all of the g-code via the $C check mode, so all errors are vetted prior to streaming. This will get resolved in later versions of Grbl.

[t-oster]

Seem reasonable. If we could add a setting to keep the old behavior (wait for each line), we have a backup solution for devices which are not having the exact 128 byte buffer. Or we even make the bufferlength configurable and if it's zero, then we have the old behavior.

[tatarize]

Fake GRBL simulator exists for testing purposes.

grbl/grbl-sim#20

So it really could be tested fairly safely. Runs a sandboxed fake GRBL.

[tatarize]

The core idea behind the character counting is solid and should be added to the generic. It seems like you could, without altering anything drastic. Add quick way to calculate how many bytes of gcode is being processed. Then the default functionality is wait until processing 0 bytes of data. And the GRBL functionality could be if I have a line that's 45 characters long, I need to wait until it's processing less than 83 characters before I send that line.

You could kinda generalize the character counting by telling the driver authors how many bytes are currently being processed. And when you ask, it tries reading some more 'ok.' from the stream to give you the updated value. So they could trigger the loop in waitForLine() themselves. Simply by doing:

while (processingDataLength() + line.length() > 128) { }

And the default loop would just be:

while (processingDataLength() != 0) { }

Which would try to read as many queued up ok or error replies as it can, decrementing the length queue and returning the value.

[tatarize]

The other mode in the driver is isWaitForOKafterEachLine() which basically just takes the training wheels off and sends the entire file, firehose style, for non-grbl stuff this is fine and expected. But, a super-generic character counting scheme would add a queued value for the length for each line sent, and adding values to the processing and never decrement that processing total or try to read ok responses (since you'd never query processingDataLength()). Which might be fine. But, you're basically asking for trouble if you go around open-ended storing all the values of all the every line you sent. It could be a lot of values. So I'd properly check if the queue is null, and if it is just skip don't process that (it could still add the processing length stuff value since that won't matter and is just an int). So character counting would require initializing the character counting queue, and overloading the default send routine with a slightly different one.

[tatarize]

This would at least take that idea and divide it into two easy phases. The first being pretty harmless additional functionality to GenericGcodeWriter.java to permit character counting. Adding line length queueing functionality, if a line length queue exists, and modifying the waitForLine() functionality to adjust the processing data length when an ok. is read.

Though it might need a slight tweak since it would still need waitForLine() in the case that you want to send one line, wait for the ok, then send the next but run that without initializing a queue for the data lengths. If we don't do any character counting in most cases you wouldn't really get the processingDataLength() back to zero.

It's almost an elegant solution. Seems like there'd be a better one.

[tatarize]

If the routine in the sendLine() is made to be a function called waitForOk() and in that function we do character counting, when there's a command length queue initialized. And we add a slight tweak to sendLine to store how many bytes are processing. We'd get all the parts we need for modifying that behavior without any major changes.

Then the routine in the GRBL driver would set isWaitForOkafterEachLine() to false, initialize a queue for the commandLengths(). And overload sendLine protocol. GRBL driver would then need to check the processingBytes() and while we don't have enough bytes, call waitForOk(). Then call for the actual data to be sent. It change nothing about the default driver per se, but completely permit character counting to be done easily in an overloaded driver that calls our waitForOk() function while the characterBuffer is higher than we allow, before it calls the part that sends the data along our pipe.

This might be a properly elegant solution.

But, it's also, a two phased solution. So the first would change nothing about the drivers or how it works, but, move a little bit of functionality and a queue for the lengths that we never initialize, and a counter for the buffer we've sent.

In most the standard protocol would remain:

1. Send Data.
2. If waitForOkafterEachLine, then waitForOk.

In GRBL an option would allow you to switch that out:

1. while (currentLine + buffer > 128) waitForOk.
2. Send Data.

It would just need small tweaks to allow the generic driver to keep a count of the data sent, run a queue and decrement that count by that amount if the queue exists, and a breaking up the sendLine into a couple different functions so they could be rearranged and allow the implied protocol there to be altered.

[tatarize]

See #154 for my Phase 1 solution as well as the modifications for Phase 2.

It should be quickly provable that Phase 1 is harmless. Then Phase 2 might require some GRBL device and additional testing. Since, unlike phase 1, it would modify how some things work as it would add character counting and modify the send protocol.