docs: Further docs and update planning

CHANGELOG.md

@@ -1,6 +1,6 @@
-# Major Version 0.2
+# Major Version 0.3
 
-## 0.2.1
+## 0.3.0
 
 ### New Commands
 
@@ -10,10 +10,15 @@
 
 ### To come
 
+* Use `[!let! #x = 12]` instead of `[!set! ..]`.
+* Disallow `[!let! #x =]` and require `[!let! #x = [!empty!]]` (give a good error message).
 * `[!while! cond {}]`
 * `[!for! #x in [#y] {}]`... and make it so that whether commands or variable substitutions get replaced by groups depends on the interpreter context (likely only expressions should use groups)
 * `[!group! ...]` which wraps the tokens in a transparent group. Useful with `!for!`.
-* Remove `[!increment! ...]` and replace with `[!set! #x += 1]`
+* `[!empty!]`
+* `[!is_empty! #stream]`
+* `[!range! 0..5]`
+* Remove `[!increment! ...]` and replace with `[!assign! #x += 1]`
 * Support `!else if!` in `!if!`
 * Token stream manipulation... and make it performant
   * Extend token stream
@@ -26,6 +31,8 @@
 * Work on book
   * Including documenting expressions
 
+# Major Version 0.2
+
 ## 0.2.0
 
 * Rename the string case conversion commands to be less noisy by getting rid of the case suffix

KNOWN_ISSUES.md

@@ -1,4 +1,4 @@
-# Since Major Version 0.2.1
+# Since Major Version 0.3.0
 
 * `[MAX LITERAL]` - In an expression, the maximum negative integer literal is not valid, for example `-128i8`.
   * By comparison, in `rustc`, there is special handling for such literals, so that e.g. `-128i8` and `--(-128i8)` are accepted.

README.md

@@ -418,19 +418,20 @@ preinterpret::preinterpret! {
 * `[!skip! #stream 4]` expects to receive a (possibly transparent) group, and reads and drops the first 4 token trees from the group's stream, and outputs the rest
 * `[!ungroup! #stream]` expects `#stream` to be a single group and unwraps it once
 * `[!flatten! #stream]` removes all singleton groups from `#stream`, leaving a token stream of idents, literals and punctuation
-* `[!at! #stream 2]` takes the 2nd token tree from the stream
-* `[!zip! #a #b #c]` returns a transparent group tuple of the nth token in each of `#a`, `#b` and `#c`. Errors if they are of different lengths.
+* `[!index! #stream 0]` takes the 0th token tree from the stream
+* `[!zip! #a #b #c]` returns a transparent group tuple of the nth token in each of `#a`, `#b` and `#c`. Errors if they are of different lengths
 
 We could support a piped calling convention, such as the `[!pipe! ...]` special command: `[!pipe! #stream as #x |> [!skip! #x 4] |> [!ungroup! #x]]`
 
 #### Possible extension: Better performance
 
 Do some testing and ensure there are tools to avoid `N^2` performance when doing token manipulation, e.g. with:
 
-* `[!push! #stream new tokens...]`
-* `[!take! #stream #x]` where `#x` is read as the first token tree from `#stream`
-* `[!take! #stream (<PARSE_DESTRUCTURING>)]` where the parser is read greadily from `#stream`
-* `[!is_empty! #stream]`
+* `[!append! #stream += new tokens...]`
+* `[!consume_from! #stream #x]` where `#x` is read as the first token tree from `#stream`
+* `[!consume_from! #stream (<PARSE_DESTRUCTURING>)]` where the parser is read greedily from `#stream`
+
+We could consider tweaking some commands to execute lazily, i.e. change the infrastructure to operate over a `TokenIterable` which is either a `TokenStream` or `LazyTokenStream`. Things like `[!zip! ...]` or `[!range! ...]` could then output a `LazyTokenStream`.
 
 ### Possible extension: User-defined commands