Semantic tokenizer improvements: tests, simplification, syntax-in-syntax bugfix

Semantic tokenizer improvements: tests, simplification, syntax-in-syntax bugfix #1004