Stack overflow in error listener

[kaby76]

I've been putting this off, but there is a bug in the error reporter code I wrote for the Antlr program template. I'm checking in the code to reproduce it.

[kaby76]

In order to compute the lookahead sets that a parse expects, the error listener makes a call back to the Antlr runtime to get the lookahead. If the token is invalid, round and round she goes.

pl1.dll!pl1.ErrorListener<int>.SyntaxError(System.IO.TextWriter output, Antlr4.Runtime.IRecognizer recognizer, int offendingSymbol, int line, int col, string msg, Antlr4.Runtime.RecognitionException e) Line 37
    at C:\Users\kenne\Documents\AntlrExamples\pl1\ErrorListener.cs(37)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.ProxyErrorListener<int>.SyntaxError(System.IO.TextWriter output, Antlr4.Runtime.IRecognizer recognizer, int offendingSymbol, int line, int charPositionInLine, string msg, Antlr4.Runtime.RecognitionException e) Line 43
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\ProxyErrorListener.cs(43)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.Lexer.NotifyListeners(Antlr4.Runtime.LexerNoViableAltException e) Line 561
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\Lexer.cs(561)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.Lexer.NextToken() Line 178
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\Lexer.cs(178)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.BufferedTokenStream.Fetch(int n) Line 240
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\BufferedTokenStream.cs(240)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.BufferedTokenStream.Sync(int i) Line 220
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\BufferedTokenStream.cs(220)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.CommonTokenStream.LT(int k) Line 145
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\CommonTokenStream.cs(145)
pl1.dll!pl1.LASets.Compute(Antlr4.Runtime.Parser parser, Antlr4.Runtime.CommonTokenStream token_stream, int line, int col) Line 61
    at C:\Users\kenne\Documents\AntlrExamples\pl1\LASets.cs(61)
pl1.dll!pl1.ErrorListener<int>.SyntaxError(System.IO.TextWriter output, Antlr4.Runtime.IRecognizer recognizer, int offendingSymbol, int line, int col, string msg, Antlr4.Runtime.RecognitionException e) Line 38
    at C:\Users\kenne\Documents\AntlrExamples\pl1\ErrorListener.cs(38)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.ProxyErrorListener<int>.SyntaxError(System.IO.TextWriter output, Antlr4.Runtime.IRecognizer recognizer, int offendingSymbol, int line, int charPositionInLine, string msg, Antlr4.Runtime.RecognitionException e) Line 43
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\ProxyErrorListener.cs(43)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.Lexer.NotifyListeners(Antlr4.Runtime.LexerNoViableAltException e) Line 561
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\Lexer.cs(561)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.Lexer.NextToken() Line 178
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\Lexer.cs(178)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.BufferedTokenStream.Fetch(int n) Line 240
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\BufferedTokenStream.cs(240)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.BufferedTokenStream.Sync(int i) Line 220
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\BufferedTokenStream.cs(220)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.CommonTokenStream.LT(int k) Line 145
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\CommonTokenStream.cs(145)
pl1.dll!pl1.LASets.Compute(Antlr4.Runtime.Parser parser, Antlr4.Runtime.CommonTokenStream token_stream, int line, int col) Line 61
    at C:\Users\kenne\Documents\AntlrExamples\pl1\LASets.cs(61)
pl1.dll!pl1.ErrorListener<int>.SyntaxError(System.IO.TextWriter output, Antlr4.Runtime.IRecognizer recognizer, int offendingSymbol, int line, int col, string msg, Antlr4.Runtime.RecognitionException e) Line 38
    at C:\Users\kenne\Documents\AntlrExamples\pl1\ErrorListener.cs(38)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.ProxyErrorListener<int>.SyntaxError(System.IO.TextWriter output, Antlr4.Runtime.IRecognizer recognizer, int offendingSymbol, int line, int charPositionInLine, string msg, Antlr4.Runtime.RecognitionException e) Line 43
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\ProxyErrorListener.cs(43)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.Lexer.NotifyListeners(Antlr4.Runtime.LexerNoViableAltException e) Line 561
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\Lexer.cs(561)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.Lexer.NextToken() Line 178
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\Lexer.cs(178)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.BufferedTokenStream.Fetch(int n) Line 240
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\BufferedTokenStream.cs(240)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.BufferedTokenStream.Sync(int i) Line 220
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\BufferedTokenStream.cs(220)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.CommonTokenStream.LT(int k) Line 145
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\CommonTokenStream.cs(145)
pl1.dll!pl1.LASets.Compute(Antlr4.Runtime.Parser parser, Antlr4.Runtime.CommonTokenStream token_stream, int line, int col) Line 61
    at C:\Users\kenne\Documents\AntlrExamples\pl1\LASets.cs(61)
pl1.dll!pl1.ErrorListener<int>.SyntaxError(System.IO.TextWriter output, Antlr4.Runtime.IRecognizer recognizer, int offendingSymbol, int line, int col, string msg, Antlr4.Runtime.RecognitionException e) Line 38
    at C:\Users\kenne\Documents\AntlrExamples\pl1\ErrorListener.cs(38)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.ProxyErrorListener<int>.SyntaxError(System.IO.TextWriter output, Antlr4.Runtime.IRecognizer recognizer, int offendingSymbol, int line, int charPositionInLine, string msg, Antlr4.Runtime.RecognitionException e) Line 43
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\ProxyErrorListener.cs(43)

...
pl1.dll!pl1.LASets.Compute(Antlr4.Runtime.Parser parser, Antlr4.Runtime.CommonTokenStream token_stream, int line, int col) Line 61
    at C:\Users\kenne\Documents\AntlrExamples\pl1\LASets.cs(61)
pl1.dll!pl1.ErrorListener<int>.SyntaxError(System.IO.TextWriter output, Antlr4.Runtime.IRecognizer recognizer, int offendingSymbol, int line, int col, string msg, Antlr4.Runtime.RecognitionException e) Line 38
    at C:\Users\kenne\Documents\AntlrExamples\pl1\ErrorListener.cs(38)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.ProxyErrorListener<int>.SyntaxError(System.IO.TextWriter output, Antlr4.Runtime.IRecognizer recognizer, int offendingSymbol, int line, int charPositionInLine, string msg, Antlr4.Runtime.RecognitionException e) Line 43
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\ProxyErrorListener.cs(43)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.Lexer.NotifyListeners(Antlr4.Runtime.LexerNoViableAltException e) Line 561
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\Lexer.cs(561)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.Lexer.NextToken() Line 178
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\Lexer.cs(178)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.BufferedTokenStream.Fetch(int n) Line 240
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\BufferedTokenStream.cs(240)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.BufferedTokenStream.Sync(int i) Line 220
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\BufferedTokenStream.cs(220)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.BufferedTokenStream.Consume() Line 191
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\BufferedTokenStream.cs(191)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.Parser.Consume() Line 734
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\Parser.cs(734)
Antlr4.Runtime.Standard.dll!Antlr4.Runtime.Parser.Match(int ttype) Line 245
    at C:\Users\kenne\Documents\antlr4\runtime\CSharp\runtime\CSharp\Antlr4.Runtime\Parser.cs(245)
pl1.dll!pl1_parserParser.pl1stmtlist(int _p) Line 749
    at C:\Users\kenne\Documents\AntlrExamples\pl1\obj\Debug\netcoreapp3.1\pl1_parserParser.cs(749)
pl1.dll!pl1_parserParser.pl1pgm() Line 661
    at C:\Users\kenne\Documents\AntlrExamples\pl1\obj\Debug\netcoreapp3.1\pl1_parserParser.cs(661)
pl1.dll!pl1.Program.Main(string[] args) Line 48
    at C:\Users\kenne\Documents\AntlrExamples\pl1\Program.cs(48)

[kaby76]

Correcting this problem is easy. But, there is another problem. It looks like I'm not getting any information of possible parses for an input with error (e.g., add after "PRTHDG1: PROCEDURE; asdf PUT FILE ...".

[kaby76]

For sure, something is wrong with the interpreter because it faults very early on with no viable paths through the ATN.
xx.txt
x.txt

[kaby76]

The problem is clearly the code in EnterState() to test whether this state needs to be revisited.

        if (_visited.ContainsKey(new Pair<ATNState, int>(state, token_index)))
        {
            if (_log_parse)
            {
                System.Console.Error.WriteLine(
                    new String(' ', indent * 2)
                    + "already visited.");
            }
            return null;
        }

The code notices it's visiting state 284 (varnamequal) with input PSAM1. The first time it visited this state, it was in the context of a varnameref. The second time it is in the context of prestmtlist. Since it entered this state with the same input index, it assumes that this is just a loop within the ATN. But, it's not.

The problem with this code is that there is a different context between the time state was first visited and the second. State and token_index are insufficient. It should be the entire path to this point.

I'll hold off on a solution until I thought about this a little more, after a good hike tomorrow (Welch-Dickey).

[kaby76]

I'm still getting this problem, e.g., asn.g in an Antlr3 parse, line 960.