Queries cannot contain a string literal or quoted name with a semicolon (e.g., ';')

[tconbeer]

Describe the bug
Harlequin naively splits query text on ;, even if that is inside a quote or identifier.

To Reproduce
select ';' as a, then try to run query

Expected behavior
This query should execute fine.

Actual behavior
Harlequin shows a syntax error, since it splits the query as select '

Additional context
What is the output of harlequin --version?
1.6.0

What database adapter are you using with Harlequin? (Default is DuckDB)
DuckDB (affects all)

System:

• Bash
• Windows Terminal
• Ubuntu 22.04 / WSL2
• Windows 11

[tconbeer]

Now that the editor has tree-sitter onboard, consider using a tree-sitter query to find query delimeters. Not sure this is a great idea, as the tree-sitter-sql language seems incomplete. Need to make sure this is robust to different dialects, incomplete queries, etc.

Actually the grammar looks very complete; it's just the highlight query that is very simplistic.

[tconbeer]

just spent time on this. The tree-sitter grammar isn't good enough or maybe is just too rigid. Lots of valid queries return errors. There is also a problem with set operations: the parser doesn't know about union all (and raises an error), and it considers union to end a select statement, so select 1 union select 2 becomes two select_statements, select 1 union and select 2.

That means the options are:

• use sqlfmt's lexer (yuck)
• create a new tree-sitter language that we control (yuck)
• try to improve the upstream language (author hasn't merged PR's since June 2022 and binary is getting huge, among other issues)
• use sqlglot instead?

[tconbeer]

sqlglot frustratingly doesn't include token coordinates in the AST. So we could use it for splitting queries to send to the db, but not for finding the query where the cursor currently is. Could use Dialect.tokenize and run through the token stream instead, looking for semicolons; this is what parse does to split things into chunks.

[ml31415]

I just run into this with an even more confusing behavior:

As some prompts require the SQL statements to be terminated with a semi-colon, I had added one to the end of the query. When the semi-colon is the last character of the prompt, trying to run the prompt works fine. But if it's not the last character, e.g. another new line or space after the semi-colon, then just nothing happens when trying to run the query.

This behavior is extremely confusing, as there is no error message or anything hinting on the issue. It just feels weirdly broken. I guess the queries should be just trimmed automatically before execution.

[tconbeer]

It's true that we split on semicolons and don't include them in the queries we send to the db. We could change that. But I don't understand why it's an issue? What rdbms are you using that requires that queries or statements are terminated by a semicolon? Can you provide an example that doesn't work?

[tconbeer]

@ml31415 I just re-read your initial comment; that is, in fact, a bug, but a totally different issue. I've opened #493 to address your issue.