Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Python] Add syntax highlighting for type comments #1925

Closed
wants to merge 8 commits into from
Closed

[Python] Add syntax highlighting for type comments #1925

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
137308c
[Python] Add syntax highlighting for type comments
TeamSpen210 Mar 24, 2019
ed61bb4
[Python] Add function type hint comment syntax
TeamSpen210 Mar 26, 2019
56ff18e
[Python] Small fixes/improvements to type comments
TeamSpen210 Oct 17, 2019
482f0ec
Merge branch 'master' into type-comment
wbond Jul 21, 2020
14bf602
[Python] Clean up type comment scopes
wbond Jul 21, 2020
2a57368
Merge branch 'master' into type-comment
wbond Jul 22, 2020
e3b57a9
[Python] Fixed various type comment tests to use valid code
wbond Jul 22, 2020
97ae638
[Python] Ensure type comments are exited at newline
wbond Jul 22, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 126 additions & 2 deletions Python/Python.sublime-syntax
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -411,12 +411,136 @@ contexts:
pop: true

comments:
# Type-ignore can only be used by itself.
- match: '(#)\s*(type)(:)\s*(ignore)\b'
scope: comment.line.type-hint.python
captures:
1: punctuation.definition.comment.python
2: keyword.other.annotation.type-comment.python
3: punctuation.separator.annotation.type-comment.python
4: keyword.other.annotation.type-comment.ignore.python
push: comments-afterignore
- match: '(#)\s*(type)(:)'
scope: comment.line.type-hint.python
captures:
1: punctuation.definition.comment.python
2: keyword.other.annotation.type-comment.python
3: punctuation.separator.annotation.type-comment.python
push: type-comment
- match: "#"
scope: punctuation.definition.comment.python
push:
- meta_scope: comment.line.number-sign.python
push: comments-basic-remainder

comments-basic:
- match: "#"
scope: punctuation.definition.comment.python
set: comments-basic-remainder

wbond marked this conversation as resolved.
Show resolved Hide resolved
comments-basic-remainder:
- meta_scope: comment.line.number-sign.python
- match: \n
pop: true

comments-afterignore:
- meta_content_scope: comment.line.type-hint.python
- match: \n
pop: true
- match: '(?=#)'
set: comments-basic
- match: \[
# Mypy's # type: ignore[error-code1, error-code2]
scope: punctuation.section.brackets.begin.python
set:
- meta_content_scope: comment.line.type-hint.python
- match: \n
pop: true
- match: \[
scope: invalid.illegal.python
- match: '[a-zA-Z_-][a-zA-Z_0-9-]+'
scope: constant.other.error-code.python
- match: '[0-9]\w*'
scope: invalid.illegal.python
- match: ','
scope: punctuation.separator.sequence.python
- match: \]
scope: punctuation.section.brackets.end.python
set:
# Don't allow a second block.
- meta_content_scope: comment.line.type-hint.python
- match: \n
pop: true
- match: '(?=#)'
set: comments-basic
- match: \S
scope: invalid.illegal.python
- match: \S
scope: invalid.illegal.python

type-comment:
- meta_content_scope: comment.line.type-hint.python
- match: \n
pop: true
- match: '(?=#)'
set: comments-basic
# Don't use expressions here, only a few types are reasonable
# as annotations.
- include: constants
- include: builtin-exceptions
- include: builtin-types
- include: builtin-functions
- match: '{{identifier}}'
scope: support.class.python
- match: '\s*(\[)'
captures:
1: meta.item-access.python punctuation.section.brackets.begin.python
push:
- meta_content_scope: meta.item-access.arguments.python
- match: (?=\n)
pop: true
- match: \]
scope: meta.item-access.python punctuation.section.brackets.end.python
pop: true
- match: ':'
scope: punctuation.separator.slice.python
- match: '(?=#)'
- include: type-comment
- match: '\('
scope: punctuation.section.parens.begin.python
push:
- meta_content_scope: meta.function.parameters.python
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be meta_scope so it applies to the ( and ), right?

- match: (?=\n)
pop: true
- match: \)
scope: punctuation.section.parens.end.python
pop: true
- match: \(
scope: invalid.illegal.stray.brace.round.python
- match: \*{3,}
scope: invalid.illegal.syntax.python
- match: \*\*
scope: keyword.operator.unpacking.mapping.python
- match: \*
scope: keyword.operator.unpacking.sequence.python
- match: '->'
scope: invalid.illegal.stray.return.arrow.python
- include: type-comment
- match: '->'
scope: meta.function.return-type.python punctuation.separator.sequence.python
set:
- meta_content_scope: comment.line.type-hint.python meta.function.return-type.python
- match: '->'
scope: invalid.illegal.stray.return.arrow.python
- include: type-comment
- match: ','
scope: punctuation.separator.sequence.python
- match: \)
scope: invalid.illegal.stray.brace.round.python
- match: \]
scope: invalid.illegal.stray.brace.square.python
- match: \}
scope: invalid.illegal.stray.brace.curly.python
- match: \S
scope: invalid.illegal.python

constants:
- match: \b(None|True|False|Ellipsis|NotImplemented|__debug__)\b
Expand Down
89 changes: 89 additions & 0 deletions Python/syntax_test_python.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1487,6 +1487,95 @@ class Starship:
# ^ punctuation.separator.annotation.variable.python
# ^ keyword.operator.assignment

# Type comments - type: ignore must be by itself.

primes = 5 # type: ignore # type: not-a-type-comment
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for clarification

primes = 5  # type: ignore # type: str   // technically ok but weird
primes = 5  # type: str  # type: ignore  // not tested here but typical
primes = 5  # type: str  # type: str     // not tested here and invalid

(I don't know how fine-grained these tests are here.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The impression I had gotten from the previous implementation was that line comments added into a type comment effectively ended the processing of mypy. Is that true?

From your comment above, it almost sounds like mypy will process nested comments that start with type: and the only logical combinations are type: ignore and a normal type: comment, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe mypy stops after the first type: ignore comment. 🤷 After that everything is "egal". The only thing you see in practice is # type: str # type: ignore to brutally force a type. And typical # type: ignore[foo-rule] # mypy bug https:\\issue-nr because you really get hit by a mypy bug every other day.

# ^^^^^^^^^^^^^^^ comment.line.type-hint - comment.line.number-sign
# ^ punctuation.definition.comment
# ^^^^ keyword.other.annotation.type-comment
# ^ punctuation.separator.annotation.type-comment
# ^^^^^^ keyword.other.annotation.type-comment.ignore
# ^^^^^^^^^^^^^^^^^^^^^^^^^^ comment.line.number-sign - comment.line.type-hint
# ^ punctuation.definition.comment

# This is not a type-ignore comment.
value = 3 # type: ignore_class_starting_with_ignore
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ comment.line.type-hint - comment.line.number-sign
# ^ punctuation.definition.comment
# ^^^^ keyword.other.annotation.type-comment
# ^ punctuation.separator.annotation.type-comment
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ support.class

# Mypy ignore extension
value = 3 # type: ignore[error-code-1, 4noint][error] # foo
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ comment.line.type-hint - comment.line.number-sign
# ^^^^^^ keyword.other.annotation.type-comment.ignore
# ^ punctuation.section.brackets.begin
# ^^^^^^^^^^^^ constant.other.error-code
# ^ punctuation.separator.sequence
# ^^^^^^ invalid.illegal
# ^ punctuation.section.brackets.end
# ^^^^^^^ invalid.illegal
# ^^^^^ comment.line.number-sign - comment.line.type-hint

# With/for allow commas
for a, b in lst: # type: (str, int)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parens in (str, int)are not allowed. It's actually Tuple[str, int]. They take explicit > implicit seriously.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am uncertain here. I've created a little test file with

from typing import Tuple

def testing(results):
    for a, b in results:  # type: Tuple[int, str]
        print(a, b)

Hovering a or b with LSP and LSP-pylsp installed, doesn't show anything, while an example using current syntax with or without parentheses works fine.

def testing(results):
    for a, b in results:  # type: (int, str)
        print(a, b)

It also makes sense as I found some comments about such type hints to date back to python 2.7 which doesn't know about typing module.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The related spec can be found at https://mypy.readthedocs.io/en/stable/type_inference_and_annotations.html#declaring-multiple-variable-types-at-a-time

# ^ comment.line.type-hint punctuation.separator.sequence


primes = 5 # type: List[Dict[property:str, ...]] # comment
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ comment.line.type-hint - comment.line.number-sign
# ^ punctuation.definition.comment
# ^^^^ keyword.other.annotation.type-comment
# ^ punctuation.separator.annotation.type-comment
# ^^^^ support.class
# ^^^^^^^^^^^^^^^^^^^^^^^^^ meta.item-access
# ^punctuation.section.brackets.begin
# ^^^^^^^^^^^^^^^^^^^^^^^ meta.item-access.arguments
# ^^^^ support.class
# ^ meta.item-access punctuation.section.brackets.begin
# ^^^^^^^^ support.function.builtin
# ^ punctuation.separator.slice
# ^^^ support.type
# ^ punctuation.separator.sequence
# ^^^ constant.language
# ^ punctuation.section.brackets.end
# ^ punctuation.section.brackets.end
# ^^^^^^^^^^ comment.line.number-sign
# ^ punctuation.definition.comment

# Python 2.7 function annotations.
def function(a, b, *c, **d):
# type: (int, str, List[str], bool) -> Dict[str, str] # type: noncomment
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ comment.line.type-hint
# ^ punctuation.section.parens.begin
# ^^^ meta.function.parameters support.type
# ^ meta.function.parameters punctuation.separator.sequence
# ^^^^ meta.function.parameters support.class
# ^^^^ meta.function.parameters support.type
# ^ punctuation.section.parens.end
# ^^^^^^^^^^^^^^^^^^ meta.function.return-type
# ^^ punctuation.separator.sequence
# ^^^^ support.class
# ^^^^^^^^^^^^^^^^^^ comment.line.number-sign - comment.line.type-hint
# ^ punctuation.definition.comment

class TypeCommentTest:
member = [] # type: List[dict]

def __del__(self) -> None:
# ^^^^ constant.language
# ^ punctuation.section
pass
# ^^^^ keyword.control

class TypeCommentTest2:
member = [] # type: List[dict]

def __del__(self) -> None:
pass
# ^^^^ keyword.control


##################
# Assignment Expressions
Expand Down