Merge pull request #140 from swyddfa/develop

New Release

.pre-commit-config.yaml

@@ -10,19 +10,25 @@ repos:
   - id: trailing-whitespace
 
 - repo: https://github.com/psf/black
-  rev: 23.11.0
+  rev: 23.12.1
   hooks:
   - id: black
-    exclude: 'lib/pytest-lsp/pytest_lsp/gen.py'
 
 - repo: https://github.com/PyCQA/flake8
-  rev: 6.1.0
+  rev: 7.0.0
   hooks:
   - id: flake8
+    name: flake8 (lsp-devtools)
     args: [--config=lib/lsp-devtools/setup.cfg]
+    files: 'lib/lsp-devtools/lsp_devtools/.*\.py'
+
+  - id: flake8
+    name: flake8 (pytest-lsp)
+    args: [--config=lib/pytest-lsp/setup.cfg]
+    files: 'lib/pytest-lsp/pytest_lsp/.*\.py'
 
 - repo: https://github.com/pycqa/isort
-  rev: 5.12.0
+  rev: 5.13.2
   hooks:
   - id: isort
     name: isort (lsp-devtools)
@@ -33,10 +39,9 @@ repos:
     name: isort (pytest-lsp)
     args: [--settings-file, lib/pytest-lsp/pyproject.toml]
     files: 'lib/pytest-lsp/pytest_lsp/.*\.py'
-    exclude: 'lib/pytest-lsp/pytest_lsp/gen.py'
 
 - repo: https://github.com/pre-commit/mirrors-mypy
-  rev: 'v1.7.0'
+  rev: 'v1.8.0'
   hooks:
   - id: mypy
     name: mypy (pytest-lsp)

README.md

@@ -4,8 +4,6 @@
 
 This repo is an attempt at building the developer tooling I wished existed when I first started working on [Esbonio](https://github.com/swyddfa/esbonio/).
 
-**Everything here is early in its development, so expect plenty of bugs and missing features.**
-
 This is a monorepo containing a number of sub-projects.
 
 ## `lib/lsp-devtools` - A grab bag of development utilities
@@ -18,7 +16,8 @@ A collection of cli utilities aimed at aiding the development of language server
 
 - `agent`: Used to wrap an lsp server allowing messages sent between it and the client to be intercepted and inspected by other tools.
 - `record`: Connects to an agent and record traffic to file, sqlite db or console. Supports filtering and formatting the output
-- `tui`: A text user interface to visualise and inspect LSP traffic. Powered by [textual](https://textual.textualize.io/)
+- `inspect`: A browser devtools inspired TUI to visualise and inspecting LSP traffic. Powered by [textual](https://textual.textualize.io/)
+- `client`: **Experimental** A TUI language client with built in `inspect` panel. Powered by [textual](https://textual.textualize.io/)
 
 ## `lib/pytest-lsp` - End-to-end testing of language servers with pytest
 

docs/lsp-devtools/guide/record-command.rst

@@ -29,6 +29,7 @@ Here are some example usages of the ``record`` command that you may find useful.
 
 The following command will save to a JSON file only the client's info and :class:`pygls:lsprotocol.types.ClientCapabilities` sent during the ``initialize`` request - useful for :ref:`adding clients to pytest-lsp <pytest-lsp-supported-clients>`! 😉
 
+
 ::
 
    lsp-devtools record -f '{{"clientInfo": {.params.clientInfo}, "capabilities": {.params.capabilities}}}' --to-file <client_name>_v<version>.json
@@ -291,21 +292,28 @@ Formatting messages
 Formatters
 ^^^^^^^^^^
 
-``lsp-devtools`` knows how to format the following LSP Types
+``lsp-devtools`` provides the following formatters
+
+``json`` (default)
+  Renders objects as "pretty" JSON, equivalent to ``json.dumps(obj, indent=2)``
 
-``Position``
+``json-compact``
+  Renders objects as JSON with no additional formatting, equivalent to ``json.dumps(obj)``
+
+``position``
    ``{"line": 1, "character": 2}`` will be rendered as ``1:2``
 
-``Range``
+``range``
    ``{"start": {"line": 1, "character": 2}, "end": {"line": 3, "character": 4}}`` will be rendered as ``1:2-3:4``
 
-Additionally, any enum type can be used as a formatter in which case a number will be replaced with the corresponding name, for example::
+
+Additionally, any enum type can be used as a formatter, where numbers will be replaced with their corresponding name, for example::
 
   Format String:
   "{.type|MessageType}"
 
-  Value:    Result:
-  1         Error
-  2         Warning
-  3         Info
-  4         Log
+  Value:          Result:
+  {"type": 1}     Error
+  {"type": 2}     Warning
+  {"type": 3}     Info
+  {"type": 4}     Log

flake.nix

@@ -8,6 +8,10 @@
 
   outputs = { self, nixpkgs, utils }:
     {
-      overlays.default =  import ./lib/pytest-lsp/nix/pytest-lsp-overlay.nix;
+      overlays.default = self: super:
+        nixpkgs.lib.composeManyExtensions [
+          (import ./lib/pytest-lsp/nix/pytest-lsp-overlay.nix)
+          (import ./lib/lsp-devtools/nix/lsp-devtools-overlay.nix)
+      ] self super;
     };
 }

lib/lsp-devtools/.gitignore

@@ -0,0 +1 @@
+result

lib/lsp-devtools/changes/130.enhancement.md

@@ -0,0 +1 @@
+Added formatters `json` and `json-compact` that can be used within format strings.

lib/lsp-devtools/changes/130.fix.md

@@ -0,0 +1 @@
+The `lsp-devtools record` command will now produce valid JSON when using the `--to-file` option without an explicitly provided format string.

lib/lsp-devtools/changes/132.fix.md

@@ -0,0 +1 @@
+The `lsp-devtools agent` now watches for the when the server process exits and closes itself down also.

lib/lsp-devtools/changes/133.fix.md

@@ -0,0 +1 @@
+Commands like `lsp-devtools record` should now exit cleanly when hitting `Ctrl-C`

lib/lsp-devtools/changes/134.enhancement.md

@@ -0,0 +1 @@
+When not printing messages to stdout, the `lsp-devtools record` command now displays a nice visualisation of the traffic between client and server - so that you can see that it's doing something

lib/lsp-devtools/flake.nix

@@ -0,0 +1,26 @@
+{
+  description = "lsp-devtools: Developer tooling for language servers";
+
+  inputs = {
+    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
+    utils.url = "github:numtide/flake-utils";
+  };
+
+   outputs = { self, nixpkgs, utils }:
+
+     let
+       eachPythonVersion = versions: f:
+         builtins.listToAttrs (builtins.map (version: {name = "py${version}"; value = f version; }) versions);    in {
+
+      overlays.default = import ./nix/lsp-devtools-overlay.nix;
+
+      packages = utils.lib.eachDefaultSystemMap (system:
+        let
+          pkgs = import nixpkgs { inherit system; overlays = [ self.overlays.default ]; };
+        in
+          eachPythonVersion [ "38" "39" "310" "311"] (pyVersion:
+            pkgs."python${pyVersion}Packages".lsp-devtools
+          )
+      );
+    };
+}

lib/lsp-devtools/lsp_devtools/agent/agent.py

@@ -1,10 +1,18 @@
+from __future__ import annotations
+
 import asyncio
 import inspect
 import logging
 import re
-import threading
+import sys
+import typing
 from functools import partial
-from typing import BinaryIO
+
+if typing.TYPE_CHECKING:
+    from typing import BinaryIO
+    from typing import Optional
+    from typing import Set
+    from typing import Tuple
 
 logger = logging.getLogger("lsp_devtools.agent")
 
@@ -22,15 +30,14 @@ async def forward_message(source: str, dest: asyncio.StreamWriter, message: byte
     )
 
 
-# TODO: Upstream this?
-async def aio_readline(stop_event, reader, message_handler):
+async def aio_readline(reader: asyncio.StreamReader, message_handler):
     CONTENT_LENGTH_PATTERN = re.compile(rb"^Content-Length: (\d+)\r\n$")
 
     # Initialize message buffer
     message = []
     content_length = 0
 
-    while not stop_event.is_set():
+    while True:
         # Read a header line
         header = await reader.readline()
         if not header:
@@ -42,7 +49,6 @@ async def aio_readline(stop_event, reader, message_handler):
             match = CONTENT_LENGTH_PATTERN.fullmatch(header)
             if match:
                 content_length = int(match.group(1))
-                logger.debug("Content length: %s", content_length)
 
         # Check if all headers have been read (as indicated by an empty line \r\n)
         if content_length and not header.strip():
@@ -62,7 +68,9 @@ async def aio_readline(stop_event, reader, message_handler):
             content_length = 0
 
 
-async def get_streams(stdin, stdout):
+async def get_streams(
+    stdin, stdout
+) -> Tuple[asyncio.StreamReader, asyncio.StreamWriter]:
     """Convert blocking stdin/stdout streams into async streams."""
     loop = asyncio.get_running_loop()
 
@@ -87,38 +95,66 @@ def __init__(
         self.stdin = stdin
         self.stdout = stdout
         self.server = server
-        self.stop_event = threading.Event()
+
+        self._tasks: Set[asyncio.Task] = set()
+        self.reader: Optional[asyncio.StreamReader] = None
+        self.writer: Optional[asyncio.StreamWriter] = None
 
     async def start(self):
         # Get async versions of stdin/stdout
-        reader, writer = await get_streams(self.stdin, self.stdout)
+        self.reader, self.writer = await get_streams(self.stdin, self.stdout)
+
+        # Keep mypy happy
+        assert self.server.stdin
+        assert self.server.stdout
 
         # Connect stdin to the subprocess' stdin
-        client_to_server = aio_readline(
-            self.stop_event,
-            reader,
-            partial(forward_message, "client", self.server.stdin),
+        client_to_server = asyncio.create_task(
+            aio_readline(
+                self.reader,
+                partial(forward_message, "client", self.server.stdin),
+            ),
         )
+        self._tasks.add(client_to_server)
 
         # Connect the subprocess' stdout to stdout
-        server_to_client = aio_readline(
-            self.stop_event,
-            self.server.stdout,
-            partial(forward_message, "server", writer),
+        server_to_client = asyncio.create_task(
+            aio_readline(
+                self.server.stdout,
+                partial(forward_message, "server", self.writer),
+            ),
         )
+        self._tasks.add(server_to_client)
 
         # Run both connections concurrently.
-        return await asyncio.gather(
+        await asyncio.gather(
             client_to_server,
             server_to_client,
+            self._watch_server_process(),
         )
 
+    async def _watch_server_process(self):
+        """Once the server process exits, ensure that the agent is also shutdown."""
+        ret = await self.server.wait()
+        print(f"Server process exited with code: {ret}", file=sys.stderr)
+        await self.stop()
+
     async def stop(self):
-        self.stop_event.set()
-
-        try:
-            self.server.terminate()
-            ret = await self.server.wait()
-            print(f"Server process exited with code: {ret}")
-        except TimeoutError:
-            self.server.kill()
+        # Kill the server process if necessary.
+        if self.server.returncode is None:
+            try:
+                self.server.terminate()
+                await asyncio.wait_for(self.server.wait(), timeout=5)  # s
+            except TimeoutError:
+                self.server.kill()
+
+        args = {}
+        if sys.version_info.minor > 8:
+            args["msg"] = "lsp-devtools agent is stopping."
+
+        # Cancel the tasks connecting client to server
+        for task in self._tasks:
+            task.cancel(**args)
+
+        if self.writer:
+            self.writer.close()

lib/lsp-devtools/lsp_devtools/agent/server.py

@@ -42,23 +42,26 @@ def feature(self, feature_name: str, options: Optional[Any] = None):
     async def start_tcp(self, host: str, port: int) -> None:  # type: ignore[override]
         async def handle_client(reader, writer):
             self.lsp.connection_made(writer)
-            await aio_readline(self._stop_event, reader, self.lsp.data_received)
 
-            writer.close()
-            await writer.wait_closed()
+            try:
+                await aio_readline(self._stop_event, reader, self.lsp.data_received)
+            except asyncio.CancelledError:
+                pass
+            finally:
+                writer.close()
+                await writer.wait_closed()
 
             # Uncomment if we ever need to introduce a mode where the server stops
             # automatically once a session ends.
             #
-            # if self._tcp_server is not None:
-            #     self._tcp_server.cancel()
+            # self.stop()
 
         server = await asyncio.start_server(handle_client, host, port)
         async with server:
             self._tcp_server = asyncio.create_task(server.serve_forever())
             await self._tcp_server
 
-    async def stop(self):
+    def stop(self):
         if self._tcp_server is not None:
             self._tcp_server.cancel()