Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Asynchronous cog/extension loading. #1132

Open
wants to merge 34 commits into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
3c35a8c
Port #7528
elenakrittik Nov 6, 2023
d6877d6
Port #7545
elenakrittik Nov 6, 2023
c24dc14
fix(autoshard): Errors when trying to use AutoSharded client variants…
elenakrittik Nov 6, 2023
124890b
fix(test_bot): Update according to new constraints.
elenakrittik Nov 7, 2023
8f85551
refactor: Further reduce loop state sharing.
elenakrittik Nov 7, 2023
26c4663
misc(debug): Set a name for hearbeat thread.
elenakrittik Nov 7, 2023
213edba
compat: Add Client.loop property for compatibility.
elenakrittik Nov 7, 2023
e76ce01
compat: Bring back connector and revert behavior changes.
elenakrittik Nov 12, 2023
fb8b1f2
fix: ruff errors
elenakrittik Nov 12, 2023
f812d1c
docs: Remove syncio_debug param from Client.
elenakrittik Nov 12, 2023
c3988ff
refactor: Async extensions (& cogs).
elenakrittik Nov 12, 2023
2b30e94
fix: Revert setup to be a hook instead of an event.
elenakrittik Nov 12, 2023
ebea051
docs: Fix setup_hook docstring.
elenakrittik Nov 12, 2023
f37594e
fix: Self-review fixes
elenakrittik Nov 12, 2023
1900902
docs: Add changelogs.
elenakrittik Nov 12, 2023
1daebcd
docs: Add example usage of setup_hook.
elenakrittik Nov 12, 2023
3c1baa9
docs: Post-PR-open fixes. Of course.
elenakrittik Nov 12, 2023
5566e17
fix: Formatting.
elenakrittik Nov 12, 2023
416333e
docs: Minor wording fix.
elenakrittik Nov 12, 2023
f677880
fix: run codemod
elenakrittik Nov 12, 2023
8a873a5
Merge branch 'master' into refactor/async
elenakrittik Nov 12, 2023
abd0a1d
fix: propagate ignore_session_start_limit
elenakrittik Nov 14, 2023
b68f114
Merge branch 'refactor/async' of https://github.com/elenakrittik/disn…
elenakrittik Nov 14, 2023
1d740c3
docs: 3.0
elenakrittik Nov 16, 2023
94ceb0b
Merge branch 'master' into refactor/async
elenakrittik Nov 16, 2023
f38fed2
Merge branch 'master' into refactor/async
elenakrittik Nov 18, 2023
a078653
Merge branch 'master' into refactor/async
elenakrittik Jan 21, 2024
6cc4a6d
merge latest changes
elenakrittik May 9, 2024
d8e80b0
Merge branch 'master' of https://github.com/DisnakeDev/disnake into r…
elenakrittik May 9, 2024
0d34a70
Apply suggestions from code review
elenakrittik May 10, 2024
45382d6
Merge branch 'master' into refactor/async
elenakrittik May 10, 2024
a0c5a8e
fix voice sending
elenakrittik May 14, 2024
df499a7
improve error message
elenakrittik May 14, 2024
c0ab613
remove dependency
elenakrittik May 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog/1132.breaking.0.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Removed ``loop`` and ``asyncio_debug`` parameters from :class:`Client`.
1 change: 1 addition & 0 deletions changelog/1132.breaking.1.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The majority of the library now assumes that there is an asyncio event loop running.
1 change: 1 addition & 0 deletions changelog/1132.deprecate.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Deprecated :attr:`Client.loop`. Use :func:`asyncio.get_running_loop` instead.
1 change: 1 addition & 0 deletions changelog/1132.feature.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add :meth:`Client.setup_hook`.
1 change: 1 addition & 0 deletions changelog/1132.misc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
:meth:`Client.run` now uses :func:`asyncio.run` under-the-hood, instead of custom runner logic.
1 change: 1 addition & 0 deletions changelog/641.breaking.0.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
|commands| Make :meth:`.ext.commands.Bot.load_extensions`, :meth:`.ext.commands.Bot.load_extension`, :meth:`.ext.commands.Bot.unload_extension`, :meth:`.ext.commands.Bot.reload_extension`, :meth:`.ext.commands.Bot.add_cog` and :meth:`.ext.commands.Bot.remove_cog` asynchronous.
1 change: 1 addition & 0 deletions changelog/641.breaking.1.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
|commands| :meth:`.ext.commands.Cog.cog_load` is now called *after* the cog finished loading.
1 change: 1 addition & 0 deletions changelog/641.feature.0.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
|commands| :meth:`.ext.commands.Cog.cog_load` and :meth:`.ext.commands.Cog.cog_unload` can now be either asynchronous or not.
Copy link
Contributor Author

@elenakrittik elenakrittik May 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: is this wording okay?

1 change: 1 addition & 0 deletions changelog/641.feature.1.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
|commands| The ``setup`` and ``teardown`` functions utilized by :ref:`ext_commands_extensions` can now be asynchronous.
181 changes: 78 additions & 103 deletions disnake/client.py
Original file line number Diff line number Diff line change
@@ -4,7 +4,6 @@

import asyncio
import logging
import signal
import sys
import traceback
import types
@@ -82,7 +81,7 @@
from .widget import Widget

if TYPE_CHECKING:
from typing_extensions import NotRequired
from typing_extensions import Never, NotRequired

from .abc import GuildChannel, PrivateChannel, Snowflake, SnowflakeTime
from .app_commands import APIApplicationCommand, MessageCommand, SlashCommand, UserCommand
@@ -113,41 +112,6 @@
_log = logging.getLogger(__name__)


def _cancel_tasks(loop: asyncio.AbstractEventLoop) -> None:
tasks = {t for t in asyncio.all_tasks(loop=loop) if not t.done()}

if not tasks:
return

_log.info("Cleaning up after %d tasks.", len(tasks))
for task in tasks:
task.cancel()

loop.run_until_complete(asyncio.gather(*tasks, return_exceptions=True))
_log.info("All tasks finished cancelling.")

for task in tasks:
if task.cancelled():
continue
if task.exception() is not None:
loop.call_exception_handler(
{
"message": "Unhandled exception during Client.run shutdown.",
"exception": task.exception(),
"task": task,
}
)


def _cleanup_loop(loop: asyncio.AbstractEventLoop) -> None:
try:
_cancel_tasks(loop)
loop.run_until_complete(loop.shutdown_asyncgens())
finally:
_log.info("Closing the event loop.")
loop.close()


class SessionStartLimit:
"""A class that contains information about the current session start limit,
at the time when the client connected for the first time.
@@ -237,13 +201,6 @@ class Client:

.. versionchanged:: 1.3
Allow disabling the message cache and change the default size to ``1000``.
loop: Optional[:class:`asyncio.AbstractEventLoop`]
The :class:`asyncio.AbstractEventLoop` to use for asynchronous operations.
Defaults to ``None``, in which case the default event loop is used via
:func:`asyncio.get_event_loop()`.
asyncio_debug: :class:`bool`
Whether to enable asyncio debugging when the client starts.
Defaults to False.
connector: Optional[:class:`aiohttp.BaseConnector`]
The connector to use for connection pooling.
proxy: Optional[:class:`str`]
@@ -361,8 +318,6 @@ class Client:
----------
ws
The websocket gateway the client is currently connected to. Could be ``None``.
loop: :class:`asyncio.AbstractEventLoop`
The event loop that the client uses for asynchronous operations.
session_start_limit: Optional[:class:`SessionStartLimit`]
Information about the current session start limit.
Only available after initiating the connection.
@@ -378,8 +333,6 @@ class Client:
def __init__(
self,
*,
asyncio_debug: bool = False,
loop: Optional[asyncio.AbstractEventLoop] = None,
shard_id: Optional[int] = None,
shard_count: Optional[int] = None,
enable_debug_events: bool = False,
@@ -405,23 +358,27 @@ def __init__(
# self.ws is set in the connect method
self.ws: DiscordWebSocket = None # type: ignore

if loop is None:
with warnings.catch_warnings():
warnings.simplefilter("ignore", DeprecationWarning)
self.loop: asyncio.AbstractEventLoop = asyncio.get_event_loop()
else:
self.loop: asyncio.AbstractEventLoop = loop

self.loop.set_debug(asyncio_debug)
self._listeners: Dict[str, List[Tuple[asyncio.Future, Callable[..., bool]]]] = {}
self.session_start_limit: Optional[SessionStartLimit] = None

if connector:
try:
asyncio.get_running_loop()
except RuntimeError:
raise RuntimeError(
(
"`connector` was created outside of an asyncio loop, which will likely cause"
"issues later down the line due to the client and `connector` running on"
"different asyncio loops; consider moving client instantiation to an '`async"
"main`' function and then manually asyncio.run it"
)
) from None

self.http: HTTPClient = HTTPClient(
connector,
proxy=proxy,
proxy_auth=proxy_auth,
unsync_clock=assume_unsync_clock,
loop=self.loop,
)

self._handlers: Dict[str, Callable] = {
@@ -504,7 +461,6 @@ def _get_state(
handlers=self._handlers,
hooks=self._hooks,
http=self.http,
loop=self.loop,
max_messages=max_messages,
application_id=application_id,
heartbeat_timeout=heartbeat_timeout,
@@ -525,6 +481,28 @@ def _handle_first_connect(self) -> None:
return
self._first_connect.set()

@property
def loop(self):
""":class:`asyncio.AbstractEventLoop`: Same as :func:`asyncio.get_running_loop`.

.. deprecated:: 3.0
Use :func:`asyncio.get_running_loop` directly.
"""
warnings.warn(
"Accessing `Client.loop` is deprecated. Use `asyncio.get_running_loop()` instead.",
category=DeprecationWarning,
stacklevel=2,
)
return asyncio.get_running_loop()

@loop.setter
def loop(self, _value: Never) -> None:
warnings.warn(
"Setting `Client.loop` is deprecated and has no effect. Use `asyncio.get_running_loop()` instead.",
Copy link
Contributor Author

@elenakrittik elenakrittik May 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: i'm not sure the suggestion is correct (apart from the typo (get => set), i don't know if set_event_loop will work on-the-fly); might be better to remove the suggestion altogether?

category=DeprecationWarning,
stacklevel=2,
)

@property
def latency(self) -> float:
""":class:`float`: Measures latency between a HEARTBEAT and a HEARTBEAT_ACK in seconds.
@@ -1015,12 +993,34 @@ async def before_identify_hook(self, shard_id: Optional[int], *, initial: bool =
if not initial:
await asyncio.sleep(5.0)

async def setup_hook(self) -> None:
"""A hook that allows you to perform asynchronous setup like
initiating database connections or loading cogs/extensions after
the bot is logged in but before it has connected to the websocket.

This is only called once, in :meth:`.login`, before any events are
dispatched, making it a better solution than doing such setup in
the :func:`disnake.on_ready` event.

.. warning::
Since this is called *before* the websocket connection is made,
anything that waits for the websocket will deadlock, which includes
methods like :meth:`.wait_for`, :meth:`.wait_until_ready`
and :meth:`.wait_until_first_connect`.

.. versionadded:: 3.0
"""

# login state management

async def login(self, token: str) -> None:
"""|coro|

Logs in the client with the specified credentials.
Logs in the client with the specified credentials and calls
:meth:`.setup_hook`.

.. versionchanged:: 3.0
Now also calls :meth:`.setup_hook`.

Parameters
----------
@@ -1044,6 +1044,8 @@ async def login(self, token: str) -> None:
data = await self.http.static_login(token.strip())
self._connection.user = ClientUser(state=self._connection, data=data)

await self.setup_hook()

async def connect(
self, *, reconnect: bool = True, ignore_session_start_limit: bool = False
) -> None:
@@ -1245,10 +1247,14 @@ async def start(
TypeError
An unexpected keyword argument was received.
"""
await self.login(token)
await self.connect(
reconnect=reconnect, ignore_session_start_limit=ignore_session_start_limit
)
try:
await self.login(token)
await self.connect(
reconnect=reconnect, ignore_session_start_limit=ignore_session_start_limit
)
finally:
if not self.is_closed():
await self.close()

def run(self, *args: Any, **kwargs: Any) -> None:
"""A blocking call that abstracts away the event loop
@@ -1258,57 +1264,26 @@ def run(self, *args: Any, **kwargs: Any) -> None:
function should not be used. Use :meth:`start` coroutine
or :meth:`connect` + :meth:`login`.

Roughly Equivalent to: ::
Equivalent to: ::

try:
loop.run_until_complete(start(*args, **kwargs))
asyncio.run(start(*args, **kwargs))
except KeyboardInterrupt:
loop.run_until_complete(close())
# cancel all tasks lingering
finally:
loop.close()
return

.. warning::

This function must be the last function to call due to the fact that it
is blocking. That means that registration of events or anything being
called after this function call will not execute until it returns.
"""
loop = self.loop

.. versionchanged:: 3.0
Changed to use :func:`asyncio.run`, instead of custom logic.
"""
try:
loop.add_signal_handler(signal.SIGINT, lambda: loop.stop())
loop.add_signal_handler(signal.SIGTERM, lambda: loop.stop())
except NotImplementedError:
pass

async def runner() -> None:
try:
await self.start(*args, **kwargs)
finally:
if not self.is_closed():
await self.close()

def stop_loop_on_completion(f) -> None:
loop.stop()

future = asyncio.ensure_future(runner(), loop=loop)
future.add_done_callback(stop_loop_on_completion)
try:
loop.run_forever()
asyncio.run(self.start(*args, **kwargs))
except KeyboardInterrupt:
_log.info("Received signal to terminate bot and event loop.")
finally:
future.remove_done_callback(stop_loop_on_completion)
_log.info("Cleaning up tasks.")
_cleanup_loop(loop)

if not future.cancelled():
try:
return future.result()
except KeyboardInterrupt:
# I am unsure why this gets raised here but suppress it anyway
return None
return

# properties

@@ -1798,7 +1773,7 @@ def check(reaction, user):
arguments that mirrors the parameters passed in the
:ref:`event <disnake_api_events>`.
"""
future = self.loop.create_future()
future = asyncio.get_running_loop().create_future()
if check is None:

def _check(*args) -> bool:
3 changes: 1 addition & 2 deletions disnake/context_managers.py
Original file line number Diff line number Diff line change
@@ -26,7 +26,6 @@ def _typing_done_callback(fut: asyncio.Future) -> None:

class Typing:
def __init__(self, messageable: Union[Messageable, ThreadOnlyGuildChannel]) -> None:
self.loop: asyncio.AbstractEventLoop = messageable._state.loop
self.messageable: Union[Messageable, ThreadOnlyGuildChannel] = messageable

async def do_typing(self) -> None:
@@ -42,7 +41,7 @@ async def do_typing(self) -> None:
await asyncio.sleep(5)

def __enter__(self) -> Self:
self.task: asyncio.Task = self.loop.create_task(self.do_typing())
self.task: asyncio.Task = asyncio.create_task(self.do_typing())
self.task.add_done_callback(_typing_done_callback)
return self

10 changes: 0 additions & 10 deletions disnake/ext/commands/bot.py
Original file line number Diff line number Diff line change
@@ -10,8 +10,6 @@
from .interaction_bot_base import InteractionBotBase

if TYPE_CHECKING:
import asyncio

import aiohttp
from typing_extensions import Self

@@ -237,8 +235,6 @@ def __init__(
sync_commands: bool = ...,
sync_commands_debug: bool = ...,
sync_commands_on_cog_unload: bool = ...,
asyncio_debug: bool = False,
loop: Optional[asyncio.AbstractEventLoop] = None,
shard_id: Optional[int] = None,
shard_count: Optional[int] = None,
enable_debug_events: bool = False,
@@ -289,8 +285,6 @@ def __init__(
sync_commands: bool = ...,
sync_commands_debug: bool = ...,
sync_commands_on_cog_unload: bool = ...,
asyncio_debug: bool = False,
loop: Optional[asyncio.AbstractEventLoop] = None,
shard_ids: Optional[List[int]] = None, # instead of shard_id
shard_count: Optional[int] = None,
enable_debug_events: bool = False,
@@ -438,8 +432,6 @@ def __init__(
sync_commands: bool = ...,
sync_commands_debug: bool = ...,
sync_commands_on_cog_unload: bool = ...,
asyncio_debug: bool = False,
loop: Optional[asyncio.AbstractEventLoop] = None,
shard_id: Optional[int] = None,
shard_count: Optional[int] = None,
enable_debug_events: bool = False,
@@ -483,8 +475,6 @@ def __init__(
sync_commands: bool = ...,
sync_commands_debug: bool = ...,
sync_commands_on_cog_unload: bool = ...,
asyncio_debug: bool = False,
loop: Optional[asyncio.AbstractEventLoop] = None,
shard_ids: Optional[List[int]] = None, # instead of shard_id
shard_count: Optional[int] = None,
enable_debug_events: bool = False,
4 changes: 2 additions & 2 deletions disnake/ext/commands/bot_base.py
Original file line number Diff line number Diff line change
@@ -399,8 +399,8 @@ def after_invoke(self, coro: CFT) -> CFT:

# extensions

def _remove_module_references(self, name: str) -> None:
super()._remove_module_references(name)
async def _remove_module_references(self, name: str) -> None:
await super()._remove_module_references(name)
# remove all the commands from the module
for cmd in self.all_commands.copy().values():
if cmd.module and _is_submodule(name, cmd.module):
Loading