-
Notifications
You must be signed in to change notification settings - Fork 139
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor: Asynchronous cog/extension loading. #1132
Open
elenakrittik
wants to merge
34
commits into
DisnakeDev:master
Choose a base branch
from
elenakrittik:refactor/async
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
34 commits
Select commit
Hold shift + click to select a range
3c35a8c
Port #7528
elenakrittik d6877d6
Port #7545
elenakrittik c24dc14
fix(autoshard): Errors when trying to use AutoSharded client variants…
elenakrittik 124890b
fix(test_bot): Update according to new constraints.
elenakrittik 8f85551
refactor: Further reduce loop state sharing.
elenakrittik 26c4663
misc(debug): Set a name for hearbeat thread.
elenakrittik 213edba
compat: Add Client.loop property for compatibility.
elenakrittik e76ce01
compat: Bring back connector and revert behavior changes.
elenakrittik fb8b1f2
fix: ruff errors
elenakrittik f812d1c
docs: Remove syncio_debug param from Client.
elenakrittik c3988ff
refactor: Async extensions (& cogs).
elenakrittik 2b30e94
fix: Revert setup to be a hook instead of an event.
elenakrittik ebea051
docs: Fix setup_hook docstring.
elenakrittik f37594e
fix: Self-review fixes
elenakrittik 1900902
docs: Add changelogs.
elenakrittik 1daebcd
docs: Add example usage of setup_hook.
elenakrittik 3c1baa9
docs: Post-PR-open fixes. Of course.
elenakrittik 5566e17
fix: Formatting.
elenakrittik 416333e
docs: Minor wording fix.
elenakrittik f677880
fix: run codemod
elenakrittik 8a873a5
Merge branch 'master' into refactor/async
elenakrittik abd0a1d
fix: propagate ignore_session_start_limit
elenakrittik b68f114
Merge branch 'refactor/async' of https://github.com/elenakrittik/disn…
elenakrittik 1d740c3
docs: 3.0
elenakrittik 94ceb0b
Merge branch 'master' into refactor/async
elenakrittik f38fed2
Merge branch 'master' into refactor/async
elenakrittik a078653
Merge branch 'master' into refactor/async
elenakrittik 6cc4a6d
merge latest changes
elenakrittik d8e80b0
Merge branch 'master' of https://github.com/DisnakeDev/disnake into r…
elenakrittik 0d34a70
Apply suggestions from code review
elenakrittik 45382d6
Merge branch 'master' into refactor/async
elenakrittik a0c5a8e
fix voice sending
elenakrittik df499a7
improve error message
elenakrittik c0ab613
remove dependency
elenakrittik File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
Removed ``loop`` and ``asyncio_debug`` parameters from :class:`Client`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
The majority of the library now assumes that there is an asyncio event loop running. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
Deprecated :attr:`Client.loop`. Use :func:`asyncio.get_running_loop` instead. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
Add :meth:`Client.setup_hook`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
:meth:`Client.run` now uses :func:`asyncio.run` under-the-hood, instead of custom runner logic. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
|commands| Make :meth:`.ext.commands.Bot.load_extensions`, :meth:`.ext.commands.Bot.load_extension`, :meth:`.ext.commands.Bot.unload_extension`, :meth:`.ext.commands.Bot.reload_extension`, :meth:`.ext.commands.Bot.add_cog` and :meth:`.ext.commands.Bot.remove_cog` asynchronous. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
|commands| :meth:`.ext.commands.Cog.cog_load` is now called *after* the cog finished loading. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
|commands| :meth:`.ext.commands.Cog.cog_load` and :meth:`.ext.commands.Cog.cog_unload` can now be either asynchronous or not. | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
|commands| The ``setup`` and ``teardown`` functions utilized by :ref:`ext_commands_extensions` can now be asynchronous. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -4,7 +4,6 @@ | |
|
||
import asyncio | ||
import logging | ||
import signal | ||
import sys | ||
import traceback | ||
import types | ||
|
@@ -82,7 +81,7 @@ | |
from .widget import Widget | ||
|
||
if TYPE_CHECKING: | ||
from typing_extensions import NotRequired | ||
from typing_extensions import Never, NotRequired | ||
|
||
from .abc import GuildChannel, PrivateChannel, Snowflake, SnowflakeTime | ||
from .app_commands import APIApplicationCommand, MessageCommand, SlashCommand, UserCommand | ||
|
@@ -113,41 +112,6 @@ | |
_log = logging.getLogger(__name__) | ||
|
||
|
||
def _cancel_tasks(loop: asyncio.AbstractEventLoop) -> None: | ||
tasks = {t for t in asyncio.all_tasks(loop=loop) if not t.done()} | ||
|
||
if not tasks: | ||
return | ||
|
||
_log.info("Cleaning up after %d tasks.", len(tasks)) | ||
for task in tasks: | ||
task.cancel() | ||
|
||
loop.run_until_complete(asyncio.gather(*tasks, return_exceptions=True)) | ||
_log.info("All tasks finished cancelling.") | ||
|
||
for task in tasks: | ||
if task.cancelled(): | ||
continue | ||
if task.exception() is not None: | ||
loop.call_exception_handler( | ||
{ | ||
"message": "Unhandled exception during Client.run shutdown.", | ||
"exception": task.exception(), | ||
"task": task, | ||
} | ||
) | ||
|
||
|
||
def _cleanup_loop(loop: asyncio.AbstractEventLoop) -> None: | ||
try: | ||
_cancel_tasks(loop) | ||
loop.run_until_complete(loop.shutdown_asyncgens()) | ||
finally: | ||
_log.info("Closing the event loop.") | ||
loop.close() | ||
|
||
|
||
class SessionStartLimit: | ||
"""A class that contains information about the current session start limit, | ||
at the time when the client connected for the first time. | ||
|
@@ -237,13 +201,6 @@ class Client: | |
|
||
.. versionchanged:: 1.3 | ||
Allow disabling the message cache and change the default size to ``1000``. | ||
loop: Optional[:class:`asyncio.AbstractEventLoop`] | ||
The :class:`asyncio.AbstractEventLoop` to use for asynchronous operations. | ||
Defaults to ``None``, in which case the default event loop is used via | ||
:func:`asyncio.get_event_loop()`. | ||
asyncio_debug: :class:`bool` | ||
Whether to enable asyncio debugging when the client starts. | ||
Defaults to False. | ||
connector: Optional[:class:`aiohttp.BaseConnector`] | ||
The connector to use for connection pooling. | ||
proxy: Optional[:class:`str`] | ||
|
@@ -361,8 +318,6 @@ class Client: | |
---------- | ||
ws | ||
The websocket gateway the client is currently connected to. Could be ``None``. | ||
loop: :class:`asyncio.AbstractEventLoop` | ||
The event loop that the client uses for asynchronous operations. | ||
session_start_limit: Optional[:class:`SessionStartLimit`] | ||
Information about the current session start limit. | ||
Only available after initiating the connection. | ||
|
@@ -378,8 +333,6 @@ class Client: | |
def __init__( | ||
self, | ||
*, | ||
asyncio_debug: bool = False, | ||
loop: Optional[asyncio.AbstractEventLoop] = None, | ||
shard_id: Optional[int] = None, | ||
shard_count: Optional[int] = None, | ||
enable_debug_events: bool = False, | ||
|
@@ -405,23 +358,27 @@ def __init__( | |
# self.ws is set in the connect method | ||
self.ws: DiscordWebSocket = None # type: ignore | ||
|
||
if loop is None: | ||
with warnings.catch_warnings(): | ||
warnings.simplefilter("ignore", DeprecationWarning) | ||
self.loop: asyncio.AbstractEventLoop = asyncio.get_event_loop() | ||
else: | ||
self.loop: asyncio.AbstractEventLoop = loop | ||
|
||
self.loop.set_debug(asyncio_debug) | ||
self._listeners: Dict[str, List[Tuple[asyncio.Future, Callable[..., bool]]]] = {} | ||
self.session_start_limit: Optional[SessionStartLimit] = None | ||
|
||
if connector: | ||
try: | ||
asyncio.get_running_loop() | ||
except RuntimeError: | ||
raise RuntimeError( | ||
( | ||
"`connector` was created outside of an asyncio loop, which will likely cause" | ||
"issues later down the line due to the client and `connector` running on" | ||
"different asyncio loops; consider moving client instantiation to an '`async" | ||
"main`' function and then manually asyncio.run it" | ||
) | ||
) from None | ||
|
||
self.http: HTTPClient = HTTPClient( | ||
connector, | ||
proxy=proxy, | ||
proxy_auth=proxy_auth, | ||
unsync_clock=assume_unsync_clock, | ||
loop=self.loop, | ||
) | ||
|
||
self._handlers: Dict[str, Callable] = { | ||
|
@@ -504,7 +461,6 @@ def _get_state( | |
handlers=self._handlers, | ||
hooks=self._hooks, | ||
http=self.http, | ||
loop=self.loop, | ||
max_messages=max_messages, | ||
application_id=application_id, | ||
heartbeat_timeout=heartbeat_timeout, | ||
|
@@ -525,6 +481,28 @@ def _handle_first_connect(self) -> None: | |
return | ||
self._first_connect.set() | ||
|
||
@property | ||
def loop(self): | ||
""":class:`asyncio.AbstractEventLoop`: Same as :func:`asyncio.get_running_loop`. | ||
|
||
.. deprecated:: 3.0 | ||
Use :func:`asyncio.get_running_loop` directly. | ||
""" | ||
warnings.warn( | ||
"Accessing `Client.loop` is deprecated. Use `asyncio.get_running_loop()` instead.", | ||
category=DeprecationWarning, | ||
stacklevel=2, | ||
) | ||
return asyncio.get_running_loop() | ||
|
||
@loop.setter | ||
def loop(self, _value: Never) -> None: | ||
warnings.warn( | ||
"Setting `Client.loop` is deprecated and has no effect. Use `asyncio.get_running_loop()` instead.", | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Question: i'm not sure the suggestion is correct (apart from the typo ( |
||
category=DeprecationWarning, | ||
stacklevel=2, | ||
) | ||
|
||
@property | ||
def latency(self) -> float: | ||
""":class:`float`: Measures latency between a HEARTBEAT and a HEARTBEAT_ACK in seconds. | ||
|
@@ -1015,12 +993,34 @@ async def before_identify_hook(self, shard_id: Optional[int], *, initial: bool = | |
if not initial: | ||
await asyncio.sleep(5.0) | ||
|
||
async def setup_hook(self) -> None: | ||
"""A hook that allows you to perform asynchronous setup like | ||
initiating database connections or loading cogs/extensions after | ||
the bot is logged in but before it has connected to the websocket. | ||
|
||
This is only called once, in :meth:`.login`, before any events are | ||
dispatched, making it a better solution than doing such setup in | ||
the :func:`disnake.on_ready` event. | ||
|
||
.. warning:: | ||
Since this is called *before* the websocket connection is made, | ||
anything that waits for the websocket will deadlock, which includes | ||
methods like :meth:`.wait_for`, :meth:`.wait_until_ready` | ||
and :meth:`.wait_until_first_connect`. | ||
|
||
.. versionadded:: 3.0 | ||
""" | ||
|
||
# login state management | ||
|
||
async def login(self, token: str) -> None: | ||
"""|coro| | ||
|
||
Logs in the client with the specified credentials. | ||
Logs in the client with the specified credentials and calls | ||
:meth:`.setup_hook`. | ||
|
||
.. versionchanged:: 3.0 | ||
Now also calls :meth:`.setup_hook`. | ||
|
||
Parameters | ||
---------- | ||
|
@@ -1044,6 +1044,8 @@ async def login(self, token: str) -> None: | |
data = await self.http.static_login(token.strip()) | ||
self._connection.user = ClientUser(state=self._connection, data=data) | ||
|
||
await self.setup_hook() | ||
|
||
async def connect( | ||
self, *, reconnect: bool = True, ignore_session_start_limit: bool = False | ||
) -> None: | ||
|
@@ -1245,10 +1247,14 @@ async def start( | |
TypeError | ||
An unexpected keyword argument was received. | ||
""" | ||
await self.login(token) | ||
await self.connect( | ||
reconnect=reconnect, ignore_session_start_limit=ignore_session_start_limit | ||
) | ||
try: | ||
await self.login(token) | ||
await self.connect( | ||
reconnect=reconnect, ignore_session_start_limit=ignore_session_start_limit | ||
) | ||
finally: | ||
if not self.is_closed(): | ||
await self.close() | ||
|
||
def run(self, *args: Any, **kwargs: Any) -> None: | ||
"""A blocking call that abstracts away the event loop | ||
|
@@ -1258,57 +1264,26 @@ def run(self, *args: Any, **kwargs: Any) -> None: | |
function should not be used. Use :meth:`start` coroutine | ||
or :meth:`connect` + :meth:`login`. | ||
|
||
Roughly Equivalent to: :: | ||
Equivalent to: :: | ||
|
||
try: | ||
loop.run_until_complete(start(*args, **kwargs)) | ||
asyncio.run(start(*args, **kwargs)) | ||
except KeyboardInterrupt: | ||
loop.run_until_complete(close()) | ||
# cancel all tasks lingering | ||
finally: | ||
loop.close() | ||
return | ||
|
||
.. warning:: | ||
|
||
This function must be the last function to call due to the fact that it | ||
is blocking. That means that registration of events or anything being | ||
called after this function call will not execute until it returns. | ||
""" | ||
loop = self.loop | ||
|
||
.. versionchanged:: 3.0 | ||
Changed to use :func:`asyncio.run`, instead of custom logic. | ||
""" | ||
try: | ||
loop.add_signal_handler(signal.SIGINT, lambda: loop.stop()) | ||
loop.add_signal_handler(signal.SIGTERM, lambda: loop.stop()) | ||
except NotImplementedError: | ||
pass | ||
|
||
async def runner() -> None: | ||
try: | ||
await self.start(*args, **kwargs) | ||
finally: | ||
if not self.is_closed(): | ||
await self.close() | ||
|
||
def stop_loop_on_completion(f) -> None: | ||
loop.stop() | ||
|
||
future = asyncio.ensure_future(runner(), loop=loop) | ||
future.add_done_callback(stop_loop_on_completion) | ||
try: | ||
loop.run_forever() | ||
asyncio.run(self.start(*args, **kwargs)) | ||
except KeyboardInterrupt: | ||
_log.info("Received signal to terminate bot and event loop.") | ||
finally: | ||
future.remove_done_callback(stop_loop_on_completion) | ||
_log.info("Cleaning up tasks.") | ||
_cleanup_loop(loop) | ||
|
||
if not future.cancelled(): | ||
try: | ||
return future.result() | ||
except KeyboardInterrupt: | ||
# I am unsure why this gets raised here but suppress it anyway | ||
return None | ||
return | ||
|
||
# properties | ||
|
||
|
@@ -1798,7 +1773,7 @@ def check(reaction, user): | |
arguments that mirrors the parameters passed in the | ||
:ref:`event <disnake_api_events>`. | ||
""" | ||
future = self.loop.create_future() | ||
future = asyncio.get_running_loop().create_future() | ||
if check is None: | ||
|
||
def _check(*args) -> bool: | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Question: is this wording okay?