Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pyro Proxy resource allocation fix #69

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

Tracersboy
Copy link

1.Remove invalid context management
2.Improve the example and actively release the connection

Although Python has GC that can automatically release resources, when I use it, some resources on the server still cannot be automatically released, causing an exception

Traceback(most recent call last):File "openopc2\gateway server.py", line 110,in <module>
File "Pyro5\server.py", line 293,in requestLoop
File "Pyro5\svr threads.py", line 167, in loop
File "Pyro5\svr threads.py", line 205, in events
File "Pyro5\svr threads.py", line 344, in process
File "threading.py", line 852,in startRuntimeError: can't start new thread

After checking Pyro5,The problem was solved when I added the code for actively releasing resources on the client side.
All of this PR hopes to improve this example for other project users

2.Improve the example and actively release the connection
@renzop
Copy link
Collaborator

renzop commented Dec 11, 2024

Thanks for the pull request. Could you provide some code that and explanation that shows which resources were not released. So that we can reproduce the problem that you wanted to solve before your changes and after. Keep in mind that the code the resources should also be released when an exeption happens wihin the code.

@renzop renzop changed the title I encountered some issues while using it and tried to fix them Pyro Proxy resource allocation fix Dec 11, 2024
@Tracersboy
Copy link
Author

Tracersboy commented Dec 11, 2024

Thanks for your reply !
For the first , in gateway_proxy.py, since the context management object of 'with' will call the __exit__ method at return, this method will call

    def __exit__(self, exc_type, exc_value, traceback):
        self._pyroRelease()
    def _pyroRelease(self):

        """release the connection to the pyro daemon"""
        self.__check_owner()
        if self._pyroConnection is not None:   
            self._pyroConnection.close()
            self._pyroConnection = None
            self._pyroLocalSocket = None

Since Pyro5 has not established a connection, it has no effect inside the _pyroRelease method and will not cause any problems. Anyway, using 'with' here is not wise

For the second,During my use, the server-side encountered the following error

Traceback(most recent call last):File "openopc2\gateway server.py", line 110,in <module>
File "Pyro5\server.py", line 293,in requestLoop
File "Pyro5\svr threads.py", line 167, in loop
File "Pyro5\svr threads.py", line 205, in events
File "Pyro5\svr threads.py", line 344, in process
File "threading.py", line 852,in startRuntimeError: can't start new thread

Through checking the code of Pyro5, I found that the reason for this error is that Pyro5 can support up to 80 client connections by default, with each client being an independent thread managed by a thread pool. Due to the low configuration of my server computer, even when the number of threads is less than 80, the system is unable to create new threads, resulting in an error. The key issue is that I only have one client! The pseudocode is as follows:
[I don't want to use a long connection, i want to request once and then disconnect]

while True:
    try:
        opc_client = get_opc_da_client(open_opc_config)
        print(opc_client.info())
        # read some tags and do some things
        opc_client.close()
        time.sleep(60)
    except Exception as e:
        print(f"error:{e}")

The working principle of Pyro5's server is that there are four idle threads by default. Whenever a client connects, they will use one idle thread and add it to the busy thread group. When there are no idle threads, if the total number of threads is less than 80, it will create a new thread. Whenever a client disconnects (client call _pyroRenease()), if the total number of threads on the server is greater than 4, it will release the thread. If the total number of threads is less than or equal to 4, it will build the thread from busy to idle.

So, theoretically speaking, every time my client executes a loop, the opc_client is re instantiated, old opc_client will be released by GC due to loss of references, the old opc_client __del__ will be called, and then _pyroRelease will be called to release the server-side threads without any problems. The situation is that after running on the server for a few weeks, the above threading error occurred until I manually added opc_cient._PyroRelease() solves the problem.

In addition, it is difficult for me to reproduce this issue locally in a short period of time. I am not sure if this is related to the complex GC mechanism. I have not yet obtained GC, but I can be certain that adding close() _pyroRenease() in the sample is very necessary. This allows project users to choose whether to manually close the connection to free up resources. And for users who are not familiar with Pyro5, the close() method also has a certain degree of confusion, as it only closes OpcDaClient instead of TCP connection

If I haven't described it clearly, please let me know

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants