Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add remote python console based on named pipes #1009

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions Sources/Plasma/Apps/plClient/plClient.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@ You can contact Cyan Worlds, Inc. by email [email protected]
#include "pfCharacter/pfConfirmationMgr.h"
#include "pfCharacter/pfMarkerMgr.h"
#include "pfConsole/pfConsole.h"
#include "pfConsole/pfRemoteConsole.h"
#include "pfConsole/pfConsoleDirSrc.h"
#include "pfConsoleCore/pfConsoleEngine.h"
#if defined(PLASMA_PIPELINE_DX)
Expand Down Expand Up @@ -1389,6 +1390,12 @@ bool plClient::StartInit()
fConsole->RegisterAs( kConsoleObject_KEY ); // fixedKey from plFixedKey.h
fConsole->Init( fConsoleEngine );

#ifndef PLASMA_EXTERNAL_RELEASE
fRemoteConsole = new pfRemoteConsole();
fRemoteConsole->RegisterAs( kRemoteConsoleObject_KEY );
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't look like we ever need to address messages to the RemoteConsole by its key, so we probably don't need a fixed key entry for it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You still need a key to receive messages though, so this is not really incorrect.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is mostly copy pasted from the fConsole code above. I used to not have the RegisterAs call, but the messages wouldn't ever arrive to the console. Is there another way to register the plRemoteConsole without a fixed key?

fRemoteConsole->Init();
#endif

/// Init the font cache
fFontCache = new plFontCache();

Expand Down
2 changes: 2 additions & 0 deletions Sources/Plasma/Apps/plClient/plClient.h
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ class plInputController;
class plSceneObject;
class pfConsoleEngine;
class pfConsole;
class pfRemoteConsole;
class plAudioSystem;
class plVirtualCam1;
class plKey;
Expand Down Expand Up @@ -122,6 +123,7 @@ class plClient : public hsKeyedObject

pfConsoleEngine* fConsoleEngine;
pfConsole* fConsole;
pfRemoteConsole* fRemoteConsole;

bool fDone;
bool fWindowActive;
Expand Down
2 changes: 2 additions & 0 deletions Sources/Plasma/FeatureLib/pfConsole/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,15 @@ set(pfConsole_SOURCES
pfConsoleDirSrc.cpp
pfDispatchLog.cpp
pfGameConsoleCommands.cpp
pfRemoteConsole.cpp
)

set(pfConsole_HEADERS
pfConsole.h
pfConsoleCreatable.h
pfConsoleDirSrc.h
pfDispatchLog.h
pfRemoteConsole.h
)

plasma_library(pfConsole
Expand Down
3 changes: 3 additions & 0 deletions Sources/Plasma/FeatureLib/pfConsole/pfConsoleCreatable.h
Original file line number Diff line number Diff line change
Expand Up @@ -48,4 +48,7 @@ You can contact Cyan Worlds, Inc. by email [email protected]
#include "pfConsole.h"
REGISTER_CREATABLE(pfConsole);

#include "pfRemoteConsole.h"
REGISTER_CREATABLE(pfRemoteConsole);

#endif // pfConsoleCreatable_inc
206 changes: 206 additions & 0 deletions Sources/Plasma/FeatureLib/pfConsole/pfRemoteConsole.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,206 @@
/*==LICENSE==*

CyanWorlds.com Engine - MMOG client, server and tools
Copyright (C) 2011 Cyan Worlds, Inc.

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.

Additional permissions under GNU GPL version 3 section 7

If you modify this Program, or any covered work, by linking or
combining it with any of RAD Game Tools Bink SDK, Autodesk 3ds Max SDK,
NVIDIA PhysX SDK, Microsoft DirectX SDK, OpenSSL library, Independent
JPEG Group JPEG library, Microsoft Windows Media SDK, or Apple QuickTime SDK
(or a modified version of those libraries),
containing parts covered by the terms of the Bink SDK EULA, 3ds Max EULA,
PhysX SDK EULA, DirectX SDK EULA, OpenSSL and SSLeay licenses, IJG
JPEG Library README, Windows Media SDK EULA, or QuickTime SDK EULA, the
licensors of this Program grant you additional
permission to convey the resulting work. Corresponding Source for a
non-source form of such a combination shall include the source code for
the parts of OpenSSL and IJG JPEG Library used as well as that of the covered
work.

You can contact Cyan Worlds, Inc. by email [email protected]
or by snail mail at:
Cyan Worlds, Inc.
14617 N Newport Hwy
Mead, WA 99021

*==LICENSE==*/

#include <thread>
#include <windows.h>

#include "pfRemoteConsole.h"
#include "pfMessage/pfRemoteConsoleMsg.h"
#include "plgDispatch.h"
#include "plStatusLog/plStatusLog.h"

#include <pfPython/cyPythonInterface.h>

void handlePipeServer();
void SendFullBuf(HANDLE pipe, const char* buf, size_t buflen);

pfRemoteConsole::~pfRemoteConsole()
{
plgDispatch::Dispatch()->UnRegisterForExactType(pfRemoteConsoleMsg::Index(), GetKey());
}

void pfRemoteConsole::Init() {
plStatusLog::AddLineSF("plasmadbg.log", "Starting pfRemoteConsole");
fThread = std::thread(handlePipeServer);
plgDispatch::Dispatch()->RegisterForExactType(pfRemoteConsoleMsg::Index(), GetKey());
}

bool pfRemoteConsole::MsgReceive( plMessage *msg )
{
plStatusLog::AddLineSF("plasmadbg.log", "plRemoteConsole got msg");
pfRemoteConsoleMsg *cmd = pfRemoteConsoleMsg::ConvertNoRef( msg );
if (cmd) {
plStatusLog::AddLineSF("plasmadbg.log", "Msg is pfRemoteConsoleMsg");
PyObject* mymod = PythonInterface::FindModule("__main__");
PythonInterface::RunStringInteractive(cmd->GetCommand().c_str(), mymod);
std::string output;
// get the messages
PythonInterface::getOutputAndReset(&output);
cmd->GetOutput().get()->fOutputData = ST::string::from_std_string(output);
{
std::lock_guard<std::mutex> lk(cmd->GetOutput().get()->fOutputDataSetLock);
cmd->GetOutput().get()->fOutputDataSet = true;
}
cmd->GetOutput().get()->fCondvar.notify_one();

return true;
}
return hsKeyedObject::MsgReceive(msg);
}

bool canRun(ST::string& script) {
return script.contains("\n");
}

void handlePipeServer() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the person trying to get this engine running on macOS & Linux, I'd really love a cross platform way of doing this

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Part of me wonders if pnAsyncCore should learn about named pipes, then you get callbacks on the main thread for free. Then this silly worker thread and the synchronization questions go away. Maybe @zrax will have some input on this, considering the asio work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I actually gave some thought to this while writing the code. It should actually be fairly trivial to make the code cross-platform, by making a very lightweight abstraction layer between named pipes and unix sockets on linux (and you'd connect to the unix socket with socat or something when on macOS/Linux).

Regarding pnAsyncCore, I actually started by trying to add pipe support there, but was a bit daunted by how complex it looked, and since I wanted to have a quick proof of concept runnig, I quickly changed to the worker thread approach. I do agree that it'd probably be cleaner that way though. I'll try to look into it when I've fixed the remaining TODOs.

SetThreadDescription(
GetCurrentThread(),
L"PipeServer!"
);

// First, initialize the socket
DWORD pid = GetCurrentProcessId();
ST::string pipename = ST::format(R"(\\.\pipe\URU-PYTHON-{})", pid);

HANDLE pipe = CreateNamedPipeW(
pipename.to_std_wstring().c_str(),
PIPE_ACCESS_DUPLEX,
PIPE_TYPE_BYTE | PIPE_READMODE_BYTE,
1,
// 1MiB
1 * 1024 * 1024,
1 * 1024 * 1024,
0,
nullptr
);
if (pipe == nullptr) {
// TODO: Log an error.
return;
}

if (!ConnectNamedPipe(pipe, nullptr) && GetLastError() != ERROR_PIPE_CONNECTED) {
DWORD err = GetLastError();
// TODO: Log an error
CloseHandle(pipe);
return;
}


char buf;

ST::string curScript;

SendFullBuf(pipe, ">>> ", 4);

bool isMultiline = false;
while (true) {
DWORD readSize;
if (!ReadFile(pipe, &buf, sizeof(buf), &readSize, nullptr)) {
DWORD err = GetLastError();
CloseHandle(pipe);
// TODO: Log the error.
return;
}

if (readSize == 0) {
// TODO: Log that pipe is closed.
CloseHandle(pipe);
return;
}

if (buf != '\r') {
SendFullBuf(pipe, &buf, 1);
} else {
SendFullBuf(pipe, "\r\n", 2);
}

// Turn our string into ST
ST::string readData(&buf, readSize);

readData = readData.replace("\r", "\n");
curScript += readData;

if (buf != '\r') {
continue;
}

if (!isMultiline && curScript.ends_with(":\n")) {
isMultiline = true;
SendFullBuf(pipe, "... ", 4);
continue;
}

if (isMultiline && !curScript.ends_with("\n\n")) {
SendFullBuf(pipe, "... ", 4);
continue;
}

// Send message.
pfRemoteConsoleMsg *cMsg = new pfRemoteConsoleMsg(curScript);
std::shared_ptr<pfRemoteConsoleMsgOutput> output(cMsg->GetOutput());

cMsg->Send(nullptr, true);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the implementation of plDispatch::IMsgEnqueue this should actually be safe from to call from a thread, as long as the second parameter is true.

It will lock a mutex before adding it to the queue of messages to be dispatched.

If the second parameter is false, it will also immediately try to dispatch it and probably call the receiver's MsgReceive handler on the thread which is probably not safe.


// Wait for answer.
{
std::unique_lock<std::mutex> lk(output->fOutputDataSetLock);
output->fCondvar.wait(lk, [output] {return output->fOutputDataSet;});
}

ST::string finalOutput = output.get()->fOutputData.replace("\n", "\r\n");

SendFullBuf(pipe, finalOutput.c_str(), finalOutput.size());
curScript.clear();
isMultiline = false;

SendFullBuf(pipe, ">>> ", 4);
}
}

void SendFullBuf(HANDLE pipe, const char* buf, size_t buflen) {
const char* endbuf = buf + buflen;
DWORD writeSize;
while (buf < endbuf) {
WriteFile(pipe, buf, endbuf - buf, &writeSize, nullptr);
buf += writeSize;
}
}
71 changes: 71 additions & 0 deletions Sources/Plasma/FeatureLib/pfConsole/pfRemoteConsole.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
/*==LICENSE==*

CyanWorlds.com Engine - MMOG client, server and tools
Copyright (C) 2011 Cyan Worlds, Inc.

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.

Additional permissions under GNU GPL version 3 section 7

If you modify this Program, or any covered work, by linking or
combining it with any of RAD Game Tools Bink SDK, Autodesk 3ds Max SDK,
NVIDIA PhysX SDK, Microsoft DirectX SDK, OpenSSL library, Independent
JPEG Group JPEG library, Microsoft Windows Media SDK, or Apple QuickTime SDK
(or a modified version of those libraries),
containing parts covered by the terms of the Bink SDK EULA, 3ds Max EULA,
PhysX SDK EULA, DirectX SDK EULA, OpenSSL and SSLeay licenses, IJG
JPEG Library README, Windows Media SDK EULA, or QuickTime SDK EULA, the
licensors of this Program grant you additional
permission to convey the resulting work. Corresponding Source for a
non-source form of such a combination shall include the source code for
the parts of OpenSSL and IJG JPEG Library used as well as that of the covered
work.

You can contact Cyan Worlds, Inc. by email [email protected]
or by snail mail at:
Cyan Worlds, Inc.
14617 N Newport Hwy
Mead, WA 99021

*==LICENSE==*/

#ifndef pfRemoteConsole_inc
#define pfRemoteConsole_inc

#include <thread>

#include "HeadSpin.h"

#include "pnKeyedObject/hsKeyedObject.h"

class pfRemoteConsole : public hsKeyedObject
{
public:
pfRemoteConsole() {}
~pfRemoteConsole();

CLASSNAME_REGISTER(pfRemoteConsole);
GETINTERFACE_ANY(pfRemoteConsole, plReceiver);

bool MsgReceive(plMessage* msg) override;

void Init();

private:
std::thread fThread;

std::string fOutput;
};

#endif // pfRemoteConsole_inc
1 change: 1 addition & 0 deletions Sources/Plasma/FeatureLib/pfMessage/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ set(pfMessage_HEADERS
pfMarkerMsg.h
pfMessageCreatable.h
pfMovieEventMsg.h
pfRemoteConsoleMsg.h
plArmatureEffectMsg.h
plClothingMsg.h
)
Expand Down
3 changes: 3 additions & 0 deletions Sources/Plasma/FeatureLib/pfMessage/pfMessageCreatable.h
Original file line number Diff line number Diff line change
Expand Up @@ -94,4 +94,7 @@ REGISTER_CREATABLE(pfMarkerMsg);
#include "pfMovieEventMsg.h"
REGISTER_CREATABLE(pfMovieEventMsg);

#include "pfRemoteConsoleMsg.h"
REGISTER_NONCREATABLE(pfRemoteConsoleMsg);

#endif //pfMessageCreatable_inc
Loading