-
Notifications
You must be signed in to change notification settings - Fork 304
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HPCC-31143 Extend ICodeContext with getWorkflowId #18232
Conversation
https://track.hpccsystems.com/browse/HPCC-31143 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@shamser - please see comments.
@ghalliday - a couple of questions for you also.
common/thorhelper/thorcommon.hpp
Outdated
// can construct an ICodeContext that returns the workflow id. | ||
// (It cannot be done by wrapping IGlobalCodeContext because workflow id is known only when | ||
// IEclProcess::perform is called. Workflow id isn't known before that.) | ||
struct THORHELPER_API EclProcessEx : implements IEclProcess, public CInterface |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
trivial: CInterfaceOf is preferable - avoids a VMT.
common/thorhelper/thorcommon.hpp
Outdated
struct THORHELPER_API EclProcessEx : implements IEclProcess, public CInterface | ||
{ | ||
// Wraps ICodeContext and overrides getWorkflowId() method | ||
class CodeContextEx : extends IndirectCodeContext |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
style: better to be 'public' here. There may be exceptions, I think 'extends' should only be used when defining a new interface that's an extension of another, not when deriving a concrete class and/or defining a new concrete class.
common/thorhelper/thorcommon.hpp
Outdated
virtual void fail(int, const char *) { gctx->fail(0, NULL); } | ||
|
||
virtual bool isResult(const char * name, unsigned sequence) { return gctx->isResult(name, sequence); } | ||
virtual unsigned getWorkflowId() const override { return gctx->getWorkflowId(); } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
minor: the rest of the methods in GlobalCodeContextExtra would also benefit from 'override's.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think IGlobalCodeContext::getWorkflowId() is or will be actually used anywhere?
ICodeContext::getWorkflowId() will be, but I don't see anywhere that can call IGlobalCodeContext::getWorkflowId() directly, or via generated code ?
I think it must remain untouched though (not removed), because it will break pre-existing queries if it is removed (if the VMT changes).
@ghalliday - if it is never used (I removed it and definitions as a test and can't see anywhere that generates it), and it's never needed. If we can't remove it (to avoid breaking queries), can we rename it as ```
virtual unsigned deprecatedGetWorkUnitId() // no longer used, but keeping to preserve VMT integrity
?
common/thorhelper/thorcommon.hpp
Outdated
private: | ||
Owned<IEclProcess> eclProcess; | ||
public: | ||
IMPLEMENT_IINTERFACE; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
won't be needed with change to CInterfaceOf
ecl/eclagent/eclagent.cpp
Outdated
@@ -1925,7 +1925,7 @@ void EclAgent::doProcess() | |||
} | |||
{ | |||
MTIME_SECTION(queryActiveTimer(), "Process"); | |||
Owned<IEclProcess> process = loadProcess(); | |||
Owned<IEclProcess> process = new EclProcessEx(loadProcess()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
related to our previous offline discussion, I'm still not sure why IEclProcess needs to be wrapped...
where workflow is needed, eclagent calls:
EclAgent::runProcess-> WorkflowMachine::perform, which ultimately calls IEclProcess->perform in a couple of places (WorkflowMachine::performItem and WorkflowMachine::performItemParallel)
Is there any reason why they can't use a GlobalCodeContextExtra wrapper at that point, avoiding the need to wrap IEclProcess as well e.g.:
void WorkflowMachine::performItem(unsigned wfid, unsigned scheduledWfid)
{
#ifdef TRACE_WORKFLOW
if(currentWfid)
LOG(MCworkflow, "Branching from workflow item %u", currentWfid);
LOG(MCworkflow, "Performing workflow item %u", wfid);
#endif
wfidStack.append(currentWfid);
wfidStack.append(scheduledWfid);
currentWfid = wfid;
currentScheduledWfid = scheduledWfid;
timestamp_type startTime = getTimeStampNowValue();
**GlobalCodeContextExtra gctx(ctx, wfid);**
CCycleTimer timer;
**process->perform(&gctx, wfid);**
noteTiming(wfid, startTime, timer.elapsedNs());
scheduledWfid = wfidStack.popGet();
currentWfid = wfidStack.popGet();
if(currentWfid)
{
#ifdef TRACE_WORKFLOW
LOG(MCworkflow, "Returning to workflow item %u", currentWfid);
#endif
}
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that would work equally well, and probably simpler.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a potential danger if any function does not use this context that the calling code will not get the workflow id. E.g. The IcodeContext used within a graph.
Combined with your other comment about IGlobalCodeContext::getWorkflowId(), I wonder whether it should only appear in the IGlobalCodeContext. That would still be accessible from the dfu functions if they indicated they required it.
roxie/ccd/ccdcontext.cpp
Outdated
@@ -2876,7 +2876,7 @@ class CRoxieServerContext : public CRoxieContextBase, implements IRoxieServerCon | |||
MTIME_SECTION(myTimer, "Process"); | |||
QueryTerminationCleanup threadCleanup(true); | |||
EclProcessFactory pf = (EclProcessFactory) factory->queryDll()->getEntry("createProcess"); | |||
Owned<IEclProcess> p = pf(); | |||
Owned<IEclProcess> p = new EclProcessEx(pf()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
see comment/question in eclagent.cpp re. whether IEclProcess wrapper needed
roxie/ccd/ccdcontext.cpp
Outdated
@@ -4029,7 +4032,7 @@ class CSoapRoxieServerContext : public CRoxieServerContext | |||
virtual void process() | |||
{ | |||
EclProcessFactory pf = (EclProcessFactory) factory->queryDll()->getEntry("createProcess"); | |||
Owned<IEclProcess> p = pf(); | |||
Owned<IEclProcess> p = new EclProcessEx(pf()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
see comment/question in eclagent.cpp re. whether IEclProcess wrapper needed
rtl/include/eclhelper.hpp
Outdated
@@ -727,6 +727,7 @@ interface ICodeContext : public IResourceContext | |||
virtual char *getPlatform() = 0; // caller frees return string. | |||
virtual unsigned getPriority() const = 0; | |||
virtual char *getWuid() = 0; // caller frees return string. | |||
virtual unsigned getWorkflowId() const = 0; // Note: don't use yet as it has not been fully implemented in all derived classes |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you need to place any new virtuals at end of interface.
This will allow existing queries compiled with previous version to continue to work without recompilation, if methods are placed in the middle, the VMT offsets of methods past it will change and cause problems.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it needs to be at the end of the interface, and it cannot match the name of an existing virtual either.
rtl/include/eclhelper.hpp
Outdated
@@ -2954,7 +2955,7 @@ struct IGlobalCodeContext | |||
virtual void fail(int, const char *) = 0; | |||
|
|||
virtual bool isResult(const char * name, unsigned sequence) = 0; | |||
virtual unsigned getWorkflowId() = 0; | |||
virtual unsigned getWorkflowId() const = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
related to comment in thorcommon.hpp.
Can we rename it to avoid confusion? If not, is changing it's prototype from non-const to const problematic (does it affect VMT alignment of other members. I don't think so, but would like @ghalliday to comment on this and previous comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IGlobalCodeContext had getWorkflowId() defined before this PR. I added getWorkflowId() to ICodeContext in this PR. Some classes such as EclAgent derive from both IGlobalCodeContext and ICodeContext. Would it make sense to have both const and non-const defined in the classes such as EclAgent?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, I realize it was already there, but now with a new method in ICodeContext, the pre-existing IGlobalCodeContext version (especially if never used) adds to the confusion.
Would it make sense to have both const and non-const defined in the classes such as EclAgent?
I don't think it would be a good idea to have a const and non-const version, that would be more confusing.
I think IGlobalCodeContext ::getWorkflowid() could be deprecated.
[ see other comment : https://github.com//pull/18232#discussion_r1457347909 ]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think changing from const to non const will cause any problems. The danger is with failing to load workunit dlls that have references to old base definitions, but that should not happen with anything within IGlobalCodeContext.
IGlobalCodeContext ::getWorkflowid() was deprecated because it was incompatible with the parallel workflow execution. This PR would have probably been a better way to get around that problem...
common/workunit/CMakeLists.txt
Outdated
@@ -77,6 +77,7 @@ if(NOT PLUGIN) | |||
nbcd | |||
eclrtl | |||
deftype | |||
thorhelper |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this can be avoided, better not to propagate the engine library thorhelper into workunit if it can be avoided.
This is being included because of IndirectCodeContext ?
Apart from 2 virtuals it actually implements differently, a vanilla IndirectCodeContext could be moved out of thorhelper.
And the version that is needed by the engines (by thorcommon) could derive from the generic IndirectCodeContext and overload the 2 methods.
d9f4dff
to
3226e7c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@shamser one main comment about the location of the class implementations - conclusion to move them to another file built in the same dll, and a couple of other observations.
Looks close to being ready to merge.
rtl/include/eclhelper.hpp
Outdated
}; | ||
|
||
class IndirectCodeContext : implements ICodeContext |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These classes need to be in a different file - this is used from generated code, and should contain the minimum possible - especially no implementations.
There is also a second danger of including class implementations in a header. If a compiler only sees one concrete implementation of a class it will add code to de-virtualise the function calls (possibly only if it has an inline function definition, I have forgotten!). That does not work well if the implementation is in a different dll, and also is counter-productive if it is not the most common implementation.
Having said that it isn't clear where this class should go if it is being called from the workflow. I think the correct answer may be that the code in workflow.cpp needs to split in two, the workflowMachine should be moved to throhelper, and this should live in thorhelper as it did originally.
A less painful solution would be to add a new header/c++ file to eclrtl, but move it out of this file.
rtl/include/eclhelper.hpp
Outdated
}; | ||
|
||
class IndirectCodeContext : implements ICodeContext |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs a export qualifier otherwise you may get link problem in windows, or with release versions.
class ECLRTL_API IndirectCodeContext : implements ICodeContext
rtl/include/eclhelper.hpp
Outdated
virtual unsigned getWorkflowId() const override { return wfid; } | ||
}; | ||
IGlobalCodeContext * gctx; | ||
CodeContextEx * codeContextEx; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor. This could be a member object rather than a pointer, avoiding the need for the new/delete.
rtl/include/eclhelper.hpp
Outdated
unsigned wfid; | ||
public: | ||
CodeContextEx(ICodeContext * _ctx, unsigned _wfid) : IndirectCodeContext(_ctx), wfid(_wfid) {} | ||
virtual ~CodeContextEx() {}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not needed?
@shamser The windows-2022 build is failing:
I am not sure why exactly. From the errors, it would appear that the symbols are not being exported from the eclrtl lib, but the DECL spec looks correct afaics. @ghalliday - may have some insight. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@shamser - looks good, some v. minor issues, and need to workout why it's failing to compile in windows-2022
rtl/eclrtl/wfcontext.hpp
Outdated
unsigned wfid; | ||
public: | ||
CodeContextEx(ICodeContext * _ctx, unsigned _wfid) : IndirectCodeContext(_ctx), wfid(_wfid) {} | ||
virtual ~CodeContextEx() {}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still not needed afaics.
protected: | ||
ICodeContext * ctx; | ||
}; | ||
class ECLRTL_API GlobalCodeContextExtra : implements IGlobalCodeContext |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
trivial/formatting: normally we have a newline between class declarations.
rtl/eclrtl/wfcontext.hpp
Outdated
virtual unsigned getWorkflowId() const override { return wfid; } | ||
}; | ||
IGlobalCodeContext * gctx; | ||
CodeContextEx codeContextEx; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
personal: could have named instance at end of class definition, i.e.:
virtual unsigned getWorkflowId() const override { return wfid; }
} codeContextEx;
rtl/eclrtl/wfcontext.hpp
Outdated
virtual void setResultDataset(const char * name, unsigned sequence, size32_t len, const void *val, unsigned numRows, bool extend) override { gctx->setResultDataset(name, sequence, len, val, numRows, extend); } | ||
virtual void getEventName(size32_t & outLen, char * & outStr) override { gctx->getEventName(outLen, outStr); } | ||
virtual void getEventExtra(size32_t & outLen, char * & outStr, const char * tag) override { gctx->getEventExtra(outLen, outStr, tag); } | ||
virtual void doNotify(char const * name, char const * text, const char * target) override{ gctx->doNotify(name, text, target); } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
trivial/formatting: missing space after override
I'm not convinced that GlobalCodeContextExtra and IndirectCodeContext is in the right place. sourcedoc.xml in eclrtl states: |
66b58a6
to
de52c75
Compare
@shamser @jakesmith |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As for the link problem, I have a personal dislike of classes with lots of inline functions, especially in headers. In this case the functions would be duplicated into every dll that uses them.
Having said that I'm not 100% sure why you are getting the link error. A fix would be to
i) define a cpp file within eclrtl that includes that header
ii) add ECLRTL_API to the class definitions in the header.
iii) imho also move the implementation of those virtuals into the c++ file. Although just doing the first 2 steps may fix it.
rtl/eclrtl/wfcontext.hpp
Outdated
public: | ||
GlobalCodeContextExtra(IGlobalCodeContext * _gctx, unsigned _wfid) : codeContextEx(_gctx->queryCodeContext(), _wfid), gctx(_gctx) {} | ||
virtual ICodeContext * queryCodeContext() override { return &codeContextEx; } | ||
virtual void fail(int, const char *) override { gctx->fail(0, NULL); } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not passing the parameters through correctly.
Looks like wfcontext.cpp is currently missing from the branch. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@shamser - commenting mostly, because I think you're still refactoring, but I'm tagged for review.
69482df
to
bfcc162
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@shamser - a few minor comments
process->perform(this, 0); | ||
{ | ||
GlobalCodeContextExtra gctx(this, 0); | ||
process->perform(&gctx, 0); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder whether if would be better to pass 'this' (as before), and have the EclAgent::getWorkflowId() method return 0;
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This would improve the code at best minimally. I'd prefer to leave as is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Am not concerned it is inefficient or problematic as it stands.
It is more a question of under what circumstances can this branch be used, and when it is, does it make sense for ICodeContext::getWorkflowId() to return 0 (and what does that mean?), or should it never be called in which case the EclAgent::getWorkflowId() version that returns throwUnexpected() is more correct.
I am not sure how or when workflow does not exist, so the point is probably moot anyway.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suspect this could be changed so that it throws an error if it does not have workflow. I think it has always been present for the past 15+ years.
@@ -567,6 +567,7 @@ class CRoxieAgentActivity : implements CInterfaceOf<IRoxieAgentActivity>, implem | |||
// Not yet thought about these.... | |||
|
|||
virtual char *getWuid() { throwUnexpected(); } // caller frees return string. | |||
virtual unsigned getWorkflowId() const override { throwUnexpected(); } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
trivial: in fact expected, but unimplemented at the moment? (maybe change to UNIMPLEMENTED_X("getWorkflowId"); )
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This isn't missing functionality. This getWorkflowId shouldn't be called. I think it should remain as throwUnexpected.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since it's not used anyway yet, it doesn't matter anyway,
but do still wonder if could be via CRoxieAgentActivity::onCreate()
basehelper->onCreate(this, NULL, &serializedCreate);
where this is the ICodeContext of CRoxieAgentActivity, where this is the implemented member.
Anyway, if it is an issue it can be revisited in https://track.hpccsystems.com/browse/HPCC-31154.
Extend ICodeContext with a new method called getWorkflowId. Implement wrapper for IEclProcess so that when IEclProcess::perform is called, 1) the ::perform passes the call through to the original IEclProcess perform but with a new IGlobaCodeContext. All other methods are passed through to the original IEclProcess. 2) this new IGlobalCodeContext wraps the orignal IGlobalCodeContext to call the original IGlobalCodeContext method but with a new queryCodeContext method implmentnation. The new queryCodeContext returns a new ICodeContext. 3) This new ICodeContext is the original ICodeContext wrapped with a class that overrides the getWorkflowId to return the workflow id. Signed-off-by: Shamser Ahmed <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to merge to me.
Extend ICodeContext with a new method called getWorkflowId. Implement wrapper for IEclProcess so that when IEclProcess::perform is called, 1) the ::perform passes the call through to the original IEclProcess perform but with a new IGlobaCodeContext. All other methods are passed through to the original IEclProcess.
2) this new IGlobalCodeContext wraps the orignal IGlobalCodeContext to call the original IGlobalCodeContext method but with a new queryCodeContext method implmentnation. The new queryCodeContext returns a new ICodeContext.
3) This new ICodeContext is the original ICodeContext wrapped with a class that overrides the getWorkflowId to return the workflow id.
Type of change:
Checklist:
Smoketest:
Testing: