String concatenation memory leak #389
Comments
|
Until a deeper investigation, as a memo: concatenation is done here: I suppose the "memory leak" (string heap increase) is more expressed when you have so many additions. The function VSprintf always makes a copy of the new values atop the string heap as seen here: https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/core/avisynth.cpp#L1504 First "string"+"string", then "stringstring" + "string", then "stringstringstring" + "string", etc. are stored. The above example contains 104 elements, 6, 12, 18, 24, ... 624 bytes (over 32000 bytes plus overhead) are stored on the heap after the additions, and it seems that this is done in each function call during the function body evaluation. |
|
Random idea: add a new Internal String Function like eg. |
|
Oh Memory usage seems to depend on how you use it... |
|
Yep, it depends on when strings are pushed on the (always incrementing) heap. In the second - small leak - case the string constants already exist they are probably evaluated once, during function instantiating. There is one single result (cat=) that must be pushed on the heap (and the 8*13 "string" of course). Inside "Format" there is no extra memory penalty since the temporary results are c++ std::string and not real Avisynth "AVSValue". The first - big leak - one is pushing each intermediate result on the string heap, since each evaluation is a new AVSValue which must be stored. I had a deeper look into the parser, but I could not find a way how to recognise when multiple string constants are added together and replace them with a single multi-addition. |
|
How come math operations like I thought at the end of the ScriptClip all local variables would be cleared anyway since they are no longer accessible at the next frame (which is why I have to make all my runtime vars globals so they don't disappear at the next frame). In any case I think using Format to join strings is a good enough solution and maybe it's not worth pulling apart the Avisynth core to try and get the root cause, unless you feel motivated enough to do it that is. |
|
Strings are pointers to buffers. Avisynth does not know if a pointer points to a buffer that exists forever (e.g. static constant in program code) or just exists in a temporary buffer. To be sure, Avisynth always has to make a safe copy from all strings that would be later used in script. Avisynth cannot determine by just a pointer if the specific memory area is volatile or not. The heap (better said: string store) is ever increasing. If this happens in runtime, with significant amount of new strings per frame than we'll see a more serious memory consumption. |
|
The recent days I experienced this behavior. I had to store Max(RplaneMax, GPlaneMax, BPlaneMax) (1) for every frame and then sort the values ascending. The easiest way to sort values I found is |
|
I experimented with a string cache, which helps in the topic starter script. Before After The speed is a bit slower in this specific test. The extra check whether the strings are in the cache, require a little more time. Despite of this example is an artificial one, this solution probably hugely helps in such cases when most of the strings in ScriptClip are repeatable. |
|
Test build, x64, no commit yet, I'd just like to see how it is working on your side in your usual workflow. |
I can report the synthetic test in the OP is resolved on my end. As far as my real world usage goes the issue persists. Possibly as I'm concatenating in a different way inside my ScriptClips, eg. |
|
Yes, in cases, where the string is built from formatted numbers, which are probably different for each frame, then my method would not help (unless integers 16-240 are involved, because they form a finite number of the possible strings). This change does not do any harm otherwise, and sometimes it helps a lot, so I'm gonna keep this mod. Thanks, I don't need more demo scripts, I see the problem of the other use cases. |
|
Anyway, I wonder if Asd-g's 24 GB@70000 frames has been decreased a bit or not. |
|
No change. Just for info:
|
|
Hi, downloaded the sample, despite the zip file name "Sony Camping In Nature HDR 4K Demo.zip", inside there is a "Sony Camp 4K Demo.mp4" file instead of "Sony 4K HDR Demo - Camping in Nature.mp4" you are using in the script. |
|
Sorry, from here you can download This is the ffms2 version I use. |
|
Thanks, till then, as a workaround: https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/script_ref/script_ref_arrays.html#arraysort |
|
Thanks, I will test it. |
As referenced here
The text was updated successfully, but these errors were encountered: