Huge performance improvement when dealing with Lua Modules

[serasset]

Hi, I am using bliki for a research project called DBnary. For this project, I extract Lexical data from 21 editions of wiktionary. I use bliki to parse and transcode definitions and morphological information.

When auditing my project, I saw that bliki add a huge impact on performance, and especially when using Lua modules.

I found that I was spending most of my time initializing ScribuntoLuaEngine instances (which is done for every #invoke to a Lua function by bliki).

In bliki, a ScribuntoLuaEngine is created for each invocation. Creation of such an engine takes a lot of time.

I wanted to reuse a scribuntoLuaEngine for the whole extraction process (to handle thousands of pages) rather than creating a new one for each invocation or for each page. This did not initially work because Scribunto assumes it is initialized for each query, i.e. it is initialized once for each page. In fact, this assumption is used only in mw.title.lua, where the getPageName function is created using the pagename passed at initialisation time.

I tried to inject code that will void these assumptions, but it was not possible, then I patched the mw.title.lua (I am assuming that this part of scribunto code is now very stable).

With this patched bliki version, I was able to reuse the ScribuntoLuaEngine instance accross pages and cut down the extraction time for english from 3 hours to 30 minutes. The only caveat is that template calls should not be cached (or the cache should be emptied when setPageName is called.

If you want to have a look at the impact on your code, please have a look to https://bitbucket.org/serasset/info.bliki.wiki/branch/bump-for-dbnary

If you are interested in this improvement, I can help reintegrating it to bliki.

Regards,

Gilles Sérasset,

[axkr]

Could you please create a pull request.
I think the goal should be to have one ScribuntoLuaEngine per thread and to associate this instance with the thread with a threadlocal mechanism:

• https://www.baeldung.com/java-threadlocal

[serasset]

SEE #13

[axkr]

@Serraset since I'm not a Lua expert and @jberkel doesn't seem to be available, can you add the PRs by yourself?
I've given you access to the repository.

[serasset]

Thanks, however, I need a decision from your part:

• should we create a set of new classes (reusable wiki model, configuration ang Lua engine), along the original ones
• OR should we integrate the reusable behavior into the existing wiki model, lua engine and configuration?

The first solution has been implemented, allows the user to chose between the original and previous behavior, but adds more classes that should give the same results.

I’ll try to quickly implement the second solution and launch all tests as is to validate the approach in existing code.

I implemented the first approach as I originally doubted the change would be accepted, but second one would be easier for the users.

[axkr]

If possible, integrate your reusable behavior model into the existing wiki model.

[jberkel]

@serasset thanks for your work on this. I stopped using bliki for my own project, mainly because of constant compatibility issues and the inherent slowness.

However I'm a bit sceptical that mw.title is the only part of the lua code which assumes that it is being executed in a new instance each time., so there might be some other unexpected behaviour lurking. If users can switch it on/off that should be fine, though.

[serasset]

@jberkel I read all the Lua modules in Scribunto and this is the only place where a value is passed at initialisation and kept for further calls.

But you are right, as in bliki, if a caching mecanism is used, it may break the change, but I seriously doubt it.

There are part of bliki that are very fragile, with a lot of statefull information (kept in the WikiModel instance), appart from the cache mecanism (e.g. ToC tags, number of links, ...). These are initialized by 2 methods setUp() and initialize() and may lead to inconsistant states if a user overrides a behaviour without being aware that setUp should be call before all rendering. I decided to call setUp() when the page name and/or namespace is changed.

I'm using bliki to transclude some parts of 21 language editions of wiktionaries. Its not a complete transclusion (far too long), but I transclude at least definitions and morphology in 5 languages. I am able to evaluate the difference this change will make to these parts.

I'm still hoping to find a better implementation (or craft one based on another WikiText parser of my own), but the work put in bliki is very impressive and it will take far too long to approach its potential.

I was already reusing the same wiki model for many pages without too much problems until now (it also takes a long time to initialize), but reusing the scribunto engine cuts the time by a factor of 5 to 10 depending on the languages.