Log information from Lua Filters

[estedeahora]

It would be interesting if the log file (--log=file) could store the output of the lua filters generated by the error and io.write functions. Currently it only saves the output of warn. In the related discussion, @jgm and @tarleb suggested "provide a report function" to address the problem.

Thanks in advance for the work.

Discussed in #9659

^{Originally posted by estedeahora April 14, 2024}
I am using pandoc with a set of custom lua filters. In the call to pandoc I use '--log=file.log', but I am interested in this log file incorporating information arising from the lua filters. In the lua filter I display this information in the console via io.write or error.

Is there any way to incorporate this directly into the 'file.log'?

Thanks!

[jgm]

If the output of warn currently does go into the log file, then I don't see what more is needed. We certainly don't want io.write to go to the log file in general. And error should create an error, not a warning (which is what the log file contains).

[tarleb]

My thought process when I suggested to open this issue was that warn can only produce warnings, but not info entries. What I had in mind was to allow something like this:

local orig_io_write = io.write
io.write = function (...)
  pandoc.report('This is an info', 'INFO')
  return orig_io_write(...)
end

I.e., allow users to redefine io.write such that additional info gets logged with severity INFO.

I believe that giving users more access to the logging system would generally be a good idea. But it's difficult to make it useful without just tying it to the internals, as those should probably remain hidden.

A suitable API design might be achievable, but I don't have one yet.

[jgm]

When they use warn currently, what kind of LogMessage is generated? Is there a special log message for warnings from Lua filters?

[estedeahora]

I understand @jgm's concern about not making pandoc too complex. But as @tarleb points out, I think it would be useful to have messages at the 'info' level (which is why I use io.write). In my particular use case, I use io.write to notify the user of some decisions that occur during filter execution, but which are not warnings. I understand that it would be very useful to have a pandoc.report function like the one proposed by @tarleb, that allows to make pandoc.report('This is an info', INFO') / pandoc.report('This is a warning', 'WARNING'), independently from what the final user is informed. Then each filter could customise it to print or not print console output (this way you don't interfere with the behaviour of io.write which can have other uses).

I want to clarify a few points regarding the behaviour of warn. On one hand, the @on / @off option, does not work as the first parameter of warn (i.e. warn('@on', 'my message'), but by setting the warn('@on') behaviour, and then using warn('my message') again with the message. On the other hand, even when warn('@off') is set, the function returns a message in the console (which can be intimidating to an 'ordinary' user). In fact what '@off' does is suppress 'clean' console messages. It is because of these limitations of warm`, that I would be interested in having another way of passing information to the user.

Thank you very much for the comments, they really help me also to clarify what is the behaviour I am asking for.

[tarleb]

When they use warn currently, what kind of LogMessage is generated? Is there a special log message for warnings from Lua filters?

Yes, pandoc generates a ScriptingWarning message, which was introduced specifically for that purpose. The relevant PR was #8685.

[tarleb]

The expectation is that users will use --silent if they don't want pandoc's warning messages to be printed in the terminal. However, setting warn '@on' will make sure that the warn message always gets printed to the terminal, as happens in the default lua executable. But I agree that it's confusing to see the same message twice, one time as-is, and another time with info about the location where warn was involved. There might be a better way.

[tarleb]

The following might be a reasonable design:

New module pandoc.lua with three functions:

• pandoc.log.warn -- like warn, but without support for control messages
• pandoc.log.info -- like pandoc.log.warn, but raises a ScriptingInfo log message (requires an API change)
• pandoc.log.silence -- applies a function, but returns the log messages generated by the call as the first result instead of logging them in the main log. The function's other results are returned as additional return values.

It would be enough to resolve this issues as well as #9077.

[tarleb]

I've creates #9668 for experimentation and as a proof of concept.