feat: write STDOUT log to file

[jdhoffa]

No description provided.

[cjyetman]

logger is now used rather extensively throughout the script (#76), which I think resolve this issue @jdhoffa?

[jdhoffa]

I still think capturing that output in a log file could be useful?

Could be done very easily by piping the STDOUT to a log file in the final Docker RUN

[jdhoffa]

Gonna again divert to @AlexAxthelm as he has a much better idea of good DevOps practices here

I think in development we want the logs in command line, and in production we would want them piped to a log file

EDIT: oops I thought this was transition.monitor, didn't realize this is data.preparation. Still might be relevant as we move towards a more stable containerized data prep process

[cjyetman]

@AlexAxthelm any ideas on piping logger messages to a file?

[jdhoffa]

Judging by a comment here: RMI-PACTA/workflow.pacta#47

It sounds like this is the typical way:

# add appender so that reprex captures the log output
logger::log_appender(logger::appender_stdout, index = 2)

# add file outputs to capture logs
logger::log_appender(logger::appender_file("foo.log"), index = 3)

# and because this is a demo, add one that captures in different level and format
logger::log_appender(logger::appender_file("bar.json"), index = 4)

[cjyetman]

ok, does someone want to implement that here, because I don't fully understand it and can't really test it?

[jdhoffa]

@AlexAxthelm do you think this is a useful feat here?

If you don't and @cjyetman doesn't, then I'm happy to just close it.

[AlexAxthelm]

I think grabbing logs as part of the output is a reasonable idea (not as a required file). Assigned to self, and it's on my list after I finish up some of the pacta.workflow.utils and webapp work.

but the short answer is that writing to file like @jdhoffa laid out is probably what I'll implement.

[jdhoffa]

Sweet!

I already gave it a quick go, but figured we should define some wat to identify the timestamp for the log
e.g. paste0(system_timestamp, ".log")

and then I thought: where should this be saved? in the same directory as the rest of the outputs? do we want to allow saving of multiple logs run on the same day, or overwrite the log of a single run? do we want it in the manifest?

and then I gave up lol. But it shouldn't be too hard to implement

[AlexAxthelm]

My plan is to dump it in the outputs along with everything else as workflow-data-preparation.log or something like that. Putting it in the outputs directory should keep everything separate between runs, but tie the logs to the outputs.

Do we want to consider writing logs centrally to a single store? I think capturing stdout is a better mechanism for that

[cjyetman]

This is becoming a security concern for me.

[AlexAxthelm]

@cjyetman Can you clarify what you mean by that? Are you worried about logging information we shouldn't be? or that logs will be exposed when they shouldn't be?

[cjyetman]

I think I was confused, thinking this was the workflow.tm repo, and thought combining the strategies of log-everything-possible and save-them-in-the-most-easily-accessible-public-facing-location-possible seems like a risk. But for a process that is never publicly accessible, this makes sense.