logging

[ptabandzelic]

Hey, would u recommend anything for logging.How to configure?How to use inside tasks?Should i create separate logger other than on the spark driver?Any help or advice would be helpfull.
Thanks

[renardeinside]

hi @ptabandzelic ,

logging object is already provided by the abstract class common.Task. You can use it as self.logger.info("...") or other self.logger.* methods.

Logs are visible in the Driver Logs section in the UI. Depending on your cloud and setup, it would make sense to export logs to Azure Log Analytics or AWS CloudWatch, or solutions like DataDog etc. Read more on the logging topic here.

[renardeinside]

Its an automated(job) cluster.

In this case you can configure logging properties and parameters by using the conf/deployment.yml.

Add the following lines to the cluster configuration:

cluster_log_conf:
    dbfs:
        destination: "dbfs:/some/logs/path"

[ptabandzelic]

Why i cant use file reference like this and locally make configuration file?

cluster_log_conf:
  dbfs:
    destination: "file://conf/log4j.properties"

[renardeinside]

Because it's different files that control different behaviour.

cluster_log_conf:
  dbfs:
    destination: "dbfs:/some/logs/path"

This setting configures where will be logs saved (printed/outputted).

This file:

"file://conf/log4j.properties"

Is configuring how the logs will be formatted. Therefore this file needs to be provided separately, and set via init script. Please read the blogpost above carefully for details.

[ptabandzelic]

Hm, lets be on the same page.
So i want to configure my automated(job) cluster logging. From my understanding i will have to do that with init_scripts inside deployment file like here: https://kb.databricks.com/clusters/overwrite-log4j-logs.html
So i can provide my init script, locally and file reference it or upload it to dbfs, does not matter. It should just be in deployment file cluster configuration under init_scripts.
What is the point of cluster_log_conf?

[renardeinside]

What is the point of cluster_log_conf?

Cluster log conf defines where the logs will be written (try it out - start a job cluster with this setting, then verify the content of this path - you'll see there text files with logs.