Python for Spark Tutorial - Logging in Python

Once you put your applications into production, you won’t be able to debug them any more. This is creating some issues, since you won’t know what is going on in the background. Imagine, a user does something and an application occurs – maybe, you don’t even know that this behaviour can lead to an error. To overcome this obstacle, we have a powerful tool in almost any programming environment: logging in Python.

How to do logging in Python

Basically, the logger is imported from “logger” and it is used as a singleton. This means that you don’t need to create any classes or alike. Basically, first you need to instruct the logger with some information – such as the path to store the logs in and the format to be used. In our sample, we will use these parameters:

filename: The name of the file to write to
filemode: how the file should be created or appended
format: how the logs should be written into the file (regular expressions, …)

Then, you can call different logging levels to the logger. This is done by simply typing “logger” and using the action:

logger.<<ACTION>>

Basically, we use these actions:

debug: a debug message that something was executed, …
info: some information that a new routine or alike is started
warning: something didn’t work as expected, but no error occurred
error: a severe error occurred that lead to wrong behaviour of the program
exception: an exception occurred. It is logged as “error” but in addition it includes the error message

Now, let’s start with the logging configuration:

import logging
logging.basicConfig(filename="../data/logs/log.log", filemode="w", format="%(asctime)s - %(levelname)s - %(message)s")

We store the log itself in a directory that first needs to be created. Then, we provide a format with the time, the name of the level (e.g. INFO) and the message itself. Now, we can go into writing the log itself:

logging.debug("Application started")
logging.warning("The user did an unexpected click")
logging.info("Ok, all is fine (still!)")
logging.error("Now it has crashed ... ")

This creates some log information into the file. Now, let’s see how this works with exceptions. Basically, we “provoke” an exception and log it with “exception”. We also set the parameter “exc_info” to true, which includes the exception without passing it on explicitly (Python handles that for us :))

Logging exceptions in Python

try:
    4/0
except ZeroDivisionError as ze:
    logging.exception("oh no!", exc_info=True)

Now, we can review our file and the output should be like this:

2019-08-13 16:21:04,329 - WARNING - The user did an unexpected click
2019-08-13 16:21:04,889 - ERROR - Now it has crashed ...
2019-08-13 16:21:05,461 - ERROR - oh no!
Traceback (most recent call last):
  File "<ipython-input-9-5d33bb8d3dd6>", line 2, in <module>
    4/0
ZeroDivisionError: division by zero

As you can see, logging is really straight-forward and easy to use in Python. So, no more excuses to not do it :). Have fun logging!

If you are not yet familiar with Spark, have a look at the Spark Tutorial i created here. Also, I will create more tutorials on Python and Machine Learning in the future, so make sure to check back often to the Big Data & Data Science tutorial overview. I hope you liked this tutorial. If you have any suggestions and what to improve, please feel free to get in touch with me! If you want to learn more about Python, I also recommend you the official page.