Python includes a standard logging API that provides all the basic functionality you usually need for logging information about your application. For the most part, I’ve implemented the API as follows:
Step 1: Do the standard imports
Not only do I import the logging package, I also import the os package to map the full path to my log file and the uuid package.
import logging
import uuid
import os
Step 2: Set up some global variables
I usually set up three global variables that I use for logging: current_dir, log_id, and extra. To provide the logging API a full path to my log file, I create a current_dir string that represents the full path to the current directory where my program is running.
Often, after my program has been running for a few weeks, I like to download the log file and gather different metrics on the program. One metric I’m always interested in is how long it takes for my program to run to perform its task (for programs that perform ETL tasks and the like) and is the script speeding up, slowing down, or running about the same over time. The way I do this is by generating a semi-unique value for every time the program runs. I include this unique value–I call it log_id–in every log entry. When I do my analysis, I can group by this log id, easily get the start and end times of the script, calculate the total run time per run, and determine how my script has been doing over time. The easy way to include that log_id in my log entries is to add my own user-defined LogRecord attribute. I do this by creating a dictionary called extra with my log_id key/value pair.
current_dir = os.path.dirname(os.path.realpath(__file__))
log_id = str(uuid.uuid4())[-12:]
extra = {'log_id': log_id}
Step 3: Create the configuration
Next, I create my configuration by setting the filename to the full path of my log file, my preferred date/time format, the format of the log file itself, and the minimum logging level to log. Traditionally, I’ve always just set up my configuration in the code.
logging.basicConfig(filename=current_dir + '/logger_example.log', datefmt='%Y-%m-%d %H:%M:%S', format='%(asctime)s|%(log_id)s|%(levelname)s|%(message)s', level=logging.INFO)
Step 4: Write your log events
Finally, I can start writing my log events. Since I’m including a user-defined LogRecord attribute, I have to always make sure to include the “extra” argument and pass my extra dictionary to it.
logging.debug('this is a degug statement', extra=extra)
logging.info('Did something', extra=extra)
A better way to do this
So, that approach to logging is fine, but I’d like to improve upon it in at least two ways:
- I’d like to move my configuration out to a separate properties file so that I can more easily change aspects of the logging configuration, especially the logging level and
- I’d like to implement rotating logs so that I can more easily manage the growth of my log files.
I’ve been able to achieve both goals by improving my code as follows:
Improvement Step 1: import logging.config
The first step in moving my logging configuration outside of my code and into a properties file is by importing logging.config:
import logging.config
Improvement Step 2: reference my configuration file
Next, I have to point the logging API to my logging configuration file. In this example, my configuration file is in the same directory as my Python script, so I don’t need to provide a full path to the file:
logging.config.fileConfig('logger_cfg_example.cfg')
Improvement Step 3: Setup my configuration file
Now that I’ve referenced my configuration file, I actually need to set it up.
[loggers]
keys=root
[handlers]
keys=fileHandler
[formatters]
keys=pipeFormatter
[logger_root]
level=INFO
handlers=fileHandler
[handler_fileHandler]
class=handlers.RotatingFileHandler
level=NOTSET
formatter=pipeFormatter
args=('logger_cfg_example.log', 'w', 2000, 3)
[formatter_pipeFormatter]
format=%(asctime)s|%(log_id)s|%(levelname)s|%(message)s
datefmt=
class=logging.Formatter
Check out the documentation to learn more about the configuration file format, what sections are required, and so forth. I’ve highlighted five lines of this configuration file to point out five interesting features:
- In this example, I’ve set the logging level to INFO but I can now easily change that to any level I wish by editing this file.
- I’m now able to achieve rotating logs by instructing Python to use the handlers.RotatingHandler class.
- The RotatingHandler class takes four arguments and I can easily pass those to the class in my configuration file with the “args” key. Here, I’m telling the class to write logs to the file logger_cfg_example.log, open that file for “writing”, rotate the file every 2000 bytes, and only keep three archived log files. Note that log size argument is in bytes. In practice, you’d probably want to roll your log file after so many megabytes. For my testing purposes, I’m just rolling my log file after 2 kilobytes.
- I can now move my log file format to my configuration file by setting the “format” key. Note that I can still include my user-defined attribute, log_id.
- Finally, reading the documentation, I discovered that the default date/time format basically matches the format I use most often, so I’ve opted to leave that formatting blank.
Improvement Step 4: Nothing else, just log
With that configuration file, my code looks a little cleaner. I can go ahead and log like normal:
logging.debug('this is a degug statement', extra=extra)
logging.info('Did something', extra=extra)
Conclusion
So, going forward, I will start taking advantage of Python’s rotating log file capabilities and the configuration file option. Check out my github project for the full code examples. That’s not quite the end of the story, though. Recently, I was listening to a Python Bytes podcast where the hosts were discussing a Python package called loguru. The hosts seemed pretty excited about the different features of the API. The Python community has authored other logging packages, as well. Clearly, people have found issue with the core API such that they’ve spent time crafting alternatives. Some day, I should explore some of these alternatives and if they’re worth making a change.
Recent Comments