- log rotation every day: Use a TimedRotatingFileHandler
- compression of logs: Set the
encoding='bz2' parameter. (Note this "trick" will only work for Python2. 'bz2' is no longer considered an encoding in Python3.)
- optional - delete oldest log file to preserve X MB of free space.
You could (indirectly) arrange this using a RotatingFileHandler. By setting the
maxBytes parameter, the log file will rollover when it reaches a certain size. By setting the
backupCount parameter, you can control how many rollovers are kept. The two parameters together allow you to control the maximum space consumed by the log files. You could probably subclass the
TimeRotatingFileHandler to incorporate this behavior into it as well.
Just for fun, here is how you could subclass
TimeRotatingFileHandler. When you run the script below, it will write log files to
With a small value for
time.sleep (such as 0.1), the log files fill up quickly, reach the maxBytes limit, and are then rolled over.
With a large
time.sleep (such as 1.0), the log files fill up slowly, the maxBytes limit is not reached, but they roll over anyway when the timed interval (of 10 seconds) is reached.
All the code below comes from logging/handlers.py. I simply meshed TimeRotatingFileHandler with RotatingFileHandler in the most straight-forward way possible.
import logging.handlers as handlers
Handler for logging to a set of files, which switches from one file
to the next when the current file reaches a certain size, or at certain
def __init__(self, filename, mode='a', maxBytes=0, backupCount=0, encoding=None,
delay=0, when='h', interval=1, utc=False):
# If rotation/rollover is wanted, it doesn't make sense to use another
# mode. If for example 'w' were specified, then if there were multiple
# runs of the calling application, the logs from previous runs would be
# lost if the 'w' is respected, because the log file would be truncated
# on each run.
if maxBytes > 0:
mode = 'a'
self, filename, when, interval, backupCount, encoding, delay, utc)
self.maxBytes = maxBytes
def shouldRollover(self, record):
Determine if rollover should occur.
Basically, see if the supplied record would cause the file to exceed
the size limit we have.
if self.stream is None: # delay was set...
self.stream = self._open()
if self.maxBytes > 0: # are we rolling over?
msg = "%s\n" % self.format(record)
self.stream.seek(0, 2) #due to non-posix-compliant Windows feature
if self.stream.tell() + len(msg) >= self.maxBytes:
t = int(time.time())
if t >= self.rolloverAt:
log_filename, maxBytes=100, backupCount=5,
# encoding='bz2', # uncomment for bz2 compression
for i in range(10000):
logger.debug('i=%d' % i)