Python How to add custom fields to logging formatter

logging module is a useful module for logging. The log format somehow needs to be defined to make it more readable. The format can be defined easily by using logging.Formatter(format) with pre-defined fields. However, those pre-defined fields might not be enough for our format. How can we add custom fields there?

How to define formatter only with pre-defined fields

Let’s check how to set formatter with pre-defined fields. This is the code.

import logging
import logging.handlers
import os
from datetime import datetime
from path import Path

def generate_logger_without_custom_formatter(log_path: str):
    logger = logging.getLogger("ABC_APP")
    logger.setLevel(logging.DEBUG)

    log_path_dir = Path(log_path).parent
    os.makedirs(log_path_dir, exist_ok=True)

    rotatingHandler = logging.handlers.RotatingFileHandler(
        log_path,
        encoding="utf-8",
        maxBytes=1024,
        backupCount=5,
    )
    formatter = "%(asctime)s <%(levelno)d>[%(levelname)s] %(name)s %(filename)s line: %(lineno)d %(message)s"
    rotatingHandler.setFormatter(logging.Formatter(formatter))
    logger.addHandler(rotatingHandler)

    return logger

def write_log(logger: logging.Logger):
    logger.debug("message")
    logger.info("message")
    logger.warning("message")
    logger.error("message")
    logger.critical("message")

log_path_dir = Path(__file__).parent.joinpath("temp")
log_path = Path(log_path_dir).joinpath("test.log")
logger_without = generate_logger_without_custom_formatter(str(log_path))
write_log(logger_without)

The formatter can be set by setFormatter(). Since the pre-defined fields can be used, we can easily create our preferred format.

The following log is written as a result.

2023-11-21 19:30:02,099 <10>[DEBUG] ABC_APP logger.py line: 125 message
2023-11-21 19:30:02,099 <20>[INFO] ABC_APP logger.py line: 126 message
2023-11-21 19:30:02,099 <30>[WARNING] ABC_APP logger.py line: 127 message
2023-11-21 19:30:02,099 <40>[ERROR] ABC_APP logger.py line: 128 message
2023-11-21 19:30:02,099 <50>[CRITICAL] ABC_APP logger.py line: 129 message

It is enough to use only pre-defined fields in many cases but I needed to use custom fields because a log analysis application couldn’t handle this format. Another application supports the format but my application doesn’t. Then, I had to adapt it.

Define custom fields in formatter

What I needed is the following.

• Change the time format to DD-MM-YYYY HH:MM:SS.FFF
• Change Log level number
• Show the parent directory for the module

I tried to use LoggerAdapter at first but it’s impossible to map the log level to another value. So what we can do is to extend logging.Formatter.

_LEVEL_MAP = {
    logging.CRITICAL: 5,
    logging.FATAL: 5,
    logging.ERROR: 4,
    logging.WARNING: 3,
    logging.WARN: 3,
    logging.INFO: 2,
    logging.DEBUG: 1,
    logging.NOTSET: 0,
}


class LoggingFormatter(logging.Formatter):
    """
    CustomFormatter provides a format that supports TNCanalyzer. The format is same as advanced-touch.
    """

    def __init__(self) -> None:
        formatter = "%(custom_time)s <%(custom_log_level)s>[%(levelname)s] %(name)s %(source_path)s line: %(lineno)d %(message)s"
        super().__init__(formatter)

    def format(self, record: logging.LogRecord) -> str:
        record.custom_time = datetime.now().strftime("%d.%m.%Y %H:%M:%S.%f")[:-3]
        record.custom_log_level = _LEVEL_MAP.get(record.levelno, 0)
        record.source_path = self.__get_source_path(record)

        return super().format(record)

    def __get_source_path(self, record: logging.LogRecord):
        try:
            parent_dir_name = os.path.basename(os.path.dirname(record.pathname))
            return os.path.join(parent_dir_name, record.filename)
        except BaseException:  # pylint: disable=:broad-exception-caught
            return ""

We can access the pre-defined fields in format(). Therefore, we can define custom fields by using those values. Assign a value to record.variable_name. Then, it can be used in formatter.

This is the result.

21.11.2023 19:30:10.301 <1>[DEBUG] NICE_APP src/logger.py line: 125 message
21.11.2023 19:30:10.301 <2>[INFO] NICE_APP src/logger.py line: 126 message
21.11.2023 19:30:10.301 <3>[WARNING] NICE_APP src/logger.py line: 127 message
21.11.2023 19:30:10.301 <4>[ERROR] NICE_APP src/logger.py line: 128 message
21.11.2023 19:30:10.301 <5>[CRITICAL] NICE_APP src/logger.py line: 129 message

The format is different from the previous one.