Best practices for logging in Django

This section describes using logging for logging on a Django system

Here is a simple example of a logging module, which you can preview first, and then go into detail about the specific functions of each module:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(message)s',
        }
    },
    'handlers': {
        'file_1': {
            'level': 'INFO',
            'filename': '/Users/hunter/python/log_path/file_1.log',
            'formatter': 'verbose',
            'class': 'logging.FileHandler',
        },
        'file_2': {
            'level': 'INFO',
            'class': 'logging.FileHandler',
            'filename': '/Users/hunter/python/log_path/file_2.log',
            'formatter': 'verbose',
        },
        'custom_file': {
            'level': 'INFO',
            'class': 'logging.FileHandler',
            'filename': '/Users/hunter/python/log_path/custome_file.log',
            'formatter': 'verbose',
        }
    },
    'loggers': {
        '': {
            'handlers': ['file_1'],
            'level': 'INFO',
            'propagate': False,
        },
        'django': { 
            'handlers': ['file_2'],
            'level': 'INFO',
            'propagate': True,
        },
        'custom': {
            'handlers': ['custom_file'],
            'level': 'INFO',
            'propagate': False,
        }
    }
}

1. Module overview

In Django, logging can also be configured in setting.py with the key logging, and there are several main modules:

loggers, handlers, filters, formatters

After receiving the log information, the system enters the logger and sends it to the handler according to the specified handler list

Depending on how the handler handles it, the message is written to a file, sent to an email, or otherwise

This information can be further filtered by filter, and processed by handler according to the information organization of formatter

2. Loggers

Loggers are an entry point for learning the log system. Each logger is a named bucket. Information processed by loggers can be written to the logger as logs

Each logger can be configured with a log level. The log level describes the severity of information recorded in a logger. python defines the following log levels:

• DEBUG: indicates low-level system information based on the purpose of debugging
• INFO: indicates a common system message
• WARNING: Indicates that a minor problem has occurred
• ERROR: Indicates that a major problem has occurred
• CRITICAL: indicates that a critical problem has occurred

Each message written to a logger is called a Log Record.

Each log record has a log level to indicate the severity of the information when it is sent to the logger. For example:

logger.info("xxx")

These log records should contain useful information about the cause of the problem

When a message is sent to the logger, the level of the message is compared with the logging level of the logger. Only when the level of the message is greater than or equal to the logging level of the logger, the message is processed by the current logger

If this message is received by the logger, it is sent to Handlers

3. Handlers

We can understand that handler is a handler that decides what to do with the information sent to the logger every day, that is, the logging form

Such as writing a file, sending an email, etc

Like Logger, the handler has a log level. The handler processes information only when the log level sent to the handler is greater than or equal to the log record of the handler

A Logger can have multiple handler handlers, and each handler has its own log level. Therefore, different output can be determined according to the importance of the information

For example, you can use one handler to forward ERROR and CRITICAL information to the service page, and another handler to log all the information to a file for subsequent analysis

4. Filters

Filters are often used to provide additional control over logging from logger to handler

In theory, any log message that meets the log level requirements will be sent to handler for processing, and if you add a filter filter, you can add additional criteria to log processing

For example, you can add a filter that only allows ERROR levels from a certain source to be processed

filter can also be used to change the severity level of messages. For example, if certain conditions are met, you can downgrade an ERROR log to a WARNING

In this note, I will not cover the use of filter, because it is simple to be simple and do not need to configure so much for now

5. Formatters

Formatting, a log record needs to be rendered as text. formatter provides some properties of the formatter. The formatter consists of some LogRecord property values

6. Log recording method

When you configure loggers, handlers, filters, and formatters, you can get an instance of a logger and log through it

Here's an example:

import logging

logger = logging.getLogger(__name__)

def my_view(request):
    logger.info("this is a log")

When my_view is called, the system logs a log

If it is another level of logging, it will be:

logger.debug()
logger.info()
logger.warning()
logger.error()
logger.critical()

The following is a summary of the log recording process:

• When a log message needs to be recorded, it is sent to the corresponding logger
• The logger is then sent to the corresponding handler according to the specified handler
• The handler filters logs according to the log level or defined filter
• After the specified logs are defined by formatter, the generated logs are consoled, recorded to a file, or sent to an email

In the logging example at the beginning of the note, the configuration of the logs in this dict is in the opposite order to the order in which the messages are processed

There is no operation to set the filter, so the processing of the message is from logger to handler to formatter

We manually write a log message to the interface, defined in urls.py and views.py as follows:

# blog/urls.py
from django.urls.conf import path
from blog.views import time_view


urlpatterns = [
    path("curr_time", time_view),
]

# blog/views.py import datetime from django.http import HttpResponse import logging logger = logging.getLogger(__name__)  def time_view(request): now = datetime.datetime.now() html = "<h1>now: %s</h1>" % now logger.info("this is a log !" ) return HttpResponse(html)

After starting system, visit http://localhost:9898/blog/curr_time in the browser, you can see definition of log data has been written into the directory: file_1. Log and file_2. Log

Open the two log files, and you can see that the file_1.log written by the handler corresponding to the logger specified by the empty string under loggers is as follows:

INFO this is a log ! xxxx
INFO "GET /blog/curr_time HTTP/1.1" 200 40 xxxx

It contains the interface access information and our custom 'this is a log! 'information

In file_2.log, only the interface access information is available:

INFO  200 40 xxxx

When instantiating a logger, if the name of the logger is not specified, the logger will be written to the empty string we defined by default

Not specifying the logger name means not specifying the logger parameters when getLogger:

logger = logging.getLogger(__name__)

7. logger parameter analysis

Here, loggers set two keys, one for an empty string and one for django.

It is understood that the logger with the key django' is a fixed value and will receive all log information from the system, such as interface requests, but not user-specified logger output.

Empty string logger here can receive the user-defined logger output and some interface request information, but this requires the configuration of propagate

In the configuration of loggers:

    'loggers': {
        '': {
            'handlers': ['file_1'],
            'level': 'INFO',
            'propagate': False,
        },
        'django': {
            'handlers': ['file_2'],
            'level': 'INFO',
            'propagate': True,
        }
    }

There are a couple of arguments: handlers are message handlers, value is a list of handlers, you can specify multiple handlers, for example, for a message, and you can specify both writing to a file and sending a message, or writing to different files

The level parameter indicates the log level. If the level of the received message is less than INFO, the message is not processed. If the level is greater than or equal to INFO, the message is sent to the handler for processing

The meaning of propagate parameter can be understood as whether the propagate parameter is passed or not. In these two loggers, if the propagate value of django's logger is set to True, django sends messages to a logger that is set to an empty string

In other words, all messages received by django will be sent back to the empty string logger and processed again using its logger. If propagate is set to False, then the empty string logger will only receive the user-defined messages

8. handler parameter parsing

When a message is sent from a logger to a handler, the handlers parameter can define multiple handlers, which are distinguished by different keys

Under each handler we set four values here:

level Specifies the log level to be processed by a handler. This handler processes logs only when the level of the sent logs is greater than or equal to this level

class sets the method of log processing, here our value is logging.FileHandler, indicating the file processing method

There are StreamHandler outputs to Stream and prints to standard output, HttpHandler sends logs to the server over HTTP, and SMTPHandler sends logs via email

filename specifies the log address to be output. Previously, our class is defined as output to the file, so filename here defines the address of the output file

The formatter is the formatter that specifies the output format of the next level of log text

Log file processing policy

For log files, if the system is running all the time, then there is a problem that the log file is getting larger and larger, which is not suitable for the storage of the system and for us to find logs

So let's add a few more parameters to define the log file handling strategy

maxBytes, which defines the maximum number of bytes in a log file. If the log file is full, it will create a new file to continue writing, rather than continuing to add content to the original file

If we need to set the maximum size of a file to 5M, we can set it to 5 * 1024 * 1024

backupCount indicates the maximum number of log files. When the number of files exceeds the threshold, the earliest log files are deleted and only backupCount log files are retained

But use these two parameters above, the value of the class have to be replaced with logging. Handlers. RotatingFileHandler

The following describes several types of handler information processing

1. Configure RotatingFileHandler

rotate means to rotate positions on a regular basis

The role of RotatingFileHandler is to decide whether to write a new file based on the size of the file. Here is an example:

        'custom_file': {
            'level': 'INFO',
            'filename': '/home/hunter/python/log_path/custom.log',
            'class': 'logging.handlers.RotatingFileHandler',
            'formatter': 'verbose',
            'maxBytes': 5 * 1024 * 1024,
            'backupCount': 10
        }

This example indicates that logs are written to files. The maximum capacity of each file is 5 x 1024 x 1024, that is, 5 m. When a file is full of 5 m, a new file is opened to write log information, and the latest 10 files are kept in the folder.

There are two new configuration items

backupCount Indicates the maximum number of log files to be retained

maxBytes Indicates the maximum storage capacity for each log file

2. TimedRotatingFileHandler configuration

TimedRotatingFileHandler specifies whether to write a new file based on the time interval. The following is an example:

'time_file': { 'level': 'INFO', 'filename': '/home/hunter/python/log_path/custom.log', 'class': 'logging. Handlers. TimedRotatingFileHandler' # 'formatter' record time: 'verbose', 'backupCount: 3,' when ', 'M', 'interval' : 3,}

When the handler class value is this, it means that the handler decides whether to write a new file according to the time. The previous value is determined by the capacity of the file

There are two new configuration items,

One is when, indicating the unit of time interval, S is second, M is minute, H is hour, D or MIDNIGHT is day, W0-W6 is one week interval starting from a certain day of the week from Monday to Sunday

The other is interval, which is a multiple of the interval

The time for writing logs to new files is when * interval

3.HttpHandler Basic configuration

This configuration means that an HTTP interface is called if a log message needs to be processed. Here we can do just one example:

'http_handler': { 'level': 'INFO', 'class': 'logging.handlers.HTTPHandler', 'formatter': 'verbose', 'host': '192.168.1.8:9898', 'url' : '/ test_url' and 'method' : 'POST'},

This place has a few more configuration items

host indicates the ip address and port of the interface to be invoked

url indicates the path of the interface to be invoked

method Indicates the method to be invoked

4. Configure SMTPHandler

This configuration is used to send mails. If log messages are sent to the handler configured with this configuration, the system sends mails to the specified mailbox according to the mail configuration

Here's an example:

'email': { 'level': 'WARNING', 'class': 'logging.handlers.SMTPHandler', 'mailhost': ('smtp.163.com', 25), 'fromaddr': 'xxxxxxxx@163.com', 'toaddrs': 'xxxxxxx@qq.com', 'subject': 'System error!! ', 'credentials': ('xxxxxxx@163.com', 'JBD******'), 'timeout': 20 },

In this configuration, the configuration items are described as follows:

mailhost is the host and port of the mailbox through which the system sends emails. In this case, mailbox 163 is configured

From which mailbox fromaddr is sent, we can create a mailbox 163 and specify the value

toaddrs is the mailbox to which the log message is sent, that is, the mail receiving address

subject is the subject of the message we send, and the body of the message is the information we enter in logger.warning(" Error message ")

credentials refers to the authentication information of the 163 mailbox, two values, the first value is consistent with fromaddr, and the last is a verification code, which is a string of authorization password given to us by the 163 mailbox system page after the SMTP service is enabled. You can check this for yourself

After this configuration, the handler is specified in the logger handler list, and the logger.warning(" error message ") is used to trigger the mail sending function

5. Configure AdminEmailHandler

This configuration is also used for logging mail, but uses Django's default mailbox

The configuration in logging is:

        'mail_admins': {
            'level': 'WARNING',
            'class': 'django.utils.log.AdminEmailHandler',
            'include_html': True,
        },

However, this requires some additional mailbox configuration in settings.py, which is equivalent to reusing Django system functionality

Here are the mailbox configuration items in settings.py:

EMAIL_BACKEND = 'django. Core. Mail. Backends. SMTP. EmailBackend' EMAIL_HOST = 'smtp.163.com' # 163 email configuration address EMAIL_PORT = 465 # SMTP port EMAIL_HOST_USER = 'xxxxxx@163.com' # This is the mailbox used to send mail, Same as the last email address EMAIL_HOST_PASSWORD = 'JBDM******' # This is the authorization password EMAIL_USE_SSL = True EMAIL_FROM = SERVER_EMAIL = 'xxxxxxx@163.com' # This is the address from which the email is sent. It is the same as the 163 mailbox above. ADMINS = [['Hunter', 'xxxxxx@qq.com'],] # Email receiving address

After the above parameters are configured, you can also log trigger mail.

9. formatter parameter parsing

The formatter parameter is a little simpler, through different keys to distinguish different formatters, which can set a format parameter to format the information

    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(message)s',
        }
    },

In the example, only two parameters are set: levelname indicates the level of the log message, and message indicates the message content

For the message message of the request interface, the returned content is fixed, as in the previous example:

"GET /blog/curr_time HTTP/1.1" 200 40

The first is the request mode of the interface, the interface path and the HTTP protocol, and then the status code returned by the interface, here is 200, followed by the 40 number is the character length returned by the interface

If the message is manually written by the user in the system, what is the defined content and what is the output content

There are many parameters for the definition of format, the following is a few commonly used summary:

10. Specify the logger output

The log that we set before is sent to the logger when the key is empty string. What if we want to output some log information to a specific file?

When obtaining a logger, it is necessary to specify the corresponding logger according to its key. For example, we create a logger named custom and the corresponding handler, and then specify the output as follows:

        'custom': {
            'handlers': ['custom_file'],
            'level': 'INFO',
            'propagate': False,
        }

To specify the logger output:

import datetime from django.http import HttpResponse import logging custom_logger = logging.getLogger("custom") # Corresponding logger def time_view(request): now = datetime.datetime.now() html = "<h1>now: %s</h1>" % now custom_logger.info("this is a custom log") return HttpResponse(html)

In this way, specialized log output to specialized files can be implemented in corresponding places.

11. Log configuration examples

Next we implement such a log configuration function:

• All normal manual output from the user is written to a manual.log file
• The request data for all interfaces is entered into a request.log file
• Set up a separate log output that can be output to a specified file
• All logs with an INFO level are output to a file. Logs with a higher level than INFO are sent to the specified contact in an email
• The maximum capacity of each log file is 50 Mbit/s, and a folder contains a maximum of 10 logs of each type
• The log information structure is as follows: log level - Time - File name where the log output resides - Function name where the log output resides - Number of lines in the file where the log output resides - Message content

Here is the logging configuration to do this:

LOGGING = { 'version': 1, 'disable_existing_loggers': False, 'formatters': { 'verbose': { 'format': '%(levelname)s %(asctime)s %(filename)s %(funcName)s %(lineno)d %(message)s', } }, 'handlers': { 'manual_file': { 'level': 'INFO', 'filename': '/Users/hunter/python/log_path/manual.log', 'formatter': 'verbose', 'class': 'logging.handlers.RotatingFileHandler', 'maxBytes': 5 * 1024 * 1024, 'backupCount': 10 }, 'request_file': { 'level': 'INFO', 'filename': '/Users/hunter/python/log_path/request.log', 'class': 'logging.handlers.RotatingFileHandler', 'formatter': 'verbose', 'maxBytes': 5 * 1024 * 1024, 'backupCount': 10 }, 'custom_file': { 'level': 'INFO', 'filename': '/Users/hunter/python/log_path/custom.log', 'class': 'logging.handlers.RotatingFileHandler', 'formatter': 'verbose', 'maxBytes': 5 * 1024 * 1024, 'backupCount': 10 }, 'email': { 'level': 'WARNING', 'class': 'logging.handlers.SMTPHandler', 'mailhost': ('smtp.163.com', 25), 'fromaddr': 'xxxxxx@163.com', 'toaddrs': 'xxxxxxx@qq.com', 'subject': 'System error!! ', 'credentials': ('xxxxxx@163.com', 'JBD*******'), 'timeout': 20 }, }, 'loggers': { '': { 'handlers': ['manual_file', 'email'], 'level': 'INFO', 'propagate': False, }, 'django': { 'handlers': ['request_file'], 'level': 'INFO', 'propagate': True, }, 'custom': { 'handlers': ['custom_file'], 'level': 'INFO', 'propagate': False, }, }, }

Then we define an interface content:

import datetime from django.http import HttpResponse, JsonResponse import logging logger = logging.getLogger(__name__) custom_logger = logging.getLogger("custom") def time_view(request): now = datetime.datetime.now() html = "<h1>now: %s</h1>abc\nabc" % now logger.info("this is a log !" Info ("this is a custom log") logger.warning(" custom_logger.info ") Custom_Logger.info (" This is a custom log") Logger.warning (" ) return HttpResponse(html)

Call this interface to find that we have achieved the functionality we want!

This article introduces the basic concepts of log logging in Django and how to use it, including how to configure the log logger, how to set the log level, and how to record log information. At the same time, this article also provides some best practices, such as how to avoid log information leakage, how to log exception information, and so on. By learning from this article, developers can make better use of Django's log logging function to improve the reliability and stability of their applications.

So much for this article on best practices for logging in Django. For more information on logging in Django, please search for previous articles from Script House or continue reading the related articles below.

Latest comments