Log plugin

How it works

The log plugin can collect data from any logs on your server. It uses regular expressions to parse information from a log file. It can count the number of lines that match the expression and get a value from within that line.

Configuration

When configuring the log plugin you will have to specify the following attributes:

Parameter
Description
File path The full file path for the log file that you need to collect data from.
Data type The possible values of this field are:
  • Count: For calculating the count of the matched entries in the log file.
  • Average: For calculating the average values of the matched entries in the log file.
  • Histogram: For calculating the histogram values of the matched entries in the log file.
  • Minimum: For calculating the minimum values of the matched entries in the log file.
  • Maximum: For calculating the maximum values of the matched entries in the log file.
Metric name The metric name that will be used to store the data.
Regex The regex that will be used to match against each entry in the log file.
The regex can be in two formats:

It is advisable to test the regex on a line before you use it. regexr is a good online tool for testing regexes.

The regex will match against every log entry separately
Regex group name This field is required in case that the chosen data type is average, histogram, minimum or maximum. This is required to know which part of the matched regex we should use to calculate the average, histogram, minimum or maximum.
Unit The unit for the calculated data. We ask for this for cosmetic purposes so when the numbers are displayed on any widgets the unit will appear next to it.

Use-cases and examples

Web server logs

We have a log file for a web server with these kind of logs:

coscale.com GET /index.html 15824 43

The number 15824 represents the response size. If we wanted to calculate the average response size per second we can:

  • Use your own regex:
    • \.\*? \.\*? \.\*? (?<responseSize>[0-9]+) [0-9]+
  • Use a regex defined using grok patterns:
    • %{IP} %{WORD} %{URIPATHPARAM} %{NUMBER:responseSize} %{NUMBER}

Now if you configure your plugin like this:

Log plugin configuration screenshot

the log plugin will read the average response size from your logs and insert the data into CoScale to the metric Response size per second.

Errors count

You can use the log plugin to get the count of errors logged in the log file. For example if the error log entry looks like this 2016-01-11 15:48:33,327 - [error] The error description, your configuration will be as follows:

  • DataType: Count
  • Regex: .*[error].*

Latency for 3rd party calls

You can use the log plugin to store the latency histogram data of a 3rd party call. For example if the log entry looks like this CALL LAT 100 ms, your configuration will be as follows:

  • DataType: Histogram
  • Regex: CALL LAT (?'latency'\d+) ms
  • Regex group name: latency