This tool is especially useful for syslog, apache or other web servers or mysql and other miscellaneous items. Moreover, the format of the log is only for more humanized data display and will not increase the calculation consumption.

Logstash itself has 120 default matching patterns in different languages (in fact, it is easy to see that it is a regular expression). You can also write your own expression and make a pull request; .

Grok formats by matching text labels with log content.

Format:% {Syntax: Semantic}

Grammar is the name of the tag, and semantics is the storage variable of the data obtained through tag mapping.

By default, if other storage types are needed, the storage types of all fields are strings.

%{NUMBER:num:int} Using this match will generate a field of type int.

You can use it directly to customize a specification for storing data in field_name.

You can also write a file to customize the pattern.

The most commonly used is to format message fields.

}

If you want to format the same field multiple times

Where "Duration:" is the corresponding character of regular direct matching, and% {} is the matching label of grok, the former is regular and the latter is a field.

Yes, it is very easy to use. You can add this to each filter.

Literally, delete some fields.

It is worth noting that the use of some tags requires the filter to work successfully. If your label is invalid, remember to check the prefilter (some labels must be filtered before they can work).

The purpose of this filter is to aggregate the data of multiple messages into one message, and provide a "code" field to customize the increase or decrease of the int attribute, and then throw it into a final message before entering the output process.

However, in order to use this filter, you need to set the filter parameter of Logstash to 1 (-w 1 flag) so that the filter can work normally. Otherwise, you will lose your hair.

Overall, this is a very chaotic filter. Try to complete message aggregation in the source or Kibana. Using this filter is extremely troublesome.

Refuse to translate this thing = =

English document

Deformation (? ) filter. Allows you to make general changes to fields. You can rename, delete, replace and modify the parameters in the received message.

If you read the above carefully, you will find that grok also provides the function of deleting fields. In fact, quite a few filters provide many repetitive functions, but I think calling the corresponding filters for different operations will make the configuration simple and clear.

There are some conversion rules involving authenticity. For details, please click on the English document link below the title. (That's right. See for yourself. hhhh)

You will find that there are two backslashes, yes, you need to add backslashes to all backslashes in the regular. . . I know it's a little circuitous, hhh.

Hashes can also be merged.

You play by yourself anyway. =

The date filter is used to parse the date format, and the parsed date is used as the timestamp of logstash.

chestnut

Date filter is a very important filter, which is used to sort messages and repopulate old data. If you don't get the right time in your information, then the search for them is likely to be out of order in the future.

If there is no such filter and no time stamp is set, logstash will set a time stamp according to the time when he first got the message, such as reading the message from a file, and the time of each reading will be used as a time stamp.