fluentd

Log Event Middleware

Event Structure

The log events consists of tag, time and record

  • tag where an event comes from. For message routing.
  • time when an event happens. Epoch time.
  • record Actual log content. JSON object

Event Processing

The Router Engine contains rules to apply for different input data

Input

Retrieving log event from external sources

  • tail
  • forward
  • udp
  • tcp
  • http
  • syslog
  • exec
  • dummy
  • windows eventlog

Parser

Sometimes the <parse> directive for input plugins like tail cannot parse the users custom data format, e.g. a context-dependent grammar that cannot be parsed with regular expressions. In this case one can write it’s own parser plugin.

  • regexp
  • apache2
  • apache_error
  • nginx
  • syslog
  • csv
  • tsv
  • ltsv
  • json
  • multiline
  • none

Filters

Filters are applied before matching for example to reject some events.

<filter test.cycle>
  @type grep
  <exclude>
    key action
    pattern ^logout$
  </exclude>
</filter>

Output

Is configured via <match myservice_name>

Formaters

Mostly for file out

  • json
  • ltsv
  • csv
  • msgpack
  • hash
<match foo.bar>
  @type file
  path /path/to/file
  <format>
    @type json
  </format>
</match>

Storage

Used to store key value pairs in e.g. Redis

Labels

Labels work like a GOTO instruction, so global filters are skipped

<source>
  @type http
  bind 0.0.0.0
  port 8888
  @label @STAGING
</source>

<filter test.cycle>
  @type grep
  <exclude>
    key action
    pattern ^login$
  </exclude>
</filter>

<label @STAGING>
  <filter test.cycle>
    @type grep
    <exclude>
      key action
      pattern ^logout$
    </exclude>
  </filter>

  <match test.cycle>
    @type stdout
  </match>
</label>

Buffers

Output Plugins support the <buffer> section to specify how to buffer events:

<match test.cycle>
  @type stdout
  <buffer>
    @type memory
    flush_mode immediate
  </buffer>
</match>