Fluentd Guide

This guide describes how to send user definitive logs that uses in Hive Analytics v2 via fluentd.

Fluentd Install

  – We recommend installing the td–agent version 1.x.
  – The td–agent libraries are provided for each languages.  (https://docs.fluentd.org/language-bindings)

Installing td–agent

• http://docs.fluentd.org/articles/install-by-rpm
  
  $ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

  1	$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

• Edit the td–agent settings file
  
  $ vim /etc/td-agent/td-agent.conf

  1	$ vim /etc/td-agent/td-agent.conf

 

---

Fluentd Default Settings

• Log file: The default path of log file is /var/log/td-agent, and the td–agent.log file is located the sub folder.
• Event: Event in fluentd means record and log.
• <source>: <source> enables to receive various input and many plugins are able to use by using @ type.
• <match>: It handles the input event that sent via <source>.

 

---

Fluentd Plugin

• Check and install the plugin: Install the plugin you need after checking the list of plugins.
  (The plugins such as tail, forward and stdout are provided in default, but others like forest need to be install separately.)
  $ td-agent-gem list #Check the list of plugin # Available to install a plugin as the formatted td-agent-gem install [plugin you want] $ td-agent-gem install fluent-plugin-forest # Install the fluent-plugin-forest plugin

  1 2 3 4 5

  $ td-agent-gem list  #Check the list of plugin   # Available to install a plugin as the formatted td-agent-gem install [plugin you want] $ td-agent-gem install fluent-plugin-forest          # Install the fluent-plugin-forest plugin

• tag: Tag is a value that consists of Event, and the <match> section handles the value matched with tag.
  # Usage example for tag <source> @type tail tag ha2union.game.com2us # Attached this tag to an event that sent using in_tail path /var/log/sample.log </source> # An input event transferred through sources is sent to a match tag mapped with an event tag. <match ha2union.game.com2us> @type stdout <match>

  1 2 3 4 5 6 7 8 9 10 11 12

  # Usage example for tag <source> @type tail tag ha2union.game.com2us        # Attached this tag to an event that sent using in_tail path /var/log/sample.log </source>   # An input event transferred through sources is sent to a match tag mapped with an event tag. <match ha2union.game.com2us> @type stdout <match>

 

[Input Plugin]

• in_tail: The in_tail plugin reads data by using the tail command.
  
  <source> @type tail path /var/log/com2us.log # Reading path such as files pos_file /var/log/fluent/com2us_posfile.log.pos # Records the last reading part tag ha2union.game.com2us # Event tag. Attached the tag to the sending event. </source> # pos_file: When re-executing fluentd, handle the file from the last reading part.

  1 2 3 4 5 6 7

  <source>   @type tail                                       path /var/log/com2us.log                           # Reading path such as files   pos_file /var/log/fluent/com2us_posfile.log.pos    # Records the last reading part   tag ha2union.game.com2us                           # Event tag. Attached the tag to the sending event. </source> # pos_file: When re-executing fluentd, handle the file from the last reading part.

• in_forward: Generally, it is used for receiving the data from other fluentd nodes.
  
  <source> @type forward port 24224 # Port to send ( default : 24224 ) bind 0.0.0.0 # Bind address to send ( default : 0.0.0.0(all of address)) </source>

  1 2 3 4 5

  <source>   @type forward   port 24224        # Port to send ( default : 24224 )   bind 0.0.0.0      # Bind address to send ( default : 0.0.0.0(all of address)) </source>

 

[Output Plugin]

The output plugin defined in the <match> section defines the settings for buffering and flushing as the <buffer> sub section of the <match> section.

• output_stdout: Exports event logs with stdout and mainly uses for debugging.
  
  <match **> @type stdout </match>

  1 2 3

  <match **>   @type stdout </match>

• output_forward: Uses when sending event logs to other fluentd nodes.
  
  <match **> @type forward <server> name com2us.fluentd1 host [Server1's IP or domain for receiving] port 24224 weight 50 # Set up the load balancing weight in the <server> section </server> <server> name com2us.fluentd2 host [Server2's IP or domain for receiving] port 24224 weight 50 # Set up the load balancing weight in the <server> section </server> </match>

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

  <match **>   @type forward   <server>     name com2us.fluentd1     host [Server1's IP or domain for receiving]     port 24224     weight 50                    # Set up the load balancing weight in the <server> section   </server>   <server>     name com2us.fluentd2     host [Server2's IP or domain for receiving]     port 24224     weight 50                   # Set up the load balancing weight in the <server> section   </server> </match>

• fluent–plugin–forest : Enables to save the log files using a tag name. (By setting as below, logs can be saved in local.)
  
  <store> @type forest subtype file <template> time_slice_format %Y%m%d%H time_slice_wait 10s path /com2us/log/td-agent/data/${tag}/${tag}.*.log compress gz format json </template> </store>

  1 2 3 4 5 6 7 8 9 10 11

  <store>   @type forest   subtype file   <template>     time_slice_format %Y%m%d%H     time_slice_wait 10s     path /com2us/log/td-agent/data/${tag}/${tag}.*.log     compress gz     format json   </template> </store>

 

---

Usage examples of fluentd

Using fluentd enables to send logs with the methods as below. For more details, click the links below.

• Send logs using application libraries
• Send logs by reading a specific log file

 

---

Edit settings

• Check the td–agent.conf context
  
  $ td-agent -c /etc/td-agent/td-agent.conf

  1	$ td-agent -c /etc/td-agent/td-agent.conf

• Re-start td–agent
  
  $ service td-agent restart

  1	$ service td-agent restart

 

---