fluentd + elasticsearch 설치 하기

fluentd + elasticsearch 설치 하기

fluentd와 elasticsearch를 연동하기 위해 해야 할 일 그리고 설치가 잘 안되는 경우 해결 방안

 

 

 

1. fluentd 설치는 아래와 같이 패키지 설치 두가지만 잘 되면 됩니다.

 gem install fluentd

 gem install fluent-plugin-elasticsearch

$ 그런데 종종 잘 안되는 경우가 생길 겁니다. 

위의 설치가 되지 않는다 면 아래 "4. 패키지 설치가 안되는 경우 해결 방안" 를 참고 한다.

     fluent-plugin-elasticsearch 문제 해결 : gem install patron -v0.4.9 로 하자 

    (아래 참조)

2. Configure 환경 설정 예제

[fluent.conf]

## ************************** ## elasticsearch include fluent_elasticsearch.conf

 

[fluent_elasticsearch.conf]

<match td.nginx.**>     type copy   <store>      type elasticsearch     # logstash_format true      host 10.20.4.96      port 9200      request_timeout 15s      logstash_format true      logstash_prefix logstash      type_name  accesslog      logstash_dateformat %Y%m%d      utc_index true   </store>   <store>      type stdout   </store> </match>

 

3. 기동 환경 테스트

[root@ specifications]# /etc/rc.d/init.d/fluent configtest

 > fluent : /usr/local/rvm/gems/ruby-1.9.3-p547/bin/fluentd
 > options :  -c /etc/fluent/fluent.conf --daemon /var/log/fluent/fluent.pid --user hadmin --group hadmin -q  --dry-run  [  OK  ]

 

4.패키지 설치가 안되는 경우 해결 방안

  [힌트]
  *patron 패키지를 설치하여야 하는데 설치가 안되는 문제가 있습니다.
   gem install patron -v0.4.9 로 특정 버전을 지정해서 설치하니 되는군요 .
  아래 뻘짓 과정을 참고 바랍니다.

 

[root@ gems]# gem install fluentd  #> fluentd를 설치합니다.

/usr/local/rvm/rubies/ruby-1.9.3-p547/lib/ruby/1.9.1/yaml.rb:84:in `<top (required)>':
It seems your ruby installation is missing psych (for YAML output).
To eliminate this warning, please install libyaml and reinstall your ruby.
Successfully installed fluentd-0.10.53
Parsing documentation for fluentd-0.10.53
Done installing documentation for fluentd after 4 seconds
1 gem installed

 

[root@ gems]# /etc/rc.d/init.d/fluent configtest   #> fluentd를 설치하고 환경 테스트를 합니다.

 > fluent : /usr/local/rvm/gems/ruby-1.9.3-p547/bin/fluentd
 > options :  -c /etc/fluent/fluent.conf --daemon /var/log/fluent/fluent.pid --user hadmin --group hadmin -q  --dry-run
configtest fluent: 2014-09-24 09:09:36 +0900 [error]: dry run failed: Unknown output plugin 'elasticsearch'. Run 'gem search -rd fluent-plugin' to find plugins

 

추가 패키지 설치 ( gem install  패키지명 )

 

[ fluent-plugin-elasticsearch  설치 ]

[root@ gems]# gem install fluent-plugin-elasticsearch  #> ElasticSearch 플러그인을 설치 합니다.

/usr/local/rvm/rubies/ruby-1.9.3-p547/lib/ruby/1.9.1/yaml.rb:84:in `<top (required)>':
It seems your ruby installation is missing psych (for YAML output).
To eliminate this warning, please install libyaml and reinstall your ruby.
Building native extensions.  This could take a while...
ERROR:  Error installing fluent-plugin-elasticsearch:
        ERROR: Failed to build gem native extension.

    /usr/local/rvm/rubies/ruby-1.9.3-p547/bin/ruby extconf.rb
checking for curl-config... yes
checking for rb_thread_blocking_region()... yes
creating Makefile
make  clean
make
compiling membuffer.c
compiling session_ext.c
session_ext.c: In function ‘Init_session_ext’:
session_ext.c:727: error: ‘CURLPROXY_HTTP_1_0’ undeclared (first use in this function)
session_ext.c:727: error: (Each undeclared identifier is reported only once
session_ext.c:727: error: for each function it appears in.)
session_ext.c:730: error: ‘CURLPROXY_SOCKS4A’ undeclared (first use in this function)
session_ext.c:731: error: ‘CURLPROXY_SOCKS5_HOSTNAME’ undeclared (first use in this function)
make: *** [session_ext.o] 오류 1
make failed, exit code 2
Gem files will remain installed in /usr/local/rvm/gems/ruby-1.9.3-p547/gems/patron-0.4.18 for inspection.
Results logged to /usr/local/rvm/gems/ruby-1.9.3-p547/extensions/x86_64-linux/1.9.1/patron-0.4.18/gem_make.out
*  patron-0.4.18 이 문제라고 하는듯 합니다. (버전이 최신이거나 낮거나의 문제일텐테요)

 

[ patron  설치 (오류) ]

[root@ gems]# gem install patron  #> patron 플러그인을 설치 하는데 에러가 나네요.

/usr/local/rvm/rubies/ruby-1.9.3-p547/lib/ruby/1.9.1/yaml.rb:84:in `<top (required)>':
It seems your ruby installation is missing psych (for YAML output).
To eliminate this warning, please install libyaml and reinstall your ruby.
Building native extensions.  This could take a while...
ERROR:  Error installing patron:
        ERROR: Failed to build gem native extension.
    /usr/local/rvm/rubies/ruby-1.9.3-p547/bin/ruby extconf.rb
checking for curl-config... yes
checking for rb_thread_blocking_region()... yes
creating Makefile
make  clean
make
compiling membuffer.c
compiling session_ext.c
session_ext.c: In function ‘Init_session_ext’:
session_ext.c:727: error: ‘CURLPROXY_HTTP_1_0’ undeclared (first use in this function)
session_ext.c:727: error: (Each undeclared identifier is reported only once
session_ext.c:727: error: for each function it appears in.)
session_ext.c:730: error: ‘CURLPROXY_SOCKS4A’ undeclared (first use in this function)
session_ext.c:731: error: ‘CURLPROXY_SOCKS5_HOSTNAME’ undeclared (first use in this function)
make: *** [session_ext.o] 오류 1
make failed, exit code 2
Gem files will remain installed in /usr/local/rvm/gems/ruby-1.9.3-p547/gems/patron-0.4.18 for inspection.
Results logged to /usr/local/rvm/gems/ruby-1.9.3-p547/extensions/x86_64-linux/1.9.1/patron-0.4.18/gem_make.out

 

[ psych   설치 ] 

[root@ gems]# gem install psych   #> missing psych 이라 하여 이것도 설치해 봅니다만 에러입니다..

/usr/local/rvm/rubies/ruby-1.9.3-p547/lib/ruby/1.9.1/yaml.rb:84:in `<top (required)>':
It seems your ruby installation is missing psych (for YAML output).
To eliminate this warning, please install libyaml and reinstall your ruby.
Fetching: psych-2.0.6.gem (100%)
Building native extensions.  This could take a while...
Successfully installed psych-2.0.6
Parsing documentation for psych-2.0.6
Installing ri documentation for psych-2.0.6
Done installing documentation for psych after 2 seconds
1 gem installed

 

[ patron 설치 ]

  : 문제가 발생한 패키지 재설치 후 연관된 패키지도 다시 설치 합니다.

[root@ gems]# gem install patron

Building native extensions.  This could take a while...
ERROR:  Error installing patron:
        ERROR: Failed to build gem native extension.
    /usr/local/rvm/rubies/ruby-1.9.3-p547/bin/ruby extconf.rb
checking for curl-config... yes
checking for rb_thread_blocking_region()... yes
creating Makefile
make  clean
make
compiling membuffer.c
compiling session_ext.c
session_ext.c: In function ‘Init_session_ext’:
session_ext.c:727: error: ‘CURLPROXY_HTTP_1_0’ undeclared (first use in this function)
session_ext.c:727: error: (Each undeclared identifier is reported only once
session_ext.c:727: error: for each function it appears in.)
session_ext.c:730: error: ‘CURLPROXY_SOCKS4A’ undeclared (first use in this function)
session_ext.c:731: error: ‘CURLPROXY_SOCKS5_HOSTNAME’ undeclared (first use in this function)
make: *** [session_ext.o] 오류 1
make failed, exit code 2
Gem files will remain installed in /usr/local/rvm/gems/ruby-1.9.3-p547/gems/patron-0.4.18 for inspection.
Results logged to /usr/local/rvm/gems/ruby-1.9.3-p547/extensions/x86_64-linux/1.9.1/patron-0.4.18/gem_make.out

 

[ patron -v0.4.9 설치]  (설치가 안되면 버전을 낮추어 설치해 봅니다.)

[root@gems]# gem install patron -v0.4.9   #> 버전이 낮은 플러그인을 설치 합니다.

Fetching: patron-0.4.9.gem (100%)
Building native extensions.  This could take a while...
Successfully installed patron-0.4.9
Parsing documentation for patron-0.4.9
Installing ri documentation for patron-0.4.9
Done installing documentation for patron after 0 seconds
1 gem installed

 

[fluent-plugin-elasticsearch 설치] - 성공

[root@ gems]# gem install fluent-plugin-elasticsearch   #> Elastic search 플러그인이 설치 됩니다.

Fetching: elasticsearch-transport-0.4.11.gem (100%)
Successfully installed elasticsearch-transport-0.4.11
Fetching: elasticsearch-api-0.4.11.gem (100%)
Successfully installed elasticsearch-api-0.4.11
Fetching: elasticsearch-0.4.11.gem (100%)
Successfully installed elasticsearch-0.4.11
Fetching: fluent-plugin-elasticsearch-0.4.0.gem (100%)
Successfully installed fluent-plugin-elasticsearch-0.4.0
Parsing documentation for elasticsearch-0.4.11
Installing ri documentation for elasticsearch-0.4.11
Parsing documentation for elasticsearch-api-0.4.11
Installing ri documentation for elasticsearch-api-0.4.11
Parsing documentation for elasticsearch-transport-0.4.11
Installing ri documentation for elasticsearch-transport-0.4.11
Parsing documentation for fluent-plugin-elasticsearch-0.4.0
Installing ri documentation for fluent-plugin-elasticsearch-0.4.0
Done installing documentation for elasticsearch, elasticsearch-api, elasticsearch-transport, fluent-plugin-elasticsearch after 3 seconds
4 gems installed

* 관련 플러그인이 잘 설치 안되는 경우 설치 순서나 버전에 대한 호환성을 잘 봐야 할 부분입니다.