k8s容器環(huán)境收集應(yīng)用日志到已有的ELK日志平臺(tái)

Tags: k8s環(huán)境下的容器日志收集
K8S環(huán)境下面如何收集應(yīng)用日志
===
在本文中重點(diǎn)講一下K8S容器環(huán)境中如何收集容器的日志;

創(chuàng)新互聯(lián)-專業(yè)網(wǎng)站定制、快速模板網(wǎng)站建設(shè)、高性價(jià)比鹽亭網(wǎng)站開發(fā)、企業(yè)建站全套包干低至880元,成熟完善的模板庫(kù),直接使用。一站式鹽亭網(wǎng)站制作公司更省心,省錢,快速模板網(wǎng)站建設(shè)找我們,業(yè)務(wù)覆蓋鹽亭地區(qū)。費(fèi)用合理售后完善,十余年實(shí)體公司更值得信賴。

1. 容器日志收集方案的選擇:

??在K8S集群中,容器的日志收集方案一般有三種;第一種方案是通過(guò)在每一個(gè)k8s節(jié)點(diǎn)安裝日志收集客戶端軟件,比如fluentd。這種方案不好的一點(diǎn)是應(yīng)用的日志必須輸出到標(biāo)準(zhǔn)輸出,并且是通過(guò)在每一臺(tái)計(jì)算節(jié)點(diǎn)的/var/log/containers目錄下面的日志文件,這個(gè)日志文件的名稱是這種格式user-center-765885677f-j68zt_default_user-center-0867b9c2f8ede64cebeb359dd08a6b05f690d50427aa89f7498597db8944cccc.log,文件名稱有很多隨機(jī)字符串,很難和容器里面的應(yīng)用對(duì)應(yīng)起來(lái)。并且在網(wǎng)上看到別人說(shuō)這個(gè)里面的日志,對(duì)于JAVA的報(bào)錯(cuò)內(nèi)容沒有多行合并,不過(guò)我還沒有測(cè)試過(guò)此方案。

??第二種方案就是在應(yīng)用的pods里面在運(yùn)行一個(gè)sidecar container(邊角容器),這個(gè)容器會(huì)和應(yīng)用的容器掛載同一個(gè)volume日志卷。比如這個(gè)sidecar容器可以是filebeat或者flunetd等;這種方案不足之處是部署了sidecar , 所以會(huì)消耗資源 , 每個(gè)pod都要起一個(gè)日志收集容器。
?第三種方案就是直接將應(yīng)用的日志收集到kafka,然后通過(guò)kafka再發(fā)送到logstash,再處理成json格式的日志發(fā)送到es集群,最后在kibana展示。我實(shí)驗(yàn)的就是這種方案。通過(guò)修改logsbak配置文件實(shí)現(xiàn)了日志直接發(fā)送到kafka緩存的功能;下面直接看配置了

1. logsbak配置:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <jmxConfigurator/>  <!-- 動(dòng)態(tài)加載-->

    <property name="log-path" value="/apptestlogs" />  <!-- 統(tǒng)一 /applogs 下面 -->
    <property name="app-name" value="test" />  <!-- 應(yīng)用系統(tǒng)名稱 -->
    <property name="filename" value="test-test" />  <!---日志文件名,默認(rèn)組件名稱 -->
    <property name="dev-group-name" value="test" /> <!-- 開發(fā)團(tuán)隊(duì)名稱 -->

    <conversionRule conversionWord="traceId"  converterClass="org.lsqt.components.log.logback.TraceIdConvert"/>

    <!-- 根據(jù)實(shí)際情況修改變量 end-->
    -<appender name="consoleAppender" class="ch.qos.logback.core.ConsoleAppender">
    <!-- 典型的日志pattern -->
        <!-- -<encoder>-->
          <!--<pattern>[%date{ISO8601}] [%level] %logger{80} [%thread] [%traceId] ${dev-group-name} ${app-name} Line:%-3L - %msg%n</pattern>-->
        <!--</encoder>-->
    -<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
    <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
        <pattern>[%date{ISO8601}] [%level] %logger{80} [%thread] [%tid] ${dev-group-name} ${app-name} Line:%-3L - %msg%n</pattern>
    </layout>
    </encoder>
    </appender>

    -<appender name="fileAppender" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${log-path}/${app-name}/${filename}.log</file>
    -<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
    <fileNamePattern>/${log-path}/${app-name}/${filename}.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
    <maxHistory>15</maxHistory>
    <!--用來(lái)指定日志文件的上限大小,例如設(shè)置為300M的話,那么到了這個(gè)值,就會(huì)刪除舊的日志。-->
    <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
        <maxFileSize>300MB</maxFileSize>
    </timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
    <!-- -<encoder>-->
    <!--<pattern>[%date{ISO8601}] [%level] %logger{80} [%thread] [%traceId] ${dev-group-name} ${app-name} Line:%-3L - %msg%n</pattern>-->
<!--</encoder>-->
    -<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
    <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
        <pattern>[%date{ISO8601}] [%level] %logger{80} [%thread] [%tid] ${dev-group-name} ${app-name} Line:%-3L - %msg%n</pattern>
    </layout>
</encoder>
</appender>
    <appender name="errorAppender" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${log-path}/${app-name}/${filename}-error.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>/${log-path}/${app-name}/${filename}-error.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>300MB</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
            <maxHistory>15</maxHistory>
        </rollingPolicy>
        <!--<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">-->
            <!--<pattern>[%date{ISO8601}] [%level] %logger{80} [%thread] [%traceId] ${dev-group-name} ${app-name} Line:%-3L - %msg%n</pattern>-->
        <!--</encoder>-->
        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
        <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
            <pattern>[%date{ISO8601}] [%level] %logger{80} [%thread] [%tid] ${dev-group-name} ${app-name} Line:%-3L - %msg%n</pattern>
        </layout>
    </encoder>
        <filter class="ch.qos.logback.classic.filter.LevelFilter">
            <level>ERROR</level>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>
    </appender>

    <!-- This example configuration is probably most unreliable under
    failure conditions but wont block your application at all -->
    <appender name="very-relaxed-and-fast-kafka-appender" class="com.github.danielwegener.logback.kafka.KafkaAppender">
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
           <pattern>[%date{ISO8601}] [%level] %logger{80} [%thread] [%tid] ${dev-group-name} ${app-name} Line:%-3L - %msg%n</pattern>
        </encoder>
        <topic>elk-stand-sit-fkp-eureka</topic>
        <!-- we don't care how the log messages will be partitioned  -->
        <keyingStrategy class="com.github.danielwegener.logback.kafka.keying.NoKeyKeyingStrategy" />

        <!-- use async delivery. the application threads are not blocked by logging -->
        <deliveryStrategy class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy" />

        <!-- each <producerConfig> translates to regular kafka-client config (format: key=value) -->
        <!-- producer configs are documented here: https://kafka.apache.org/documentation.html#newproducerconfigs -->
        <!-- bootstrap.servers is the only mandatory producerConfig -->
        <producerConfig>bootstrap.servers=192.168.1.12:9092,192.168.1.14:9092,192.168.1.15:9092</producerConfig>
        <!-- don't wait for a broker to ack the reception of a batch.  -->
        <producerConfig>acks=0</producerConfig>
        <!-- wait up to 1000ms and collect log messages before sending them as a batch -->
        <producerConfig>linger.ms=1000</producerConfig>
        <!-- even if the producer buffer runs full, do not block the application but start to drop messages -->
        <producerConfig>max.block.ms=0</producerConfig>
        <!-- define a client-id that you use to identify yourself against the kafka broker -->
        <producerConfig>client.id=${HOSTNAME}-${CONTEXT_NAME}-logback-relaxed</producerConfig>
        <!-- define All log messages that cannot be delivered fast enough will then immediately go to the fallback appenders -->
        <producerConfig>block.on.buffer.full=false</producerConfig>

         <!-- this is the fallback appender if kafka is not available. -->
        <appender-ref ref="consoleAppender" />
    </appender>

    <root level="debug">
        <appender-ref ref="very-relaxed-and-fast-kafka-appender" /> 
        <appender-ref ref="fileAppender"/>
        <appender-ref ref="consoleAppender"/>
        <appender-ref ref="errorAppender"/>

    </root>
</configuration>

###2. 針對(duì)logsbak配置說(shuō)明:###

  1. logsbak直接發(fā)送日志到kafka有幾種方式,一種是異步模式,一種是同步模式。異步模式的意思就是如果kafka因?yàn)榫W(wǎng)絡(luò)情況出現(xiàn)故障,則阻塞發(fā)送日志或者直接將日志發(fā)送到后備存儲(chǔ),比如后備存儲(chǔ)是發(fā)送到日志文件;同步模式的意思就是即使kafka出現(xiàn)網(wǎng)絡(luò)情況不可達(dá),則就會(huì)影響到日志線程,進(jìn)而影響到應(yīng)用的性能。不過(guò)這個(gè)模式的我沒有測(cè)試過(guò);配置如下:
    <!-- This example configuration is more restrictive and will try to ensure that every message
     is eventually delivered in an ordered fashion (as long the logging application stays alive) -->
    <appender name="very-restrictive-kafka-appender" class="com.github.danielwegener.logback.kafka.KafkaAppender">
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>

        <topic>important-logs</topic>
        <!-- ensure that every message sent by the executing host is partitioned to the same partition strategy -->
        <keyingStrategy class="com.github.danielwegener.logback.kafka.keying.HostNameKeyingStrategy" />
        <!-- block the logging application thread if the kafka appender cannot keep up with sending the log messages -->
        <deliveryStrategy class="com.github.danielwegener.logback.kafka.delivery.BlockingDeliveryStrategy">
            <!-- wait indefinitely until the kafka producer was able to send the message -->
            <timeout>0</timeout>
        </deliveryStrategy>

        <!-- each <producerConfig> translates to regular kafka-client config (format: key=value) -->
        <!-- producer configs are documented here: https://kafka.apache.org/documentation.html#newproducerconfigs -->
        <!-- bootstrap.servers is the only mandatory producerConfig -->
        <producerConfig>bootstrap.servers=localhost:9092</producerConfig>
        <!-- restrict the size of the buffered batches to 8MB (default is 32MB) -->
        <producerConfig>buffer.memory=8388608</producerConfig>

        <!-- If the kafka broker is not online when we try to log, just block until it becomes available -->
        <producerConfig>metadata.fetch.timeout.ms=99999999999</producerConfig>
        <!-- define a client-id that you use to identify yourself against the kafka broker -->
        <producerConfig>client.id=${HOSTNAME}-${CONTEXT_NAME}-logback-restrictive</producerConfig>
        <!-- use gzip to compress each batch of log messages. valid values: none, gzip, snappy  -->
        <producerConfig>compression.type=gzip</producerConfig>

        <!-- Log every log message that could not be sent to kafka to STDERR -->
        <appender-ref ref="STDERR"/>
    </appender>  

通過(guò)配置logsbak直接輸出到kafka,并且使用異步模式,就成功的在kibana里面看到了容器的日志了;
k8s容器環(huán)境收集應(yīng)用日志到已有的ELK日志平臺(tái)

博文的更詳細(xì)內(nèi)容請(qǐng)關(guān)注我的個(gè)人微信公眾號(hào) “云時(shí)代IT運(yùn)維”,本公眾號(hào)旨在共享互聯(lián)網(wǎng)運(yùn)維新技術(shù),新趨勢(shì); 包括IT運(yùn)維行業(yè)的咨詢,運(yùn)維技術(shù)文檔分享。重點(diǎn)關(guān)注devops、jenkins、zabbix監(jiān)控、kubernetes、ELK、各種中間件的使用,比如redis、MQ等;shell和python等運(yùn)維編程語(yǔ)言;本人從事IT運(yùn)維相關(guān)的工作有十多年。2008年開始專職從事Linux/Unix系統(tǒng)運(yùn)維工作;對(duì)運(yùn)維相關(guān)技術(shù)有一定程度的理解。本公眾號(hào)所有博文均是我的實(shí)際工作經(jīng)驗(yàn)總結(jié),基本都是原創(chuàng)博文。我很樂意將我積累的經(jīng)驗(yàn)、心得、技術(shù)與大家分享交流!希望和大家在IT運(yùn)維職業(yè)道路上一起成長(zhǎng)和進(jìn)步;

k8s容器環(huán)境收集應(yīng)用日志到已有的ELK日志平臺(tái)

文章名稱:k8s容器環(huán)境收集應(yīng)用日志到已有的ELK日志平臺(tái)
URL網(wǎng)址:http://muchs.cn/article30/jcjspo.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián),為您提供網(wǎng)頁(yè)設(shè)計(jì)公司、移動(dòng)網(wǎng)站建設(shè)搜索引擎優(yōu)化、響應(yīng)式網(wǎng)站、網(wǎng)站收錄網(wǎng)站改版

廣告

聲明:本網(wǎng)站發(fā)布的內(nèi)容(圖片、視頻和文字)以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主,如果涉及侵權(quán)請(qǐng)盡快告知,我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如需處理請(qǐng)聯(lián)系客服。電話:028-86922220;郵箱:631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載,或轉(zhuǎn)載時(shí)需注明來(lái)源: 創(chuàng)新互聯(lián)

綿陽(yáng)服務(wù)器托管