flume的基本使用可以参考Flume(NG)架构设计要点及配置实践和flume user guide

sink group

flume的负载均衡和容错都是通过sink groups实现的。sink groups可以将多个sink绑定到一个channel上，同时定义channel发往这个group的event的方式。定义时需要首先正常的将sink配置好（包括指定channel），并且定义sink间如何group。

failover

#配置文件：failover_sink_case13.conf
#Name the components on this agent
a1.sources= r1
a1.sinks= k1 k2
a1.channels= c1 c2
#将k1 k2 group起来，并定义优先级和failover time 上限 
a1.sinkgroups= g1
a1.sinkgroups.g1.sinks= k1 k2
a1.sinkgroups.g1.processor.type= failover
a1.sinkgroups.g1.processor.priority.k1= 5
a1.sinkgroups.g1.processor.priority.k2= 10
a1.sinkgroups.g1.processor.maxpenalty= 10000
 
#Describe/configure the source
a1.sources.r1.type= syslogtcp
a1.sources.r1.port= 50000
a1.sources.r1.host= 192.168.233.128
a1.sources.r1.channels= c1 c2
 
#Describe the sink
a1.sinks.k1.type= avro
a1.sinks.k1.channel= c1
a1.sinks.k1.hostname= 192.168.233.129
a1.sinks.k1.port= 50000
 
a1.sinks.k2.type= avro
a1.sinks.k2.channel= c2
a1.sinks.k2.hostname= 192.168.233.130
a1.sinks.k2.port= 50000
# Usea channel which buffers events in memory
a1.channels.c1.type= memory
a1.channels.c1.capacity= 1000
a1.channels.c1.transactionCapacity= 100

官网文档

Load balance

load balance也是通过sink group实现的

a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type = load_balance
a1.sinkgroups.g1.processor.backoff = true
a1.sinkgroups.g1.processor.selector = random

selector必须是“round_robin, random or FQCN of custom class that inherits from AbstractSinkSelector”

官方文档