We'd love to use a ruby-based solution like this, but the docs say it will lose data whenever the receiving end crashes. Any plans to fix that?
The way it was described in the docs gave me the impression there is no acknowledgement of network writes - if that's true won't even clean shutdowns lose data sometimes?
ACK of network transfer is available ("require_ack_response" option). This option ends up choice of at-most-once semantics vs. at-least-once semantics. You need to choose and you can choose.
Fuentd provides "buffer_type file" to buffer records on disk. Shutting down won't loose data. If you need to choose memory buffer for performance reasons, fluentd enables "flush_at_shutdown" option by default.
You would also want to use <secondary> feature. This lets you to write a buffer chunk to another storage if the primary destination is not available "retry_limit" times.
Ah, thanks - require_ack_response sounds like what I was missing. Some blogs are from before this was added in 0.12 so I didn't know about it.
I am still interested in forwarder failure cases - I have replied to kiyoto's comment talking about the HA docs, which still talk about some other cases that can lose messages.
In this case:
* The process dies immediately after receiving the events, but before writing them into the buffer.
Is it possible to require acknowledgement that the log event has been written to the buffer? Is that separate to what require_ack_response does?
>We'd love to use a ruby-based solution like this, but the docs say it will lose data whenever the receiving end crashes. Any plans to fix that?
Where does it say this? I don't think this was ever the case for Fluentd.
>The way it was described in the docs gave me the impression there is no acknowledgement of network writes - if that's true won't even clean shutdowns lose data sometimes?
This is not true. All writes are acknowledged over TCP, at least between Fluentd and Fluentd.
No, in the described case, the message can get lost. This is a really unlikely scenario though. The only real-world case that I know of first-hand is using file buffer and somehow being unable to write to disk, possibly because the disk is full. Something like that can be prevented by a fairly routine set of server monitoring alerts.
The way it was described in the docs gave me the impression there is no acknowledgement of network writes - if that's true won't even clean shutdowns lose data sometimes?