The Author Online Book Forums are Moving

The Author Online Book Forums will soon redirect to Manning's liveBook and liveVideo. All book forum content will migrate to liveBook's discussion forum and all video forum content will migrate to liveVideo. Log in to liveBook or liveVideo with your Manning credentials to join the discussion!

Thank you for your engagement in the AoF over the years! We look forward to offering you a more enhanced forum experience.

rcoe67 (4) [Avatar] Offline
#1
I'm curious what the overlap is between the capabilities of ULP using Kafka and the ELK stack (Elasticsearch, Logstash, Kibana)? Considering that Nagios Log Server is built on the ELK stack, I'm not sure of the value-add in choosing ULP, especially considering that the book mentions how difficult it is to manage Zookeeper in a multi-node production environment (p. 37).

Not sure this is the right question for this forum but perhaps the book should cover the disparities, if any exist, that would help in the decision making process. In fact, I am in that situation right now, so would love some insight.

Thanks,
Robin.
jfs.world (121) [Avatar] Offline
#2
EDIT: sorry, meant to edit this, but in my haste created a new message instead when I clicked on the quote button. There isn't any way for me to delete my own post (duh), so... am leaving this note here.
jfs.world (121) [Avatar] Offline
#3
rcoe67 wrote:I'm curious what the overlap is between the capabilities of ULP using Kafka and the ELK stack (Elasticsearch, Logstash, Kibana)? Considering that Nagios Log Server is built on the ELK stack, I'm not sure of the value-add in choosing ULP, especially considering that the book mentions how difficult it is to manage Zookeeper in a multi-node production environment (p. 37).

Not sure this is the right question for this forum but perhaps the book should cover the disparities, if any exist, that would help in the decision making process. In fact, I am in that situation right now, so would love some insight.

Thanks,
Robin.


I would be curious as well, and would like to see a comparison. I see the ELK stack as pretty much tackling the same issue here. Reading the description of the book (http://manning.com/dean/), it seems like it means to help you (?) build your own equivalent ELK stack? Is that true? (And even if were so, it would be good to talk about ELK, and why or why not; ie. the pros and cons....)
vimat (5) [Avatar] Offline
#4
I am also in a similar process of evaluation. As far I as understand, ELK is more related to products such as Splunk that aims to collect and process untyped logs. For instance, I use Splunk in my company to search for patterns is untyped logs using its full text search capabilities. This is more related to troubles diagnosis or monitoring. I am studying ULP for collecting typed events produced by a large set of applications. My goals are more related to business analysis and also monitoring but more from a SLA point of view. I expect to use Kafka for short-term storage of events, real time analysis and ElasticSearch for long-term storage of events and batch analysis. It would be great to have your opinion on the subject just to be sure that i am not following the wrong way.

Sincerly,
Vincent
jfs.world (121) [Avatar] Offline
#5
jfs.world wrote:
rcoe67 wrote:I'm curious what the overlap is between the capabilities of ULP using Kafka and the ELK stack (Elasticsearch, Logstash, Kibana)? Considering that Nagios Log Server is built on the ELK stack, I'm not sure of the value-add in choosing ULP, especially considering that the book mentions how difficult it is to manage Zookeeper in a multi-node production environment (p. 37).

Not sure this is the right question for this forum but perhaps the book should cover the disparities, if any exist, that would help in the decision making process. In fact, I am in that situation right now, so would love some insight.

Thanks,
Robin.


I would be curious as well, and would like to see a comparison. I see the ELK stack as pretty much tackling the same issue here. Reading the description of the book (http://manning.com/dean/), it seems like it means to help you (?) build your own equivalent ELK stack? Is that true? (And even if were so, it would be good to talk about ELK, and why or why not; ie. the pros and cons....)


Coming back to this now after some time and correcting myself after discovering Kafka serendipitously, and then also after having read the book: ELK is but a specific stack (and not a ULP setup at that!), and is not a processing methodology or architecture; and it only serves a very narrow and specific purpose. ElasticSearch (loosely calling it the "log" of the stack) is also not a unified log and for these reasons, it would not fit to discuss ELK in the book.