The Author Online Book Forums are Moving

The Author Online Book Forums will soon redirect to Manning's liveBook and liveVideo. All book forum content will migrate to liveBook's discussion forum and all video forum content will migrate to liveVideo. Log in to liveBook or liveVideo with your Manning credentials to join the discussion!

Thank you for your engagement in the AoF over the years! We look forward to offering you a more enhanced forum experience. (5) [Avatar] Offline

I try to explain my problem.

I work in a small startup, we work in Energy management system in the cloud, we receive millions of data every five minutes.

Now we have the typical system with queues and workers, and this is not scalable.

I have designed two topologies :

- One, Accumulate Topology, receive data from a RabbitMQ, one layer of bolts accumulate data grouped by some field, and every minute with a tick tuples the grouped data is passed to another layers that perform some operations an change the stream ... finally I have two outputs, one a persistor to a nosqldatabase, and the other to another topology.

- Second topology, an Alert Topology, from the output of the Accumulate Topology I have to check if I must to trigger some alerts doing some operations on streams of data , this is a complex topology with many layers and bolts.

My questions are :

- What is the best approach to connect this two topologies ? Could I connect them sending tuples from first topology to a RbbitMQ and consume them from the second? or maybe connect directly an create a spout from the last bolt of first topology? or maybe merge all in one topology?

I hope heard from you soon,

matthew.jankowski (17) [Avatar] Offline
Re: Complex topology
Sorry for the delayed response. Sean can probably provide more detail, but we would normally use a queue in such instances. We find that queues result in a nice decoupling between topologies.
sean.allen (19) [Avatar] Offline
Re: Complex topology
As you are already using rabbitmq, I would suggest using it to communicate between the 2 topologies. It is, in your case, the simplest solution and quite likely, the optimal one as well. (5) [Avatar] Offline
Re: Complex topology