albert.ortizl@gmail.com (5) [Avatar] Offline
#1
Hello,

I try to explain my problem.

I work in a small startup, we work in Energy management system in the cloud, we receive millions of data every five minutes.

Now we have the typical system with queues and workers, and this is not scalable.

I have designed two topologies :

- One, Accumulate Topology, receive data from a RabbitMQ, one layer of bolts accumulate data grouped by some field, and every minute with a tick tuples the grouped data is passed to another layers that perform some operations an change the stream ... finally I have two outputs, one a persistor to a nosqldatabase, and the other to another topology.

- Second topology, an Alert Topology, from the output of the Accumulate Topology I have to check if I must to trigger some alerts doing some operations on streams of data , this is a complex topology with many layers and bolts.

My questions are :

- What is the best approach to connect this two topologies ? Could I connect them sending tuples from first topology to a RbbitMQ and consume them from the second? or maybe connect directly an create a spout from the last bolt of first topology? or maybe merge all in one topology?

I hope heard from you soon,

Thanks,
matthew.jankowski (17) [Avatar] Offline
#2
Re: Complex topology
Sorry for the delayed response. Sean can probably provide more detail, but we would normally use a queue in such instances. We find that queues result in a nice decoupling between topologies.
sean.allen (19) [Avatar] Offline
#3
Re: Complex topology
As you are already using rabbitmq, I would suggest using it to communicate between the 2 topologies. It is, in your case, the simplest solution and quite likely, the optimal one as well.
albert.ortizl@gmail.com (5) [Avatar] Offline
#4
Re: Complex topology
Thanks!