The Author Online Book Forums are Moving

The Author Online Book Forums will soon redirect to Manning's liveBook and liveVideo. All book forum content will migrate to liveBook's discussion forum and all video forum content will migrate to liveVideo. Log in to liveBook or liveVideo with your Manning credentials to join the discussion!

Thank you for your engagement in the AoF over the years! We look forward to offering you a more enhanced forum experience.

oarmand (11) [Avatar] Offline
#1
v6 of the MEAP edition contains p.161 in the resource contention chapter contains a note about contentions at the OS level, suggesting either virtualization or single worker per node configurations.

An alternative way may come from resource management frameworks, either the one provided by YARN or by container-based execution environment such as Docker or Apache Mesos, to at least control allocated memory and CPU resources.

As anyone already tried one of these ones and has feedback to provide?
sean.allen (19) [Avatar] Offline
#2
Re: YARN/containter-based resource management
Storm w/ Yarn is still rather new and we aren't covering it in the book.

Running Storm inside of docker in no way sidesteps issues with resource contention.

If you run multiple virtualized machines on a single host machine, you can have contention for resources on the host machine.

If you run multiple docker instances inside a host (be it virtualized itself or not), you are still going to have resource contention issues at the host level. Docker containers share resources.

The single worker process per OS is a way to sidestep contention for OS resources by different topologies. Its primitive, simple and saves you a ton of overhead assuming you can afford the "wasted" resources.