The Author Online Book Forums are Moving

The Author Online Book Forums will soon redirect to Manning's liveBook and liveVideo. All book forum content will migrate to liveBook's discussion forum and all video forum content will migrate to liveVideo. Log in to liveBook or liveVideo with your Manning credentials to join the discussion!

Thank you for your engagement in the AoF over the years! We look forward to offering you a more enhanced forum experience.

jfcha (1) [Avatar] Offline
The book is great so far. Thanks for doing this.

I'm building a streaming stock prediction application and I'm looking to use Kafka Streams. Is there a good way to build a best effort stream with Kafka Streams? My inputs come in relatively fast and my prediction algorithm is relatively slow, so what I want is for the stream to take the latest message, run the calculation, and then, whenever it finishes, take the new latest message from the partition and begin again. For my particular use case, completeness isn't important, but timeliness (minus the lag for the actual calculation) is. In essence, I'd like to do something like a consumer's seekToEnd call at the end of each process call. Any suggestions on how to approach this with streams?

Bill Bejeck (47) [Avatar] Offline

My apologies for the very delayed response.

If I understand your question, what you'll need to do is process each record and forward to another processor that uses a state store. The first processor will perform the calculation on each record and forward to the processor with the state store.

In the second processor, you will update records in the state store (newer records with the same key will update older records with the same key). But you only emit records by scheduling a punctuation call, that way it will be a best effort latest record that gets emitted.

Let me know if you have any other questions.