jfcha (1) [Avatar] Offline
The book is great so far. Thanks for doing this.

I'm building a streaming stock prediction application and I'm looking to use Kafka Streams. Is there a good way to build a best effort stream with Kafka Streams? My inputs come in relatively fast and my prediction algorithm is relatively slow, so what I want is for the stream to take the latest message, run the calculation, and then, whenever it finishes, take the new latest message from the partition and begin again. For my particular use case, completeness isn't important, but timeliness (minus the lag for the actual calculation) is. In essence, I'd like to do something like a consumer's seekToEnd call at the end of each process call. Any suggestions on how to approach this with streams?

Bill Bejeck (44) [Avatar] Offline

My apologies for the very delayed response.

If I understand your question, what you'll need to do is process each record and forward to another processor that uses a state store. The first processor will perform the calculation on each record and forward to the processor with the state store.

In the second processor, you will update records in the state store (newer records with the same key will update older records with the same key). But you only emit records by scheduling a punctuation call, that way it will be a best effort latest record that gets emitted.

Let me know if you have any other questions.