Kafka Streams: How to force quicker updates on stores
I have an aggregated store via DSL via windowed aggregation over a stream.
Then, I get a stream from it via toStream, do some small transformation and send to another topic.
In my tests, from the publishing time to the time when the data is in the destination stream, it may well spend 10-20 seconds. Which makes my tests having to wait for it that long.
How can I reduce this waiting time so computations are triggered faster in my tests?
Which settings are involved?
I tried setting commit.interval.ms to 100 but it is of no help.
apache-kafka apache-kafka-streams
asked Nov 21 '18 at 15:18
xmarxmar
475314
add a comment |
I have an aggregated store via DSL via windowed aggregation over a stream.
Then, I get a stream from it via toStream, do some small transformation and send to another topic.
In my tests, from the publishing time to the time when the data is in the destination stream, it may well spend 10-20 seconds. Which makes my tests having to wait for it that long.
How can I reduce this waiting time so computations are triggered faster in my tests?
Which settings are involved?
I tried setting commit.interval.ms to 100 but it is of no help.
apache-kafka apache-kafka-streams
asked Nov 21 '18 at 15:18
xmarxmar
475314
Reducing the commit interval should actually work... You can also try to disable caching by setting the cache size to zero in the config. Hope this helps.
– Matthias J. Sax
Nov 21 '18 at 17:52
I tried setting cache to zero. However, I see messages coming as null downstream in the topoogy in stages after the.toStream()- where they weren't before. I'll investigate, but any hint on this?
– xmar
Nov 22 '18 at 11:27
Not sure exactly what computation you do. But<key,null>messages indicate tombstones (ie, delete on the KTable). With caching, those might get deduplicate?
– Matthias J. Sax
Nov 23 '18 at 5:11
add a comment |
I have an aggregated store via DSL via windowed aggregation over a stream.
Then, I get a stream from it via toStream, do some small transformation and send to another topic.
In my tests, from the publishing time to the time when the data is in the destination stream, it may well spend 10-20 seconds. Which makes my tests having to wait for it that long.
How can I reduce this waiting time so computations are triggered faster in my tests?
Which settings are involved?
I tried setting commit.interval.ms to 100 but it is of no help.
apache-kafka apache-kafka-streams
asked Nov 21 '18 at 15:18
xmarxmar
475314
I have an aggregated store via DSL via windowed aggregation over a stream.
Then, I get a stream from it via toStream, do some small transformation and send to another topic.
In my tests, from the publishing time to the time when the data is in the destination stream, it may well spend 10-20 seconds. Which makes my tests having to wait for it that long.
How can I reduce this waiting time so computations are triggered faster in my tests?
Which settings are involved?
I tried setting commit.interval.ms to 100 but it is of no help.
apache-kafka apache-kafka-streams
apache-kafka apache-kafka-streams
asked Nov 21 '18 at 15:18
xmarxmar
475314
asked Nov 21 '18 at 15:18
xmarxmar
475314
asked Nov 21 '18 at 15:18
xmarxmar
475314
asked Nov 21 '18 at 15:18
xmarxmar
475314
asked Nov 21 '18 at 15:18
xmarxmar
475314
475314
Reducing the commit interval should actually work... You can also try to disable caching by setting the cache size to zero in the config. Hope this helps.
– Matthias J. Sax
Nov 21 '18 at 17:52
I tried setting cache to zero. However, I see messages coming as null downstream in the topoogy in stages after the.toStream()- where they weren't before. I'll investigate, but any hint on this?
– xmar
Nov 22 '18 at 11:27
Not sure exactly what computation you do. But<key,null>messages indicate tombstones (ie, delete on the KTable). With caching, those might get deduplicate?
– Matthias J. Sax
Nov 23 '18 at 5:11
add a comment |
Reducing the commit interval should actually work... You can also try to disable caching by setting the cache size to zero in the config. Hope this helps.
– Matthias J. Sax
Nov 21 '18 at 17:52
I tried setting cache to zero. However, I see messages coming as null downstream in the topoogy in stages after the.toStream()- where they weren't before. I'll investigate, but any hint on this?
– xmar
Nov 22 '18 at 11:27
Not sure exactly what computation you do. But<key,null>messages indicate tombstones (ie, delete on the KTable). With caching, those might get deduplicate?
– Matthias J. Sax
Nov 23 '18 at 5:11
Reducing the commit interval should actually work... You can also try to disable caching by setting the cache size to zero in the config. Hope this helps.
– Matthias J. Sax
Nov 21 '18 at 17:52
Reducing the commit interval should actually work... You can also try to disable caching by setting the cache size to zero in the config. Hope this helps.
– Matthias J. Sax
Nov 21 '18 at 17:52
I tried setting cache to zero. However, I see messages coming as null downstream in the topoogy in stages after the
.toStream() - where they weren't before. I'll investigate, but any hint on this?– xmar
Nov 22 '18 at 11:27
I tried setting cache to zero. However, I see messages coming as null downstream in the topoogy in stages after the
.toStream() - where they weren't before. I'll investigate, but any hint on this?– xmar
Nov 22 '18 at 11:27
Not sure exactly what computation you do. But
<key,null> messages indicate tombstones (ie, delete on the KTable). With caching, those might get deduplicate?– Matthias J. Sax
Nov 23 '18 at 5:11
Not sure exactly what computation you do. But
<key,null> messages indicate tombstones (ie, delete on the KTable). With caching, those might get deduplicate?– Matthias J. Sax
Nov 23 '18 at 5:11
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53415184%2fkafka-streams-how-to-force-quicker-updates-on-stores%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53415184%2fkafka-streams-how-to-force-quicker-updates-on-stores%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Reducing the commit interval should actually work... You can also try to disable caching by setting the cache size to zero in the config. Hope this helps.
– Matthias J. Sax
Nov 21 '18 at 17:52
I tried setting cache to zero. However, I see messages coming as null downstream in the topoogy in stages after the
.toStream()- where they weren't before. I'll investigate, but any hint on this?– xmar
Nov 22 '18 at 11:27
Not sure exactly what computation you do. But
<key,null>messages indicate tombstones (ie, delete on the KTable). With caching, those might get deduplicate?– Matthias J. Sax
Nov 23 '18 at 5:11