How do I asynchronously calculate the checksum of a file on the hard disk in Rust?
I have a TCP file server in Rust / Tokio stack.
When a client is uploading a file, the data is being read from a tokio::net::TcpStream and written to a futures_fs::FsWriteSink, which has been started on a separate futures_fs::FsPool.
When the file is completely uploaded, I need to check its consistency by checking its checksum against the one sent by the client.
What is the easiest way to asynchronously calculate the checksum, especially if the file does not fit into RAM?
file asynchronous rust checksum rust-tokio
edited Nov 21 '18 at 14:53
Shepmaster
158k15322462
asked Nov 21 '18 at 13:17
hedgar2017hedgar2017
3741824
add a comment |
I have a TCP file server in Rust / Tokio stack.
When a client is uploading a file, the data is being read from a tokio::net::TcpStream and written to a futures_fs::FsWriteSink, which has been started on a separate futures_fs::FsPool.
When the file is completely uploaded, I need to check its consistency by checking its checksum against the one sent by the client.
What is the easiest way to asynchronously calculate the checksum, especially if the file does not fit into RAM?
file asynchronous rust checksum rust-tokio
edited Nov 21 '18 at 14:53
Shepmaster
158k15322462
asked Nov 21 '18 at 13:17
hedgar2017hedgar2017
3741824
Using aBufReader?
– hellow
Nov 21 '18 at 13:19
2
You might also want to explore calculating the checksum for the actual data uploaded as it's being uploaded. You already have the data passing through your processing at that point, so calculating the checksum then is almost a free operation. This is especially true if you have performance considerations, because by calculating the checksum off the data after it's been saved to disk you're effectively doubling the IO operations you need to do to support file uploading.
– Andrew Henle
Nov 21 '18 at 13:49
Yeah, I totally agree. But it seems that I need to implement such thing myself, doesn't it? I'd like to have something like anotherSink, where I could feed the chunks so theSinkcould calculate the checksum on the fly.
– hedgar2017
Nov 21 '18 at 14:47
What checksum? Is the implementation of the checksum naturally asynchronous? If so, just use it. If it's not, then this is a duplicate of What is the best approach to encapsulate blocking I/O in future-rs?.
– Shepmaster
Nov 21 '18 at 17:59
add a comment |
I have a TCP file server in Rust / Tokio stack.
When a client is uploading a file, the data is being read from a tokio::net::TcpStream and written to a futures_fs::FsWriteSink, which has been started on a separate futures_fs::FsPool.
When the file is completely uploaded, I need to check its consistency by checking its checksum against the one sent by the client.
What is the easiest way to asynchronously calculate the checksum, especially if the file does not fit into RAM?
file asynchronous rust checksum rust-tokio
edited Nov 21 '18 at 14:53
Shepmaster
158k15322462
asked Nov 21 '18 at 13:17
hedgar2017hedgar2017
3741824
I have a TCP file server in Rust / Tokio stack.
When a client is uploading a file, the data is being read from a tokio::net::TcpStream and written to a futures_fs::FsWriteSink, which has been started on a separate futures_fs::FsPool.
When the file is completely uploaded, I need to check its consistency by checking its checksum against the one sent by the client.
What is the easiest way to asynchronously calculate the checksum, especially if the file does not fit into RAM?
file asynchronous rust checksum rust-tokio
file asynchronous rust checksum rust-tokio
edited Nov 21 '18 at 14:53
Shepmaster
158k15322462
asked Nov 21 '18 at 13:17
hedgar2017hedgar2017
3741824
edited Nov 21 '18 at 14:53
Shepmaster
158k15322462
asked Nov 21 '18 at 13:17
hedgar2017hedgar2017
3741824
edited Nov 21 '18 at 14:53
Shepmaster
158k15322462
edited Nov 21 '18 at 14:53
Shepmaster
158k15322462
edited Nov 21 '18 at 14:53
Shepmaster
158k15322462
158k15322462
asked Nov 21 '18 at 13:17
hedgar2017hedgar2017
3741824
asked Nov 21 '18 at 13:17
hedgar2017hedgar2017
3741824
asked Nov 21 '18 at 13:17
hedgar2017hedgar2017
3741824
3741824
Using aBufReader?
– hellow
Nov 21 '18 at 13:19
2
You might also want to explore calculating the checksum for the actual data uploaded as it's being uploaded. You already have the data passing through your processing at that point, so calculating the checksum then is almost a free operation. This is especially true if you have performance considerations, because by calculating the checksum off the data after it's been saved to disk you're effectively doubling the IO operations you need to do to support file uploading.
– Andrew Henle
Nov 21 '18 at 13:49
Yeah, I totally agree. But it seems that I need to implement such thing myself, doesn't it? I'd like to have something like anotherSink, where I could feed the chunks so theSinkcould calculate the checksum on the fly.
– hedgar2017
Nov 21 '18 at 14:47
What checksum? Is the implementation of the checksum naturally asynchronous? If so, just use it. If it's not, then this is a duplicate of What is the best approach to encapsulate blocking I/O in future-rs?.
– Shepmaster
Nov 21 '18 at 17:59
add a comment |
Using aBufReader?
– hellow
Nov 21 '18 at 13:19
2
You might also want to explore calculating the checksum for the actual data uploaded as it's being uploaded. You already have the data passing through your processing at that point, so calculating the checksum then is almost a free operation. This is especially true if you have performance considerations, because by calculating the checksum off the data after it's been saved to disk you're effectively doubling the IO operations you need to do to support file uploading.
– Andrew Henle
Nov 21 '18 at 13:49
Yeah, I totally agree. But it seems that I need to implement such thing myself, doesn't it? I'd like to have something like anotherSink, where I could feed the chunks so theSinkcould calculate the checksum on the fly.
– hedgar2017
Nov 21 '18 at 14:47
What checksum? Is the implementation of the checksum naturally asynchronous? If so, just use it. If it's not, then this is a duplicate of What is the best approach to encapsulate blocking I/O in future-rs?.
– Shepmaster
Nov 21 '18 at 17:59
Using a
BufReader?– hellow
Nov 21 '18 at 13:19
Using a
BufReader?– hellow
Nov 21 '18 at 13:19
2
2
You might also want to explore calculating the checksum for the actual data uploaded as it's being uploaded. You already have the data passing through your processing at that point, so calculating the checksum then is almost a free operation. This is especially true if you have performance considerations, because by calculating the checksum off the data after it's been saved to disk you're effectively doubling the IO operations you need to do to support file uploading.
– Andrew Henle
Nov 21 '18 at 13:49
You might also want to explore calculating the checksum for the actual data uploaded as it's being uploaded. You already have the data passing through your processing at that point, so calculating the checksum then is almost a free operation. This is especially true if you have performance considerations, because by calculating the checksum off the data after it's been saved to disk you're effectively doubling the IO operations you need to do to support file uploading.
– Andrew Henle
Nov 21 '18 at 13:49
Yeah, I totally agree. But it seems that I need to implement such thing myself, doesn't it? I'd like to have something like another
Sink, where I could feed the chunks so the Sink could calculate the checksum on the fly.– hedgar2017
Nov 21 '18 at 14:47
Yeah, I totally agree. But it seems that I need to implement such thing myself, doesn't it? I'd like to have something like another
Sink, where I could feed the chunks so the Sink could calculate the checksum on the fly.– hedgar2017
Nov 21 '18 at 14:47
What checksum? Is the implementation of the checksum naturally asynchronous? If so, just use it. If it's not, then this is a duplicate of What is the best approach to encapsulate blocking I/O in future-rs?.
– Shepmaster
Nov 21 '18 at 17:59
What checksum? Is the implementation of the checksum naturally asynchronous? If so, just use it. If it's not, then this is a duplicate of What is the best approach to encapsulate blocking I/O in future-rs?.
– Shepmaster
Nov 21 '18 at 17:59
add a comment |
2 Answers
2
active
oldest
votes
It depends on what checksum algorithm you want to use, but using the md5 crate as an example, you can compute the checksum on the fly. Something like this should do it:
// When starting the file transfer
let mut md5_context = md5::Context::new();
// ...
// as part of your existing processing for each block of data
md5_context.consume (&block);
// ...
// once the last block has been processed
return md5_context.compute();
answered Nov 21 '18 at 15:04
JmbJmb
2,831523
1
This appears to be completely synchronous. The OP has explicitly requested an asynchronous solution.
– Shepmaster
Nov 21 '18 at 15:11
1
@Shepmaster I don't see how you can tell whether this is synchronous or asynchronous: I only said that for each block of data he should callmd5_context.consume, I never said that he should do it in a synchronous loop.
– Jmb
Nov 21 '18 at 15:46
1
@Shepmaster edited the answer to make it clearer
– Jmb
Nov 21 '18 at 15:50
add a comment |
Actually, making simple hashing algorithms asynchronous is in such cases somewhat redundant, as long as one MD5 calculation takes less then 1 us (about 500 ns).
But, a new blocking API is now available in tokio. It allows executing blocking or CPU heavy operations using internal threading mechanisms.
answered Nov 22 '18 at 20:56
hedgar2017hedgar2017
3741824
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53412922%2fhow-do-i-asynchronously-calculate-the-checksum-of-a-file-on-the-hard-disk-in-rus%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
It depends on what checksum algorithm you want to use, but using the md5 crate as an example, you can compute the checksum on the fly. Something like this should do it:
// When starting the file transfer
let mut md5_context = md5::Context::new();
// ...
// as part of your existing processing for each block of data
md5_context.consume (&block);
// ...
// once the last block has been processed
return md5_context.compute();
answered Nov 21 '18 at 15:04
JmbJmb
2,831523
1
This appears to be completely synchronous. The OP has explicitly requested an asynchronous solution.
– Shepmaster
Nov 21 '18 at 15:11
1
@Shepmaster I don't see how you can tell whether this is synchronous or asynchronous: I only said that for each block of data he should callmd5_context.consume, I never said that he should do it in a synchronous loop.
– Jmb
Nov 21 '18 at 15:46
1
@Shepmaster edited the answer to make it clearer
– Jmb
Nov 21 '18 at 15:50
add a comment |
It depends on what checksum algorithm you want to use, but using the md5 crate as an example, you can compute the checksum on the fly. Something like this should do it:
// When starting the file transfer
let mut md5_context = md5::Context::new();
// ...
// as part of your existing processing for each block of data
md5_context.consume (&block);
// ...
// once the last block has been processed
return md5_context.compute();
answered Nov 21 '18 at 15:04
JmbJmb
2,831523
1
This appears to be completely synchronous. The OP has explicitly requested an asynchronous solution.
– Shepmaster
Nov 21 '18 at 15:11
1
@Shepmaster I don't see how you can tell whether this is synchronous or asynchronous: I only said that for each block of data he should callmd5_context.consume, I never said that he should do it in a synchronous loop.
– Jmb
Nov 21 '18 at 15:46
1
@Shepmaster edited the answer to make it clearer
– Jmb
Nov 21 '18 at 15:50
add a comment |
It depends on what checksum algorithm you want to use, but using the md5 crate as an example, you can compute the checksum on the fly. Something like this should do it:
// When starting the file transfer
let mut md5_context = md5::Context::new();
// ...
// as part of your existing processing for each block of data
md5_context.consume (&block);
// ...
// once the last block has been processed
return md5_context.compute();
answered Nov 21 '18 at 15:04
JmbJmb
2,831523
It depends on what checksum algorithm you want to use, but using the md5 crate as an example, you can compute the checksum on the fly. Something like this should do it:
// When starting the file transfer
let mut md5_context = md5::Context::new();
// ...
// as part of your existing processing for each block of data
md5_context.consume (&block);
// ...
// once the last block has been processed
return md5_context.compute();
answered Nov 21 '18 at 15:04
JmbJmb
2,831523
edited Nov 21 '18 at 15:49
answered Nov 21 '18 at 15:04
JmbJmb
2,831523
answered Nov 21 '18 at 15:04
JmbJmb
2,831523
answered Nov 21 '18 at 15:04
JmbJmb
2,831523
2,831523
1
This appears to be completely synchronous. The OP has explicitly requested an asynchronous solution.
– Shepmaster
Nov 21 '18 at 15:11
1
@Shepmaster I don't see how you can tell whether this is synchronous or asynchronous: I only said that for each block of data he should callmd5_context.consume, I never said that he should do it in a synchronous loop.
– Jmb
Nov 21 '18 at 15:46
1
@Shepmaster edited the answer to make it clearer
– Jmb
Nov 21 '18 at 15:50
add a comment |
1
This appears to be completely synchronous. The OP has explicitly requested an asynchronous solution.
– Shepmaster
Nov 21 '18 at 15:11
1
@Shepmaster I don't see how you can tell whether this is synchronous or asynchronous: I only said that for each block of data he should callmd5_context.consume, I never said that he should do it in a synchronous loop.
– Jmb
Nov 21 '18 at 15:46
1
@Shepmaster edited the answer to make it clearer
– Jmb
Nov 21 '18 at 15:50
1
1
This appears to be completely synchronous. The OP has explicitly requested an asynchronous solution.
– Shepmaster
Nov 21 '18 at 15:11
This appears to be completely synchronous. The OP has explicitly requested an asynchronous solution.
– Shepmaster
Nov 21 '18 at 15:11
1
1
@Shepmaster I don't see how you can tell whether this is synchronous or asynchronous: I only said that for each block of data he should call
md5_context.consume, I never said that he should do it in a synchronous loop.– Jmb
Nov 21 '18 at 15:46
@Shepmaster I don't see how you can tell whether this is synchronous or asynchronous: I only said that for each block of data he should call
md5_context.consume, I never said that he should do it in a synchronous loop.– Jmb
Nov 21 '18 at 15:46
1
1
@Shepmaster edited the answer to make it clearer
– Jmb
Nov 21 '18 at 15:50
@Shepmaster edited the answer to make it clearer
– Jmb
Nov 21 '18 at 15:50
add a comment |
Actually, making simple hashing algorithms asynchronous is in such cases somewhat redundant, as long as one MD5 calculation takes less then 1 us (about 500 ns).
But, a new blocking API is now available in tokio. It allows executing blocking or CPU heavy operations using internal threading mechanisms.
answered Nov 22 '18 at 20:56
hedgar2017hedgar2017
3741824
add a comment |
Actually, making simple hashing algorithms asynchronous is in such cases somewhat redundant, as long as one MD5 calculation takes less then 1 us (about 500 ns).
But, a new blocking API is now available in tokio. It allows executing blocking or CPU heavy operations using internal threading mechanisms.
answered Nov 22 '18 at 20:56
hedgar2017hedgar2017
3741824
add a comment |
Actually, making simple hashing algorithms asynchronous is in such cases somewhat redundant, as long as one MD5 calculation takes less then 1 us (about 500 ns).
But, a new blocking API is now available in tokio. It allows executing blocking or CPU heavy operations using internal threading mechanisms.
answered Nov 22 '18 at 20:56
hedgar2017hedgar2017
3741824
Actually, making simple hashing algorithms asynchronous is in such cases somewhat redundant, as long as one MD5 calculation takes less then 1 us (about 500 ns).
But, a new blocking API is now available in tokio. It allows executing blocking or CPU heavy operations using internal threading mechanisms.
answered Nov 22 '18 at 20:56
hedgar2017hedgar2017
3741824
edited Feb 23 at 12:05
answered Nov 22 '18 at 20:56
hedgar2017hedgar2017
3741824
answered Nov 22 '18 at 20:56
hedgar2017hedgar2017
3741824
answered Nov 22 '18 at 20:56
hedgar2017hedgar2017
3741824
3741824
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53412922%2fhow-do-i-asynchronously-calculate-the-checksum-of-a-file-on-the-hard-disk-in-rus%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Using a
BufReader?– hellow
Nov 21 '18 at 13:19
2
You might also want to explore calculating the checksum for the actual data uploaded as it's being uploaded. You already have the data passing through your processing at that point, so calculating the checksum then is almost a free operation. This is especially true if you have performance considerations, because by calculating the checksum off the data after it's been saved to disk you're effectively doubling the IO operations you need to do to support file uploading.
– Andrew Henle
Nov 21 '18 at 13:49
Yeah, I totally agree. But it seems that I need to implement such thing myself, doesn't it? I'd like to have something like another
Sink, where I could feed the chunks so theSinkcould calculate the checksum on the fly.– hedgar2017
Nov 21 '18 at 14:47
What checksum? Is the implementation of the checksum naturally asynchronous? If so, just use it. If it's not, then this is a duplicate of What is the best approach to encapsulate blocking I/O in future-rs?.
– Shepmaster
Nov 21 '18 at 17:59