Use pv with find -exec
I use the following command to verify ~700 GiB of backed-up files:
$ find -type f -exec md5sum {} + | sort > ~/checksums
This takes many hours, so I would like to integrate pv into the command to show the progress.
I could do this:
$ find -type f -exec pv {} + | md5sum
But it concatenates all of the files, resulting in only one checksum.
So how could I include pv and still get a text file full of checksums at the end?
find checksum pv
asked Jan 19 at 12:01
EmmaVEmmaV
1,1581332
add a comment |
I use the following command to verify ~700 GiB of backed-up files:
$ find -type f -exec md5sum {} + | sort > ~/checksums
This takes many hours, so I would like to integrate pv into the command to show the progress.
I could do this:
$ find -type f -exec pv {} + | md5sum
But it concatenates all of the files, resulting in only one checksum.
So how could I include pv and still get a text file full of checksums at the end?
find checksum pv
asked Jan 19 at 12:01
EmmaVEmmaV
1,1581332
add a comment |
I use the following command to verify ~700 GiB of backed-up files:
$ find -type f -exec md5sum {} + | sort > ~/checksums
This takes many hours, so I would like to integrate pv into the command to show the progress.
I could do this:
$ find -type f -exec pv {} + | md5sum
But it concatenates all of the files, resulting in only one checksum.
So how could I include pv and still get a text file full of checksums at the end?
find checksum pv
asked Jan 19 at 12:01
EmmaVEmmaV
1,1581332
I use the following command to verify ~700 GiB of backed-up files:
$ find -type f -exec md5sum {} + | sort > ~/checksums
This takes many hours, so I would like to integrate pv into the command to show the progress.
I could do this:
$ find -type f -exec pv {} + | md5sum
But it concatenates all of the files, resulting in only one checksum.
So how could I include pv and still get a text file full of checksums at the end?
find checksum pv
find checksum pv
asked Jan 19 at 12:01
EmmaVEmmaV
1,1581332
asked Jan 19 at 12:01
EmmaVEmmaV
1,1581332
edited Jan 19 at 12:17
EmmaV
asked Jan 19 at 12:01
EmmaVEmmaV
1,1581332
asked Jan 19 at 12:01
EmmaVEmmaV
1,1581332
asked Jan 19 at 12:01
EmmaVEmmaV
1,1581332
1,1581332
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Your first command should not be able to run at all as you can't use a pipe in an -exec like that (this was apparently a typo in the original question).
Instead:
find . -type f -exec md5sum {} + | sort -o ~/checksums
or, with pv,
find . -type f -exec md5sum {} + | pv | sort -o ~/checksums
In both of the above, md5sum would be called with as many pathnames as possible in batches. sort would take the output of find (which is the output of md5sum) and sort it into the given filename. The second variation additionally inserts pv between the find and the sort.
You can't use -exec pv {} | md5sum for individual files as the pipe would need to be embedded in a in-line shell script that you call from -exec for each file. But even the correct -exec sh -c 'pv "$1" | md5sum' sh {} ; would discard the filename from the generated md5sum output, so that can't be used either. The pv utility acts like a drop-in replacement for cat.
answered Jan 19 at 12:05
KusalanandaKusalananda
127k16239393
I wrote the first command wrong. I'll change it.
– EmmaV
Jan 19 at 12:17
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f495440%2fuse-pv-with-find-exec%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Your first command should not be able to run at all as you can't use a pipe in an -exec like that (this was apparently a typo in the original question).
Instead:
find . -type f -exec md5sum {} + | sort -o ~/checksums
or, with pv,
find . -type f -exec md5sum {} + | pv | sort -o ~/checksums
In both of the above, md5sum would be called with as many pathnames as possible in batches. sort would take the output of find (which is the output of md5sum) and sort it into the given filename. The second variation additionally inserts pv between the find and the sort.
You can't use -exec pv {} | md5sum for individual files as the pipe would need to be embedded in a in-line shell script that you call from -exec for each file. But even the correct -exec sh -c 'pv "$1" | md5sum' sh {} ; would discard the filename from the generated md5sum output, so that can't be used either. The pv utility acts like a drop-in replacement for cat.
answered Jan 19 at 12:05
KusalanandaKusalananda
127k16239393
I wrote the first command wrong. I'll change it.
– EmmaV
Jan 19 at 12:17
add a comment |
Your first command should not be able to run at all as you can't use a pipe in an -exec like that (this was apparently a typo in the original question).
Instead:
find . -type f -exec md5sum {} + | sort -o ~/checksums
or, with pv,
find . -type f -exec md5sum {} + | pv | sort -o ~/checksums
In both of the above, md5sum would be called with as many pathnames as possible in batches. sort would take the output of find (which is the output of md5sum) and sort it into the given filename. The second variation additionally inserts pv between the find and the sort.
You can't use -exec pv {} | md5sum for individual files as the pipe would need to be embedded in a in-line shell script that you call from -exec for each file. But even the correct -exec sh -c 'pv "$1" | md5sum' sh {} ; would discard the filename from the generated md5sum output, so that can't be used either. The pv utility acts like a drop-in replacement for cat.
answered Jan 19 at 12:05
KusalanandaKusalananda
127k16239393
I wrote the first command wrong. I'll change it.
– EmmaV
Jan 19 at 12:17
add a comment |
Your first command should not be able to run at all as you can't use a pipe in an -exec like that (this was apparently a typo in the original question).
Instead:
find . -type f -exec md5sum {} + | sort -o ~/checksums
or, with pv,
find . -type f -exec md5sum {} + | pv | sort -o ~/checksums
In both of the above, md5sum would be called with as many pathnames as possible in batches. sort would take the output of find (which is the output of md5sum) and sort it into the given filename. The second variation additionally inserts pv between the find and the sort.
You can't use -exec pv {} | md5sum for individual files as the pipe would need to be embedded in a in-line shell script that you call from -exec for each file. But even the correct -exec sh -c 'pv "$1" | md5sum' sh {} ; would discard the filename from the generated md5sum output, so that can't be used either. The pv utility acts like a drop-in replacement for cat.
answered Jan 19 at 12:05
KusalanandaKusalananda
127k16239393
Your first command should not be able to run at all as you can't use a pipe in an -exec like that (this was apparently a typo in the original question).
Instead:
find . -type f -exec md5sum {} + | sort -o ~/checksums
or, with pv,
find . -type f -exec md5sum {} + | pv | sort -o ~/checksums
In both of the above, md5sum would be called with as many pathnames as possible in batches. sort would take the output of find (which is the output of md5sum) and sort it into the given filename. The second variation additionally inserts pv between the find and the sort.
You can't use -exec pv {} | md5sum for individual files as the pipe would need to be embedded in a in-line shell script that you call from -exec for each file. But even the correct -exec sh -c 'pv "$1" | md5sum' sh {} ; would discard the filename from the generated md5sum output, so that can't be used either. The pv utility acts like a drop-in replacement for cat.
answered Jan 19 at 12:05
KusalanandaKusalananda
127k16239393
edited Jan 19 at 12:32
answered Jan 19 at 12:05
KusalanandaKusalananda
127k16239393
answered Jan 19 at 12:05
KusalanandaKusalananda
127k16239393
answered Jan 19 at 12:05
KusalanandaKusalananda
127k16239393
127k16239393
I wrote the first command wrong. I'll change it.
– EmmaV
Jan 19 at 12:17
add a comment |
I wrote the first command wrong. I'll change it.
– EmmaV
Jan 19 at 12:17
I wrote the first command wrong. I'll change it.
– EmmaV
Jan 19 at 12:17
I wrote the first command wrong. I'll change it.
– EmmaV
Jan 19 at 12:17
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f495440%2fuse-pv-with-find-exec%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
