How to run commands in the shell in parallel without disrupting the output?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}
2
I have the following shell script:
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.2 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.5 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 0.1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 0.2 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 0.5 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 0.1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 0.2 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 0.5 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 1
And my main function has a structure like this:
int main(int argc, char** argv)
{
if(argc<18)
{
cout<<"Insufficient parameters"<<endl;
cout<<"loop #ofGen popSize chrLen Pc PmNumerator randPopRate BOAimmigrantsRate Pn algoType #ofBOAsamples mkpfileNo noiseType prb env per sev"<<endl;
exit(1);
}
int algoType;// GA or PBIL
int mkpfile;
loop = atoi(argv[1]);
GA.generation = atoi(argv[2]);
GA.popSize = atoi(argv[3]);
GA.chromosomeLength = atoi(argv[4]);
GA.Pc = atof(argv[5]);
GA.PmNumerator = atoi(argv[6]);
GA.randomPopulationRate = atof(argv[7]);
GA.ImmigrantRateFromBOA = atof(argv[8]);
DE.Pn = atof(argv[9]);
algoType = atoi(argv[10]);
CM.numOfSamples = atoi(argv[11]);
mkpfile=atoi(argv[12]);
DE.noiseType=atoi(argv[13]);
DE.problemType=atoi(argv[14]);
DE.environmentType = atoi(argv[15]);
DE.period=atoi(argv[16]);
DE.severity=atof(argv[17]);
printf("nRunning... Problem type: %d...",DE.problemType);
fflush(stdout);
for(int i=1; i<=loop; i++)
{
myAlgorithm(i,DE.problemType,algoType,mkpfile);
}
cout<<"Done!"<<endl;
return 0;
}
When I run the above code, I want output to first prints the printf() part which doesn't have newline thing, and then prints the cout part:
Running bla bla bla... Done!
It runs correctly if I run only one test case, however when I use shell script to run more than one test cases in parallel, it becomes like this:
Running... Problem type: 1...Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 2...
Running... Problem type: 2...
Running... Problem type: 2...
Running... Problem type: 2...Done!
Done!
Done!
Running... Problem type: 2...
Running... Problem type: 2...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 2...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 2...Running... Problem type: 3...
Running... Problem type: 3...Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Is there way to make that happen properly? I am asking this here since I think it is an Ubuntu-related problem.
command-line scripts
edited Feb 21 at 9:47
dessert
25.6k674108
asked Feb 21 at 7:44
WhoCaresWhoCares
203
add a comment |
2
I have the following shell script:
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.2 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.5 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 0.1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 0.2 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 0.5 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 0.1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 0.2 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 0.5 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 1
And my main function has a structure like this:
int main(int argc, char** argv)
{
if(argc<18)
{
cout<<"Insufficient parameters"<<endl;
cout<<"loop #ofGen popSize chrLen Pc PmNumerator randPopRate BOAimmigrantsRate Pn algoType #ofBOAsamples mkpfileNo noiseType prb env per sev"<<endl;
exit(1);
}
int algoType;// GA or PBIL
int mkpfile;
loop = atoi(argv[1]);
GA.generation = atoi(argv[2]);
GA.popSize = atoi(argv[3]);
GA.chromosomeLength = atoi(argv[4]);
GA.Pc = atof(argv[5]);
GA.PmNumerator = atoi(argv[6]);
GA.randomPopulationRate = atof(argv[7]);
GA.ImmigrantRateFromBOA = atof(argv[8]);
DE.Pn = atof(argv[9]);
algoType = atoi(argv[10]);
CM.numOfSamples = atoi(argv[11]);
mkpfile=atoi(argv[12]);
DE.noiseType=atoi(argv[13]);
DE.problemType=atoi(argv[14]);
DE.environmentType = atoi(argv[15]);
DE.period=atoi(argv[16]);
DE.severity=atof(argv[17]);
printf("nRunning... Problem type: %d...",DE.problemType);
fflush(stdout);
for(int i=1; i<=loop; i++)
{
myAlgorithm(i,DE.problemType,algoType,mkpfile);
}
cout<<"Done!"<<endl;
return 0;
}
When I run the above code, I want output to first prints the printf() part which doesn't have newline thing, and then prints the cout part:
Running bla bla bla... Done!
It runs correctly if I run only one test case, however when I use shell script to run more than one test cases in parallel, it becomes like this:
Running... Problem type: 1...Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 2...
Running... Problem type: 2...
Running... Problem type: 2...
Running... Problem type: 2...Done!
Done!
Done!
Running... Problem type: 2...
Running... Problem type: 2...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 2...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 2...Running... Problem type: 3...
Running... Problem type: 3...Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Is there way to make that happen properly? I am asking this here since I think it is an Ubuntu-related problem.
command-line scripts
edited Feb 21 at 9:47
dessert
25.6k674108
asked Feb 21 at 7:44
WhoCaresWhoCares
203
2
This question might be better suited for stack overflow (?) that being said (I am sure someone will correct me if I am wrong, as I am far from an expert...) but are you sure you want to be flushing to stdout afterprintf(Running...)? IIRC flushing immediately prints tostdoutor wherever you're printing, so you print immediately then when another instance runs in parallel you print that instantly without waiting for the previous instance to finish the loop
– j-money
Feb 21 at 9:11
1
This is expected behaviour. Each process running in the background (or in parallel as you write) output when the code says to do. If you remove the ampersands from the script, the./runprocesses will run one at the time, and the output will be in correct order.
– Soren A
Feb 21 at 9:36
@j-money OP asks how to get proper output order when running shell commands, that’s not a good fit for SO I think.
– dessert
Feb 21 at 9:45
add a comment |
2
2
2
I have the following shell script:
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.2 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.5 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 0.1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 0.2 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 0.5 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 0.1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 0.2 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 0.5 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 1
And my main function has a structure like this:
int main(int argc, char** argv)
{
if(argc<18)
{
cout<<"Insufficient parameters"<<endl;
cout<<"loop #ofGen popSize chrLen Pc PmNumerator randPopRate BOAimmigrantsRate Pn algoType #ofBOAsamples mkpfileNo noiseType prb env per sev"<<endl;
exit(1);
}
int algoType;// GA or PBIL
int mkpfile;
loop = atoi(argv[1]);
GA.generation = atoi(argv[2]);
GA.popSize = atoi(argv[3]);
GA.chromosomeLength = atoi(argv[4]);
GA.Pc = atof(argv[5]);
GA.PmNumerator = atoi(argv[6]);
GA.randomPopulationRate = atof(argv[7]);
GA.ImmigrantRateFromBOA = atof(argv[8]);
DE.Pn = atof(argv[9]);
algoType = atoi(argv[10]);
CM.numOfSamples = atoi(argv[11]);
mkpfile=atoi(argv[12]);
DE.noiseType=atoi(argv[13]);
DE.problemType=atoi(argv[14]);
DE.environmentType = atoi(argv[15]);
DE.period=atoi(argv[16]);
DE.severity=atof(argv[17]);
printf("nRunning... Problem type: %d...",DE.problemType);
fflush(stdout);
for(int i=1; i<=loop; i++)
{
myAlgorithm(i,DE.problemType,algoType,mkpfile);
}
cout<<"Done!"<<endl;
return 0;
}
When I run the above code, I want output to first prints the printf() part which doesn't have newline thing, and then prints the cout part:
Running bla bla bla... Done!
It runs correctly if I run only one test case, however when I use shell script to run more than one test cases in parallel, it becomes like this:
Running... Problem type: 1...Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 2...
Running... Problem type: 2...
Running... Problem type: 2...
Running... Problem type: 2...Done!
Done!
Done!
Running... Problem type: 2...
Running... Problem type: 2...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 2...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 2...Running... Problem type: 3...
Running... Problem type: 3...Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Is there way to make that happen properly? I am asking this here since I think it is an Ubuntu-related problem.
command-line scripts
edited Feb 21 at 9:47
dessert
25.6k674108
asked Feb 21 at 7:44
WhoCaresWhoCares
203
I have the following shell script:
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.2 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.5 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 0.1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 0.2 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 0.5 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 0.1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 0.2 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 0.5 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 1
And my main function has a structure like this:
int main(int argc, char** argv)
{
if(argc<18)
{
cout<<"Insufficient parameters"<<endl;
cout<<"loop #ofGen popSize chrLen Pc PmNumerator randPopRate BOAimmigrantsRate Pn algoType #ofBOAsamples mkpfileNo noiseType prb env per sev"<<endl;
exit(1);
}
int algoType;// GA or PBIL
int mkpfile;
loop = atoi(argv[1]);
GA.generation = atoi(argv[2]);
GA.popSize = atoi(argv[3]);
GA.chromosomeLength = atoi(argv[4]);
GA.Pc = atof(argv[5]);
GA.PmNumerator = atoi(argv[6]);
GA.randomPopulationRate = atof(argv[7]);
GA.ImmigrantRateFromBOA = atof(argv[8]);
DE.Pn = atof(argv[9]);
algoType = atoi(argv[10]);
CM.numOfSamples = atoi(argv[11]);
mkpfile=atoi(argv[12]);
DE.noiseType=atoi(argv[13]);
DE.problemType=atoi(argv[14]);
DE.environmentType = atoi(argv[15]);
DE.period=atoi(argv[16]);
DE.severity=atof(argv[17]);
printf("nRunning... Problem type: %d...",DE.problemType);
fflush(stdout);
for(int i=1; i<=loop; i++)
{
myAlgorithm(i,DE.problemType,algoType,mkpfile);
}
cout<<"Done!"<<endl;
return 0;
}
When I run the above code, I want output to first prints the printf() part which doesn't have newline thing, and then prints the cout part:
Running bla bla bla... Done!
It runs correctly if I run only one test case, however when I use shell script to run more than one test cases in parallel, it becomes like this:
Running... Problem type: 1...Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 2...
Running... Problem type: 2...
Running... Problem type: 2...
Running... Problem type: 2...Done!
Done!
Done!
Running... Problem type: 2...
Running... Problem type: 2...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 2...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 2...Running... Problem type: 3...
Running... Problem type: 3...Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Is there way to make that happen properly? I am asking this here since I think it is an Ubuntu-related problem.
command-line scripts
command-line scripts
edited Feb 21 at 9:47
dessert
25.6k674108
asked Feb 21 at 7:44
WhoCaresWhoCares
203
edited Feb 21 at 9:47
dessert
25.6k674108
asked Feb 21 at 7:44
WhoCaresWhoCares
203
edited Feb 21 at 9:47
dessert
25.6k674108
edited Feb 21 at 9:47
dessert
25.6k674108
edited Feb 21 at 9:47
dessert
25.6k674108
25.6k674108
asked Feb 21 at 7:44
WhoCaresWhoCares
203
asked Feb 21 at 7:44
WhoCaresWhoCares
203
asked Feb 21 at 7:44
WhoCaresWhoCares
203
203
2
This question might be better suited for stack overflow (?) that being said (I am sure someone will correct me if I am wrong, as I am far from an expert...) but are you sure you want to be flushing to stdout afterprintf(Running...)? IIRC flushing immediately prints tostdoutor wherever you're printing, so you print immediately then when another instance runs in parallel you print that instantly without waiting for the previous instance to finish the loop
– j-money
Feb 21 at 9:11
1
This is expected behaviour. Each process running in the background (or in parallel as you write) output when the code says to do. If you remove the ampersands from the script, the./runprocesses will run one at the time, and the output will be in correct order.
– Soren A
Feb 21 at 9:36
@j-money OP asks how to get proper output order when running shell commands, that’s not a good fit for SO I think.
– dessert
Feb 21 at 9:45
add a comment |
2
This question might be better suited for stack overflow (?) that being said (I am sure someone will correct me if I am wrong, as I am far from an expert...) but are you sure you want to be flushing to stdout afterprintf(Running...)? IIRC flushing immediately prints tostdoutor wherever you're printing, so you print immediately then when another instance runs in parallel you print that instantly without waiting for the previous instance to finish the loop
– j-money
Feb 21 at 9:11
1
This is expected behaviour. Each process running in the background (or in parallel as you write) output when the code says to do. If you remove the ampersands from the script, the./runprocesses will run one at the time, and the output will be in correct order.
– Soren A
Feb 21 at 9:36
@j-money OP asks how to get proper output order when running shell commands, that’s not a good fit for SO I think.
– dessert
Feb 21 at 9:45
2
2
This question might be better suited for stack overflow (?) that being said (I am sure someone will correct me if I am wrong, as I am far from an expert...) but are you sure you want to be flushing to stdout after
printf(Running...)? IIRC flushing immediately prints to stdout or wherever you're printing, so you print immediately then when another instance runs in parallel you print that instantly without waiting for the previous instance to finish the loop– j-money
Feb 21 at 9:11
This question might be better suited for stack overflow (?) that being said (I am sure someone will correct me if I am wrong, as I am far from an expert...) but are you sure you want to be flushing to stdout after
printf(Running...)? IIRC flushing immediately prints to stdout or wherever you're printing, so you print immediately then when another instance runs in parallel you print that instantly without waiting for the previous instance to finish the loop– j-money
Feb 21 at 9:11
1
1
This is expected behaviour. Each process running in the background (or in parallel as you write) output when the code says to do. If you remove the ampersands from the script, the
./run processes will run one at the time, and the output will be in correct order.– Soren A
Feb 21 at 9:36
This is expected behaviour. Each process running in the background (or in parallel as you write) output when the code says to do. If you remove the ampersands from the script, the
./run processes will run one at the time, and the output will be in correct order.– Soren A
Feb 21 at 9:36
@j-money OP asks how to get proper output order when running shell commands, that’s not a good fit for SO I think.
– dessert
Feb 21 at 9:45
@j-money OP asks how to get proper output order when running shell commands, that’s not a good fit for SO I think.
– dessert
Feb 21 at 9:45
add a comment |
2 Answers
2
active
oldest
votes
2
Meet GNU parallel (
sudo apt install parallel):
GNU parallel is a shell tool for executing jobs in parallel using one
or more computers. (…) [It] makes sure output from the commands is the
same output as you would get had you run the commands sequentially.
(man parallel)
In your case with repeating arguments you can make use of bash Brace Expansion to easily build the command lines. The basic syntax is parallel COMMAND {} ::: ARGUMENTS, if you want to give each run multiple arguments take care of proper quoting to prevent word splitting, e.g.:
$ parallel echo ./run some args {} ::: {"1 0 "{10,25},"2 0 10"} {0.{1,2,5},1}
./run some args 1 0 10 0.1
./run some args 1 0 10 0.2
./run some args 1 0 10 0.5
./run some args 1 0 10 1
./run some args 1 0 25 0.1
./run some args 1 0 25 0.2
./run some args 1 0 25 0.5
./run some args 1 0 25 1
./run some args 2 0 10 0.1
./run some args 2 0 10 0.2
./run some args 2 0 10 0.5
./run some args 2 0 10 1
answered Feb 21 at 9:15
dessertdessert
25.6k674108
Hey @dessert I hope you are there. When I run this command, it says "insufficient parameters" (see the code above).parallel ./run 4 50 5000 100 100 1 5 0 0 0.05 1 101 6 2 {} ::: {1,2,3,4} {0,1,2} {10,25} {0.{1,2,5},1}. I think it doesn't take the arguments. What should I do?
– WhoCares
Mar 19 at 12:57
@WhoCares it’s hard to say what you need here, but generally add anechoas in my answer to see the expansion’s results and be able to track the issue down. You find me in the general chat room.
– dessert
Mar 19 at 13:11
add a comment |
2
Your script output won’t get mixed together if you redirect the output of each command to a different file. For example:
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.1 > run-0.1.log &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.2 > run-0.2.log &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.5 > run-0.5.log &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 1 > run-1.0.log &
You can later open these files and check the details of each process.
Or, if you are not interested in seeing the outputs at all, redirect them to /dev/null.
answered Feb 21 at 10:20
MelebiusMelebius
5,11852041
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "89"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f1120040%2fhow-to-run-commands-in-the-shell-in-parallel-without-disrupting-the-output%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
2
Meet GNU parallel (
sudo apt install parallel):
GNU parallel is a shell tool for executing jobs in parallel using one
or more computers. (…) [It] makes sure output from the commands is the
same output as you would get had you run the commands sequentially.
(man parallel)
In your case with repeating arguments you can make use of bash Brace Expansion to easily build the command lines. The basic syntax is parallel COMMAND {} ::: ARGUMENTS, if you want to give each run multiple arguments take care of proper quoting to prevent word splitting, e.g.:
$ parallel echo ./run some args {} ::: {"1 0 "{10,25},"2 0 10"} {0.{1,2,5},1}
./run some args 1 0 10 0.1
./run some args 1 0 10 0.2
./run some args 1 0 10 0.5
./run some args 1 0 10 1
./run some args 1 0 25 0.1
./run some args 1 0 25 0.2
./run some args 1 0 25 0.5
./run some args 1 0 25 1
./run some args 2 0 10 0.1
./run some args 2 0 10 0.2
./run some args 2 0 10 0.5
./run some args 2 0 10 1
answered Feb 21 at 9:15
dessertdessert
25.6k674108
Hey @dessert I hope you are there. When I run this command, it says "insufficient parameters" (see the code above).parallel ./run 4 50 5000 100 100 1 5 0 0 0.05 1 101 6 2 {} ::: {1,2,3,4} {0,1,2} {10,25} {0.{1,2,5},1}. I think it doesn't take the arguments. What should I do?
– WhoCares
Mar 19 at 12:57
@WhoCares it’s hard to say what you need here, but generally add anechoas in my answer to see the expansion’s results and be able to track the issue down. You find me in the general chat room.
– dessert
Mar 19 at 13:11
add a comment |
2
Meet GNU parallel (
sudo apt install parallel):
GNU parallel is a shell tool for executing jobs in parallel using one
or more computers. (…) [It] makes sure output from the commands is the
same output as you would get had you run the commands sequentially.
(man parallel)
In your case with repeating arguments you can make use of bash Brace Expansion to easily build the command lines. The basic syntax is parallel COMMAND {} ::: ARGUMENTS, if you want to give each run multiple arguments take care of proper quoting to prevent word splitting, e.g.:
$ parallel echo ./run some args {} ::: {"1 0 "{10,25},"2 0 10"} {0.{1,2,5},1}
./run some args 1 0 10 0.1
./run some args 1 0 10 0.2
./run some args 1 0 10 0.5
./run some args 1 0 10 1
./run some args 1 0 25 0.1
./run some args 1 0 25 0.2
./run some args 1 0 25 0.5
./run some args 1 0 25 1
./run some args 2 0 10 0.1
./run some args 2 0 10 0.2
./run some args 2 0 10 0.5
./run some args 2 0 10 1
answered Feb 21 at 9:15
dessertdessert
25.6k674108
Hey @dessert I hope you are there. When I run this command, it says "insufficient parameters" (see the code above).parallel ./run 4 50 5000 100 100 1 5 0 0 0.05 1 101 6 2 {} ::: {1,2,3,4} {0,1,2} {10,25} {0.{1,2,5},1}. I think it doesn't take the arguments. What should I do?
– WhoCares
Mar 19 at 12:57
@WhoCares it’s hard to say what you need here, but generally add anechoas in my answer to see the expansion’s results and be able to track the issue down. You find me in the general chat room.
– dessert
Mar 19 at 13:11
add a comment |
2
2
2
Meet GNU parallel (
sudo apt install parallel):
GNU parallel is a shell tool for executing jobs in parallel using one
or more computers. (…) [It] makes sure output from the commands is the
same output as you would get had you run the commands sequentially.
(man parallel)
In your case with repeating arguments you can make use of bash Brace Expansion to easily build the command lines. The basic syntax is parallel COMMAND {} ::: ARGUMENTS, if you want to give each run multiple arguments take care of proper quoting to prevent word splitting, e.g.:
$ parallel echo ./run some args {} ::: {"1 0 "{10,25},"2 0 10"} {0.{1,2,5},1}
./run some args 1 0 10 0.1
./run some args 1 0 10 0.2
./run some args 1 0 10 0.5
./run some args 1 0 10 1
./run some args 1 0 25 0.1
./run some args 1 0 25 0.2
./run some args 1 0 25 0.5
./run some args 1 0 25 1
./run some args 2 0 10 0.1
./run some args 2 0 10 0.2
./run some args 2 0 10 0.5
./run some args 2 0 10 1
answered Feb 21 at 9:15
dessertdessert
25.6k674108
Meet GNU parallel (
sudo apt install parallel):
GNU parallel is a shell tool for executing jobs in parallel using one
or more computers. (…) [It] makes sure output from the commands is the
same output as you would get had you run the commands sequentially.
(man parallel)
In your case with repeating arguments you can make use of bash Brace Expansion to easily build the command lines. The basic syntax is parallel COMMAND {} ::: ARGUMENTS, if you want to give each run multiple arguments take care of proper quoting to prevent word splitting, e.g.:
$ parallel echo ./run some args {} ::: {"1 0 "{10,25},"2 0 10"} {0.{1,2,5},1}
./run some args 1 0 10 0.1
./run some args 1 0 10 0.2
./run some args 1 0 10 0.5
./run some args 1 0 10 1
./run some args 1 0 25 0.1
./run some args 1 0 25 0.2
./run some args 1 0 25 0.5
./run some args 1 0 25 1
./run some args 2 0 10 0.1
./run some args 2 0 10 0.2
./run some args 2 0 10 0.5
./run some args 2 0 10 1
answered Feb 21 at 9:15
dessertdessert
25.6k674108
edited Feb 21 at 9:21
answered Feb 21 at 9:15
dessertdessert
25.6k674108
answered Feb 21 at 9:15
dessertdessert
25.6k674108
answered Feb 21 at 9:15
dessertdessert
25.6k674108
25.6k674108
Hey @dessert I hope you are there. When I run this command, it says "insufficient parameters" (see the code above).parallel ./run 4 50 5000 100 100 1 5 0 0 0.05 1 101 6 2 {} ::: {1,2,3,4} {0,1,2} {10,25} {0.{1,2,5},1}. I think it doesn't take the arguments. What should I do?
– WhoCares
Mar 19 at 12:57
@WhoCares it’s hard to say what you need here, but generally add anechoas in my answer to see the expansion’s results and be able to track the issue down. You find me in the general chat room.
– dessert
Mar 19 at 13:11
add a comment |
Hey @dessert I hope you are there. When I run this command, it says "insufficient parameters" (see the code above).parallel ./run 4 50 5000 100 100 1 5 0 0 0.05 1 101 6 2 {} ::: {1,2,3,4} {0,1,2} {10,25} {0.{1,2,5},1}. I think it doesn't take the arguments. What should I do?
– WhoCares
Mar 19 at 12:57
@WhoCares it’s hard to say what you need here, but generally add anechoas in my answer to see the expansion’s results and be able to track the issue down. You find me in the general chat room.
– dessert
Mar 19 at 13:11
Hey @dessert I hope you are there. When I run this command, it says "insufficient parameters" (see the code above).
parallel ./run 4 50 5000 100 100 1 5 0 0 0.05 1 101 6 2 {} ::: {1,2,3,4} {0,1,2} {10,25} {0.{1,2,5},1}. I think it doesn't take the arguments. What should I do?– WhoCares
Mar 19 at 12:57
Hey @dessert I hope you are there. When I run this command, it says "insufficient parameters" (see the code above).
parallel ./run 4 50 5000 100 100 1 5 0 0 0.05 1 101 6 2 {} ::: {1,2,3,4} {0,1,2} {10,25} {0.{1,2,5},1}. I think it doesn't take the arguments. What should I do?– WhoCares
Mar 19 at 12:57
@WhoCares it’s hard to say what you need here, but generally add an
echo as in my answer to see the expansion’s results and be able to track the issue down. You find me in the general chat room.– dessert
Mar 19 at 13:11
@WhoCares it’s hard to say what you need here, but generally add an
echo as in my answer to see the expansion’s results and be able to track the issue down. You find me in the general chat room.– dessert
Mar 19 at 13:11
add a comment |
2
Your script output won’t get mixed together if you redirect the output of each command to a different file. For example:
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.1 > run-0.1.log &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.2 > run-0.2.log &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.5 > run-0.5.log &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 1 > run-1.0.log &
You can later open these files and check the details of each process.
Or, if you are not interested in seeing the outputs at all, redirect them to /dev/null.
answered Feb 21 at 10:20
MelebiusMelebius
5,11852041
add a comment |
2
Your script output won’t get mixed together if you redirect the output of each command to a different file. For example:
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.1 > run-0.1.log &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.2 > run-0.2.log &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.5 > run-0.5.log &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 1 > run-1.0.log &
You can later open these files and check the details of each process.
Or, if you are not interested in seeing the outputs at all, redirect them to /dev/null.
answered Feb 21 at 10:20
MelebiusMelebius
5,11852041
add a comment |
2
2
2
Your script output won’t get mixed together if you redirect the output of each command to a different file. For example:
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.1 > run-0.1.log &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.2 > run-0.2.log &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.5 > run-0.5.log &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 1 > run-1.0.log &
You can later open these files and check the details of each process.
Or, if you are not interested in seeing the outputs at all, redirect them to /dev/null.
answered Feb 21 at 10:20
MelebiusMelebius
5,11852041
Your script output won’t get mixed together if you redirect the output of each command to a different file. For example:
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.1 > run-0.1.log &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.2 > run-0.2.log &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.5 > run-0.5.log &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 1 > run-1.0.log &
You can later open these files and check the details of each process.
Or, if you are not interested in seeing the outputs at all, redirect them to /dev/null.
answered Feb 21 at 10:20
MelebiusMelebius
5,11852041
answered Feb 21 at 10:20
MelebiusMelebius
5,11852041
answered Feb 21 at 10:20
MelebiusMelebius
5,11852041
answered Feb 21 at 10:20
MelebiusMelebius
5,11852041
5,11852041
add a comment |
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Ask Ubuntu!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f1120040%2fhow-to-run-commands-in-the-shell-in-parallel-without-disrupting-the-output%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
This question might be better suited for stack overflow (?) that being said (I am sure someone will correct me if I am wrong, as I am far from an expert...) but are you sure you want to be flushing to stdout after
printf(Running...)? IIRC flushing immediately prints tostdoutor wherever you're printing, so you print immediately then when another instance runs in parallel you print that instantly without waiting for the previous instance to finish the loop– j-money
Feb 21 at 9:11
1
This is expected behaviour. Each process running in the background (or in parallel as you write) output when the code says to do. If you remove the ampersands from the script, the
./runprocesses will run one at the time, and the output will be in correct order.– Soren A
Feb 21 at 9:36
@j-money OP asks how to get proper output order when running shell commands, that’s not a good fit for SO I think.
– dessert
Feb 21 at 9:45