Dispatching chunks of work to backgroundworkers
up vote
-1
down vote
favorite
Using C#.
I have 100,000+ pieces of test data that need to have some calculations run with. My actual data set will be in the millions of pieces of data. The test data currently runs sequentially and takes about a minute to process. I want to split this work up and have backgroundworkers process back to back so I will hopefully get the processing done quicker.
What I have in mind is to do a foreach loop with the data and start a backgroundworker with each piece of data. I know I need to limit the number of bw's to three as I have 4 cores on this machine. I have done some testing with simple bw's but not three at the same time.
I have no idea how to go about this. How would one execute three background workers to process this data?
c# backgroundworker
asked Nov 15 at 1:57
Eric Snyder
15418
add a comment |
up vote
-1
down vote
favorite
Using C#.
I have 100,000+ pieces of test data that need to have some calculations run with. My actual data set will be in the millions of pieces of data. The test data currently runs sequentially and takes about a minute to process. I want to split this work up and have backgroundworkers process back to back so I will hopefully get the processing done quicker.
What I have in mind is to do a foreach loop with the data and start a backgroundworker with each piece of data. I know I need to limit the number of bw's to three as I have 4 cores on this machine. I have done some testing with simple bw's but not three at the same time.
I have no idea how to go about this. How would one execute three background workers to process this data?
c# backgroundworker
asked Nov 15 at 1:57
Eric Snyder
15418
could anyone explain how bw related to number of cores ?
– Z.R.T.
Nov 15 at 2:02
It would be awesome if you could provide a Minimal, Complete, and Verifiable example of your progress so far.
– mjwills
Nov 15 at 2:19
add a comment |
up vote
-1
down vote
favorite
up vote
-1
down vote
favorite
Using C#.
I have 100,000+ pieces of test data that need to have some calculations run with. My actual data set will be in the millions of pieces of data. The test data currently runs sequentially and takes about a minute to process. I want to split this work up and have backgroundworkers process back to back so I will hopefully get the processing done quicker.
What I have in mind is to do a foreach loop with the data and start a backgroundworker with each piece of data. I know I need to limit the number of bw's to three as I have 4 cores on this machine. I have done some testing with simple bw's but not three at the same time.
I have no idea how to go about this. How would one execute three background workers to process this data?
c# backgroundworker
asked Nov 15 at 1:57
Eric Snyder
15418
Using C#.
I have 100,000+ pieces of test data that need to have some calculations run with. My actual data set will be in the millions of pieces of data. The test data currently runs sequentially and takes about a minute to process. I want to split this work up and have backgroundworkers process back to back so I will hopefully get the processing done quicker.
What I have in mind is to do a foreach loop with the data and start a backgroundworker with each piece of data. I know I need to limit the number of bw's to three as I have 4 cores on this machine. I have done some testing with simple bw's but not three at the same time.
I have no idea how to go about this. How would one execute three background workers to process this data?
c# backgroundworker
c# backgroundworker
asked Nov 15 at 1:57
Eric Snyder
15418
asked Nov 15 at 1:57
Eric Snyder
15418
asked Nov 15 at 1:57
Eric Snyder
15418
asked Nov 15 at 1:57
Eric Snyder
15418
asked Nov 15 at 1:57
Eric Snyder
15418
15418
could anyone explain how bw related to number of cores ?
– Z.R.T.
Nov 15 at 2:02
It would be awesome if you could provide a Minimal, Complete, and Verifiable example of your progress so far.
– mjwills
Nov 15 at 2:19
add a comment |
could anyone explain how bw related to number of cores ?
– Z.R.T.
Nov 15 at 2:02
It would be awesome if you could provide a Minimal, Complete, and Verifiable example of your progress so far.
– mjwills
Nov 15 at 2:19
could anyone explain how bw related to number of cores ?
– Z.R.T.
Nov 15 at 2:02
could anyone explain how bw related to number of cores ?
– Z.R.T.
Nov 15 at 2:02
It would be awesome if you could provide a Minimal, Complete, and Verifiable example of your progress so far.
– mjwills
Nov 15 at 2:19
It would be awesome if you could provide a Minimal, Complete, and Verifiable example of your progress so far.
– mjwills
Nov 15 at 2:19
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
The BackgroundWorker is designed for early learning work mostly. Maybe the odd alternative threading scenario. What you are doing sounds like a very advanced opeartion. You can still use BGW, but raw Threads, Tasks, Threadpools and the like would be better at this point.
There is also the general question if this operation can even be accelerated with Multithreading. I like to say "multithreading has to pick it's problems carefully". Pick it in the wrong scenario and you end with a programm that needs more memory, is more prone to errors and slower then a single BGW or sequential programm.
Your case could be one of the rare cases of a pleasingly paralell operation. Or it could be mostly memory bound. Wich means you run into Paralell slowdown almost instantly. Resist atempts at hardcoding the number of threads. Usually you can leave that load-balancing work to a ThreadPool. To get a better answer you need to get a lot more specific.
answered Nov 15 at 3:20
Christopher
2,6071621
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
The BackgroundWorker is designed for early learning work mostly. Maybe the odd alternative threading scenario. What you are doing sounds like a very advanced opeartion. You can still use BGW, but raw Threads, Tasks, Threadpools and the like would be better at this point.
There is also the general question if this operation can even be accelerated with Multithreading. I like to say "multithreading has to pick it's problems carefully". Pick it in the wrong scenario and you end with a programm that needs more memory, is more prone to errors and slower then a single BGW or sequential programm.
Your case could be one of the rare cases of a pleasingly paralell operation. Or it could be mostly memory bound. Wich means you run into Paralell slowdown almost instantly. Resist atempts at hardcoding the number of threads. Usually you can leave that load-balancing work to a ThreadPool. To get a better answer you need to get a lot more specific.
answered Nov 15 at 3:20
Christopher
2,6071621
add a comment |
up vote
0
down vote
The BackgroundWorker is designed for early learning work mostly. Maybe the odd alternative threading scenario. What you are doing sounds like a very advanced opeartion. You can still use BGW, but raw Threads, Tasks, Threadpools and the like would be better at this point.
There is also the general question if this operation can even be accelerated with Multithreading. I like to say "multithreading has to pick it's problems carefully". Pick it in the wrong scenario and you end with a programm that needs more memory, is more prone to errors and slower then a single BGW or sequential programm.
Your case could be one of the rare cases of a pleasingly paralell operation. Or it could be mostly memory bound. Wich means you run into Paralell slowdown almost instantly. Resist atempts at hardcoding the number of threads. Usually you can leave that load-balancing work to a ThreadPool. To get a better answer you need to get a lot more specific.
answered Nov 15 at 3:20
Christopher
2,6071621
add a comment |
up vote
0
down vote
up vote
0
down vote
The BackgroundWorker is designed for early learning work mostly. Maybe the odd alternative threading scenario. What you are doing sounds like a very advanced opeartion. You can still use BGW, but raw Threads, Tasks, Threadpools and the like would be better at this point.
There is also the general question if this operation can even be accelerated with Multithreading. I like to say "multithreading has to pick it's problems carefully". Pick it in the wrong scenario and you end with a programm that needs more memory, is more prone to errors and slower then a single BGW or sequential programm.
Your case could be one of the rare cases of a pleasingly paralell operation. Or it could be mostly memory bound. Wich means you run into Paralell slowdown almost instantly. Resist atempts at hardcoding the number of threads. Usually you can leave that load-balancing work to a ThreadPool. To get a better answer you need to get a lot more specific.
answered Nov 15 at 3:20
Christopher
2,6071621
The BackgroundWorker is designed for early learning work mostly. Maybe the odd alternative threading scenario. What you are doing sounds like a very advanced opeartion. You can still use BGW, but raw Threads, Tasks, Threadpools and the like would be better at this point.
There is also the general question if this operation can even be accelerated with Multithreading. I like to say "multithreading has to pick it's problems carefully". Pick it in the wrong scenario and you end with a programm that needs more memory, is more prone to errors and slower then a single BGW or sequential programm.
Your case could be one of the rare cases of a pleasingly paralell operation. Or it could be mostly memory bound. Wich means you run into Paralell slowdown almost instantly. Resist atempts at hardcoding the number of threads. Usually you can leave that load-balancing work to a ThreadPool. To get a better answer you need to get a lot more specific.
answered Nov 15 at 3:20
Christopher
2,6071621
answered Nov 15 at 3:20
Christopher
2,6071621
answered Nov 15 at 3:20
Christopher
2,6071621
answered Nov 15 at 3:20
Christopher
2,6071621
2,6071621
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53311346%2fdispatching-chunks-of-work-to-backgroundworkers%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
could anyone explain how bw related to number of cores ?
– Z.R.T.
Nov 15 at 2:02
It would be awesome if you could provide a Minimal, Complete, and Verifiable example of your progress so far.
– mjwills
Nov 15 at 2:19