MSTest unit tests started randomly failing
up vote
1
down vote
favorite
A few days ago (11/12) our CI builds for a .NET 4.7 Windows Forms app started "partially succeeding" unexpectedly. I traced the problem back to the Visual Studio Test steps. After breaking out the unit tests into separate steps for each DLL I've been further able to isolate it to the UI test DLL.
It is a plain vanilla MSTest project where the classes test portions of the UI that can be manipulated behind the scenes. Mostly manipulating a control's data model and then inspecting details of controls to ensure that data model changes have the expected effect. UI elements are instantiated in code, either through new declarations within a class or TestClasses that are children of the controls being tested. No message boxes are presented nor UI elements actually rendered for humans or external automation to review.
These tests are running in an Azure "Hosted VS2017" environment. The test runs that fail all have exactly the same error message:
The active test run was aborted. Reason: Unhandled Exception: System.AppDomainUnloadedException: The application domain in which the thread was running has been unloaded.
I have not been able to discern a pattern with the failures. The exception can occur after different individual tests. Running the tests in isolation, in parallel, or not parallel doesn't appear to impact the failures. The failures do appear to be increasing in frequency however. All attempts to recreate this locally fail; any Visual Studio 2017 installation we've used to run the unit tests has passed them all without issue.
All I can really find on this particular topic via Google is either entries to long-past versions of Visual Studio where this was a bug that's been corrected, or to other unanswered questions. Has anyone ran into this kind of trouble before?
azure-devops mstest azure-pipelines asked Nov 15 at 15:53
Robb Bromley
62
add a comment |
up vote
1
down vote
favorite
A few days ago (11/12) our CI builds for a .NET 4.7 Windows Forms app started "partially succeeding" unexpectedly. I traced the problem back to the Visual Studio Test steps. After breaking out the unit tests into separate steps for each DLL I've been further able to isolate it to the UI test DLL.
It is a plain vanilla MSTest project where the classes test portions of the UI that can be manipulated behind the scenes. Mostly manipulating a control's data model and then inspecting details of controls to ensure that data model changes have the expected effect. UI elements are instantiated in code, either through new declarations within a class or TestClasses that are children of the controls being tested. No message boxes are presented nor UI elements actually rendered for humans or external automation to review.
These tests are running in an Azure "Hosted VS2017" environment. The test runs that fail all have exactly the same error message:
The active test run was aborted. Reason: Unhandled Exception: System.AppDomainUnloadedException: The application domain in which the thread was running has been unloaded.
I have not been able to discern a pattern with the failures. The exception can occur after different individual tests. Running the tests in isolation, in parallel, or not parallel doesn't appear to impact the failures. The failures do appear to be increasing in frequency however. All attempts to recreate this locally fail; any Visual Studio 2017 installation we've used to run the unit tests has passed them all without issue.
All I can really find on this particular topic via Google is either entries to long-past versions of Visual Studio where this was a bug that's been corrected, or to other unanswered questions. Has anyone ran into this kind of trouble before?
azure-devops mstest azure-pipelines asked Nov 15 at 15:53
Robb Bromley
62
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
A few days ago (11/12) our CI builds for a .NET 4.7 Windows Forms app started "partially succeeding" unexpectedly. I traced the problem back to the Visual Studio Test steps. After breaking out the unit tests into separate steps for each DLL I've been further able to isolate it to the UI test DLL.
It is a plain vanilla MSTest project where the classes test portions of the UI that can be manipulated behind the scenes. Mostly manipulating a control's data model and then inspecting details of controls to ensure that data model changes have the expected effect. UI elements are instantiated in code, either through new declarations within a class or TestClasses that are children of the controls being tested. No message boxes are presented nor UI elements actually rendered for humans or external automation to review.
These tests are running in an Azure "Hosted VS2017" environment. The test runs that fail all have exactly the same error message:
The active test run was aborted. Reason: Unhandled Exception: System.AppDomainUnloadedException: The application domain in which the thread was running has been unloaded.
I have not been able to discern a pattern with the failures. The exception can occur after different individual tests. Running the tests in isolation, in parallel, or not parallel doesn't appear to impact the failures. The failures do appear to be increasing in frequency however. All attempts to recreate this locally fail; any Visual Studio 2017 installation we've used to run the unit tests has passed them all without issue.
All I can really find on this particular topic via Google is either entries to long-past versions of Visual Studio where this was a bug that's been corrected, or to other unanswered questions. Has anyone ran into this kind of trouble before?
azure-devops mstest azure-pipelines asked Nov 15 at 15:53
Robb Bromley
62
A few days ago (11/12) our CI builds for a .NET 4.7 Windows Forms app started "partially succeeding" unexpectedly. I traced the problem back to the Visual Studio Test steps. After breaking out the unit tests into separate steps for each DLL I've been further able to isolate it to the UI test DLL.
It is a plain vanilla MSTest project where the classes test portions of the UI that can be manipulated behind the scenes. Mostly manipulating a control's data model and then inspecting details of controls to ensure that data model changes have the expected effect. UI elements are instantiated in code, either through new declarations within a class or TestClasses that are children of the controls being tested. No message boxes are presented nor UI elements actually rendered for humans or external automation to review.
These tests are running in an Azure "Hosted VS2017" environment. The test runs that fail all have exactly the same error message:
The active test run was aborted. Reason: Unhandled Exception: System.AppDomainUnloadedException: The application domain in which the thread was running has been unloaded.
I have not been able to discern a pattern with the failures. The exception can occur after different individual tests. Running the tests in isolation, in parallel, or not parallel doesn't appear to impact the failures. The failures do appear to be increasing in frequency however. All attempts to recreate this locally fail; any Visual Studio 2017 installation we've used to run the unit tests has passed them all without issue.
All I can really find on this particular topic via Google is either entries to long-past versions of Visual Studio where this was a bug that's been corrected, or to other unanswered questions. Has anyone ran into this kind of trouble before?
azure-devops mstest azure-pipelines azure-devops mstest azure-pipelines asked Nov 15 at 15:53
Robb Bromley
62
asked Nov 15 at 15:53
Robb Bromley
62
asked Nov 15 at 15:53
Robb Bromley
62
asked Nov 15 at 15:53
Robb Bromley
62
asked Nov 15 at 15:53
Robb Bromley
62
62
add a comment |
add a comment |
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53323190%2fmstest-unit-tests-started-randomly-failing%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53323190%2fmstest-unit-tests-started-randomly-failing%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
