scapping web page using htmlunit
Here is my code. I want to get access to a website and compare my data. I want java to put data in fields and auto click on calculate bottom and return the answer to java.
import com.gargoylesoftware.htmlunit.WebClient;
public class MyWebServiceAccess {
public static void main(String args) throws Exception
{
final WebClient webClient = new WebClient();
final HtmlPage page = webClient.getPage("https://www.socscistatistics.com/tests/signedranks/Default2.aspx");
// Inputs
HtmlTextInput treatment1 = (HtmlTextInput) page.getElementById("ctl00_MainContent_TextBox1");
HtmlTextInput treatment2 = (HtmlTextInput) page.getElementById("ctl00_MainContent_TextBox2");
// Significance Level:
HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");
// 1 or 2-tailed hypothesis?:
HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");
// Calculate
HtmlSubmitInput Calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button2");
// Result Span
HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");
// Fill in Inputs
treatment1.setValueAttribute("");
treatment2.setValueAttribute("");
s1.setChecked(true);
s2.setChecked(false);
t1.setChecked(true);
t2.setChecked(false);
Calculate.click();
// Printing the Output
System.out.println(result.asText());
webClient.closeAllWindows();
}
}
java web htmlunit
add a comment |
Here is my code. I want to get access to a website and compare my data. I want java to put data in fields and auto click on calculate bottom and return the answer to java.
import com.gargoylesoftware.htmlunit.WebClient;
public class MyWebServiceAccess {
public static void main(String args) throws Exception
{
final WebClient webClient = new WebClient();
final HtmlPage page = webClient.getPage("https://www.socscistatistics.com/tests/signedranks/Default2.aspx");
// Inputs
HtmlTextInput treatment1 = (HtmlTextInput) page.getElementById("ctl00_MainContent_TextBox1");
HtmlTextInput treatment2 = (HtmlTextInput) page.getElementById("ctl00_MainContent_TextBox2");
// Significance Level:
HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");
// 1 or 2-tailed hypothesis?:
HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");
// Calculate
HtmlSubmitInput Calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button2");
// Result Span
HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");
// Fill in Inputs
treatment1.setValueAttribute("");
treatment2.setValueAttribute("");
s1.setChecked(true);
s2.setChecked(false);
t1.setChecked(true);
t2.setChecked(false);
Calculate.click();
// Printing the Output
System.out.println(result.asText());
webClient.closeAllWindows();
}
}
java web htmlunit
What is your question? And don’t tell me it’s “how to do it?” because that is way to broad.
– Joakim Danielson
Nov 17 '18 at 8:56
my problem is that I can not connect to HTML page and do what I want.I'm looking for the problem of my code not the way to implement it.
– Farid Amyaie
Nov 17 '18 at 9:51
Update your question with your problem then including any errors and what part of the code that gives you trouble.
– Joakim Danielson
Nov 17 '18 at 11:36
add a comment |
Here is my code. I want to get access to a website and compare my data. I want java to put data in fields and auto click on calculate bottom and return the answer to java.
import com.gargoylesoftware.htmlunit.WebClient;
public class MyWebServiceAccess {
public static void main(String args) throws Exception
{
final WebClient webClient = new WebClient();
final HtmlPage page = webClient.getPage("https://www.socscistatistics.com/tests/signedranks/Default2.aspx");
// Inputs
HtmlTextInput treatment1 = (HtmlTextInput) page.getElementById("ctl00_MainContent_TextBox1");
HtmlTextInput treatment2 = (HtmlTextInput) page.getElementById("ctl00_MainContent_TextBox2");
// Significance Level:
HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");
// 1 or 2-tailed hypothesis?:
HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");
// Calculate
HtmlSubmitInput Calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button2");
// Result Span
HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");
// Fill in Inputs
treatment1.setValueAttribute("");
treatment2.setValueAttribute("");
s1.setChecked(true);
s2.setChecked(false);
t1.setChecked(true);
t2.setChecked(false);
Calculate.click();
// Printing the Output
System.out.println(result.asText());
webClient.closeAllWindows();
}
}
java web htmlunit
Here is my code. I want to get access to a website and compare my data. I want java to put data in fields and auto click on calculate bottom and return the answer to java.
import com.gargoylesoftware.htmlunit.WebClient;
public class MyWebServiceAccess {
public static void main(String args) throws Exception
{
final WebClient webClient = new WebClient();
final HtmlPage page = webClient.getPage("https://www.socscistatistics.com/tests/signedranks/Default2.aspx");
// Inputs
HtmlTextInput treatment1 = (HtmlTextInput) page.getElementById("ctl00_MainContent_TextBox1");
HtmlTextInput treatment2 = (HtmlTextInput) page.getElementById("ctl00_MainContent_TextBox2");
// Significance Level:
HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");
// 1 or 2-tailed hypothesis?:
HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");
// Calculate
HtmlSubmitInput Calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button2");
// Result Span
HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");
// Fill in Inputs
treatment1.setValueAttribute("");
treatment2.setValueAttribute("");
s1.setChecked(true);
s2.setChecked(false);
t1.setChecked(true);
t2.setChecked(false);
Calculate.click();
// Printing the Output
System.out.println(result.asText());
webClient.closeAllWindows();
}
}
java web htmlunit
java web htmlunit
edited Nov 17 '18 at 8:27
lzagkaretos
1,7572615
1,7572615
asked Nov 17 '18 at 8:19
Farid Amyaie
12
12
What is your question? And don’t tell me it’s “how to do it?” because that is way to broad.
– Joakim Danielson
Nov 17 '18 at 8:56
my problem is that I can not connect to HTML page and do what I want.I'm looking for the problem of my code not the way to implement it.
– Farid Amyaie
Nov 17 '18 at 9:51
Update your question with your problem then including any errors and what part of the code that gives you trouble.
– Joakim Danielson
Nov 17 '18 at 11:36
add a comment |
What is your question? And don’t tell me it’s “how to do it?” because that is way to broad.
– Joakim Danielson
Nov 17 '18 at 8:56
my problem is that I can not connect to HTML page and do what I want.I'm looking for the problem of my code not the way to implement it.
– Farid Amyaie
Nov 17 '18 at 9:51
Update your question with your problem then including any errors and what part of the code that gives you trouble.
– Joakim Danielson
Nov 17 '18 at 11:36
What is your question? And don’t tell me it’s “how to do it?” because that is way to broad.
– Joakim Danielson
Nov 17 '18 at 8:56
What is your question? And don’t tell me it’s “how to do it?” because that is way to broad.
– Joakim Danielson
Nov 17 '18 at 8:56
my problem is that I can not connect to HTML page and do what I want.I'm looking for the problem of my code not the way to implement it.
– Farid Amyaie
Nov 17 '18 at 9:51
my problem is that I can not connect to HTML page and do what I want.I'm looking for the problem of my code not the way to implement it.
– Farid Amyaie
Nov 17 '18 at 9:51
Update your question with your problem then including any errors and what part of the code that gives you trouble.
– Joakim Danielson
Nov 17 '18 at 11:36
Update your question with your problem then including any errors and what part of the code that gives you trouble.
– Joakim Danielson
Nov 17 '18 at 11:36
add a comment |
1 Answer
1
active
oldest
votes
When scrapping web pages you need a basic understanding how the web technologies (http/html) are working and you also need some java/programming knowledge. At least it is really helpful to be able to find problems in your programs.
At first your code produces a class cast exception because the input fields are text areas and not input text controls. At second you have to click the right button (your code clicks the 'reset' button). And finally you got a new page if the button is clicked. Your result is on the new page.
Hope that helps....
String url = "https://www.socscistatistics.com/tests/signedranks/Default2.aspx";
try (final WebClient webClient = new WebClient()) {
HtmlPage page = webClient.getPage(url);
// Inputs
HtmlTextArea treatment1 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox1");
HtmlTextArea treatment2 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox2");
// Significance Level:
HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");
s1.setChecked(true);
s2.setChecked(false);
// 1 or 2-tailed hypothesis?:
HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");
t1.setChecked(true);
t2.setChecked(false);
// Fill in Inputs
treatment1.type("4n3n2n5n5n3");
treatment2.type("1n2n3n0n0n2");
// click Calculate creates a new page
HtmlSubmitInput calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button1");
page = calculate.click();
// Result Span
HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");
// Printing the Output
System.out.println(result.asText());
}
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53349504%2fscapping-web-page-using-htmlunit%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
When scrapping web pages you need a basic understanding how the web technologies (http/html) are working and you also need some java/programming knowledge. At least it is really helpful to be able to find problems in your programs.
At first your code produces a class cast exception because the input fields are text areas and not input text controls. At second you have to click the right button (your code clicks the 'reset' button). And finally you got a new page if the button is clicked. Your result is on the new page.
Hope that helps....
String url = "https://www.socscistatistics.com/tests/signedranks/Default2.aspx";
try (final WebClient webClient = new WebClient()) {
HtmlPage page = webClient.getPage(url);
// Inputs
HtmlTextArea treatment1 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox1");
HtmlTextArea treatment2 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox2");
// Significance Level:
HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");
s1.setChecked(true);
s2.setChecked(false);
// 1 or 2-tailed hypothesis?:
HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");
t1.setChecked(true);
t2.setChecked(false);
// Fill in Inputs
treatment1.type("4n3n2n5n5n3");
treatment2.type("1n2n3n0n0n2");
// click Calculate creates a new page
HtmlSubmitInput calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button1");
page = calculate.click();
// Result Span
HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");
// Printing the Output
System.out.println(result.asText());
}
add a comment |
When scrapping web pages you need a basic understanding how the web technologies (http/html) are working and you also need some java/programming knowledge. At least it is really helpful to be able to find problems in your programs.
At first your code produces a class cast exception because the input fields are text areas and not input text controls. At second you have to click the right button (your code clicks the 'reset' button). And finally you got a new page if the button is clicked. Your result is on the new page.
Hope that helps....
String url = "https://www.socscistatistics.com/tests/signedranks/Default2.aspx";
try (final WebClient webClient = new WebClient()) {
HtmlPage page = webClient.getPage(url);
// Inputs
HtmlTextArea treatment1 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox1");
HtmlTextArea treatment2 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox2");
// Significance Level:
HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");
s1.setChecked(true);
s2.setChecked(false);
// 1 or 2-tailed hypothesis?:
HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");
t1.setChecked(true);
t2.setChecked(false);
// Fill in Inputs
treatment1.type("4n3n2n5n5n3");
treatment2.type("1n2n3n0n0n2");
// click Calculate creates a new page
HtmlSubmitInput calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button1");
page = calculate.click();
// Result Span
HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");
// Printing the Output
System.out.println(result.asText());
}
add a comment |
When scrapping web pages you need a basic understanding how the web technologies (http/html) are working and you also need some java/programming knowledge. At least it is really helpful to be able to find problems in your programs.
At first your code produces a class cast exception because the input fields are text areas and not input text controls. At second you have to click the right button (your code clicks the 'reset' button). And finally you got a new page if the button is clicked. Your result is on the new page.
Hope that helps....
String url = "https://www.socscistatistics.com/tests/signedranks/Default2.aspx";
try (final WebClient webClient = new WebClient()) {
HtmlPage page = webClient.getPage(url);
// Inputs
HtmlTextArea treatment1 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox1");
HtmlTextArea treatment2 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox2");
// Significance Level:
HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");
s1.setChecked(true);
s2.setChecked(false);
// 1 or 2-tailed hypothesis?:
HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");
t1.setChecked(true);
t2.setChecked(false);
// Fill in Inputs
treatment1.type("4n3n2n5n5n3");
treatment2.type("1n2n3n0n0n2");
// click Calculate creates a new page
HtmlSubmitInput calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button1");
page = calculate.click();
// Result Span
HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");
// Printing the Output
System.out.println(result.asText());
}
When scrapping web pages you need a basic understanding how the web technologies (http/html) are working and you also need some java/programming knowledge. At least it is really helpful to be able to find problems in your programs.
At first your code produces a class cast exception because the input fields are text areas and not input text controls. At second you have to click the right button (your code clicks the 'reset' button). And finally you got a new page if the button is clicked. Your result is on the new page.
Hope that helps....
String url = "https://www.socscistatistics.com/tests/signedranks/Default2.aspx";
try (final WebClient webClient = new WebClient()) {
HtmlPage page = webClient.getPage(url);
// Inputs
HtmlTextArea treatment1 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox1");
HtmlTextArea treatment2 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox2");
// Significance Level:
HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");
s1.setChecked(true);
s2.setChecked(false);
// 1 or 2-tailed hypothesis?:
HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");
t1.setChecked(true);
t2.setChecked(false);
// Fill in Inputs
treatment1.type("4n3n2n5n5n3");
treatment2.type("1n2n3n0n0n2");
// click Calculate creates a new page
HtmlSubmitInput calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button1");
page = calculate.click();
// Result Span
HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");
// Printing the Output
System.out.println(result.asText());
}
edited Nov 18 '18 at 21:09
answered Nov 17 '18 at 19:22
RBRi
1,241269
1,241269
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53349504%2fscapping-web-page-using-htmlunit%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What is your question? And don’t tell me it’s “how to do it?” because that is way to broad.
– Joakim Danielson
Nov 17 '18 at 8:56
my problem is that I can not connect to HTML page and do what I want.I'm looking for the problem of my code not the way to implement it.
– Farid Amyaie
Nov 17 '18 at 9:51
Update your question with your problem then including any errors and what part of the code that gives you trouble.
– Joakim Danielson
Nov 17 '18 at 11:36