scapping web page using htmlunit












0














Here is my code. I want to get access to a website and compare my data. I want java to put data in fields and auto click on calculate bottom and return the answer to java.



import com.gargoylesoftware.htmlunit.WebClient;

public class MyWebServiceAccess {

public static void main(String args) throws Exception
{

final WebClient webClient = new WebClient();

final HtmlPage page = webClient.getPage("https://www.socscistatistics.com/tests/signedranks/Default2.aspx");

// Inputs
HtmlTextInput treatment1 = (HtmlTextInput) page.getElementById("ctl00_MainContent_TextBox1");
HtmlTextInput treatment2 = (HtmlTextInput) page.getElementById("ctl00_MainContent_TextBox2");

// Significance Level:
HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");


// 1 or 2-tailed hypothesis?:
HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");

// Calculate
HtmlSubmitInput Calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button2");

// Result Span
HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");

// Fill in Inputs
treatment1.setValueAttribute("");
treatment2.setValueAttribute("");

s1.setChecked(true);
s2.setChecked(false);

t1.setChecked(true);
t2.setChecked(false);

Calculate.click();

// Printing the Output
System.out.println(result.asText());

webClient.closeAllWindows();

}

}









share|improve this question
























  • What is your question? And don’t tell me it’s “how to do it?” because that is way to broad.
    – Joakim Danielson
    Nov 17 '18 at 8:56










  • my problem is that I can not connect to HTML page and do what I want.I'm looking for the problem of my code not the way to implement it.
    – Farid Amyaie
    Nov 17 '18 at 9:51










  • Update your question with your problem then including any errors and what part of the code that gives you trouble.
    – Joakim Danielson
    Nov 17 '18 at 11:36
















0














Here is my code. I want to get access to a website and compare my data. I want java to put data in fields and auto click on calculate bottom and return the answer to java.



import com.gargoylesoftware.htmlunit.WebClient;

public class MyWebServiceAccess {

public static void main(String args) throws Exception
{

final WebClient webClient = new WebClient();

final HtmlPage page = webClient.getPage("https://www.socscistatistics.com/tests/signedranks/Default2.aspx");

// Inputs
HtmlTextInput treatment1 = (HtmlTextInput) page.getElementById("ctl00_MainContent_TextBox1");
HtmlTextInput treatment2 = (HtmlTextInput) page.getElementById("ctl00_MainContent_TextBox2");

// Significance Level:
HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");


// 1 or 2-tailed hypothesis?:
HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");

// Calculate
HtmlSubmitInput Calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button2");

// Result Span
HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");

// Fill in Inputs
treatment1.setValueAttribute("");
treatment2.setValueAttribute("");

s1.setChecked(true);
s2.setChecked(false);

t1.setChecked(true);
t2.setChecked(false);

Calculate.click();

// Printing the Output
System.out.println(result.asText());

webClient.closeAllWindows();

}

}









share|improve this question
























  • What is your question? And don’t tell me it’s “how to do it?” because that is way to broad.
    – Joakim Danielson
    Nov 17 '18 at 8:56










  • my problem is that I can not connect to HTML page and do what I want.I'm looking for the problem of my code not the way to implement it.
    – Farid Amyaie
    Nov 17 '18 at 9:51










  • Update your question with your problem then including any errors and what part of the code that gives you trouble.
    – Joakim Danielson
    Nov 17 '18 at 11:36














0












0








0







Here is my code. I want to get access to a website and compare my data. I want java to put data in fields and auto click on calculate bottom and return the answer to java.



import com.gargoylesoftware.htmlunit.WebClient;

public class MyWebServiceAccess {

public static void main(String args) throws Exception
{

final WebClient webClient = new WebClient();

final HtmlPage page = webClient.getPage("https://www.socscistatistics.com/tests/signedranks/Default2.aspx");

// Inputs
HtmlTextInput treatment1 = (HtmlTextInput) page.getElementById("ctl00_MainContent_TextBox1");
HtmlTextInput treatment2 = (HtmlTextInput) page.getElementById("ctl00_MainContent_TextBox2");

// Significance Level:
HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");


// 1 or 2-tailed hypothesis?:
HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");

// Calculate
HtmlSubmitInput Calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button2");

// Result Span
HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");

// Fill in Inputs
treatment1.setValueAttribute("");
treatment2.setValueAttribute("");

s1.setChecked(true);
s2.setChecked(false);

t1.setChecked(true);
t2.setChecked(false);

Calculate.click();

// Printing the Output
System.out.println(result.asText());

webClient.closeAllWindows();

}

}









share|improve this question















Here is my code. I want to get access to a website and compare my data. I want java to put data in fields and auto click on calculate bottom and return the answer to java.



import com.gargoylesoftware.htmlunit.WebClient;

public class MyWebServiceAccess {

public static void main(String args) throws Exception
{

final WebClient webClient = new WebClient();

final HtmlPage page = webClient.getPage("https://www.socscistatistics.com/tests/signedranks/Default2.aspx");

// Inputs
HtmlTextInput treatment1 = (HtmlTextInput) page.getElementById("ctl00_MainContent_TextBox1");
HtmlTextInput treatment2 = (HtmlTextInput) page.getElementById("ctl00_MainContent_TextBox2");

// Significance Level:
HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");


// 1 or 2-tailed hypothesis?:
HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");

// Calculate
HtmlSubmitInput Calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button2");

// Result Span
HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");

// Fill in Inputs
treatment1.setValueAttribute("");
treatment2.setValueAttribute("");

s1.setChecked(true);
s2.setChecked(false);

t1.setChecked(true);
t2.setChecked(false);

Calculate.click();

// Printing the Output
System.out.println(result.asText());

webClient.closeAllWindows();

}

}






java web htmlunit






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 17 '18 at 8:27









lzagkaretos

1,7572615




1,7572615










asked Nov 17 '18 at 8:19









Farid Amyaie

12




12












  • What is your question? And don’t tell me it’s “how to do it?” because that is way to broad.
    – Joakim Danielson
    Nov 17 '18 at 8:56










  • my problem is that I can not connect to HTML page and do what I want.I'm looking for the problem of my code not the way to implement it.
    – Farid Amyaie
    Nov 17 '18 at 9:51










  • Update your question with your problem then including any errors and what part of the code that gives you trouble.
    – Joakim Danielson
    Nov 17 '18 at 11:36


















  • What is your question? And don’t tell me it’s “how to do it?” because that is way to broad.
    – Joakim Danielson
    Nov 17 '18 at 8:56










  • my problem is that I can not connect to HTML page and do what I want.I'm looking for the problem of my code not the way to implement it.
    – Farid Amyaie
    Nov 17 '18 at 9:51










  • Update your question with your problem then including any errors and what part of the code that gives you trouble.
    – Joakim Danielson
    Nov 17 '18 at 11:36
















What is your question? And don’t tell me it’s “how to do it?” because that is way to broad.
– Joakim Danielson
Nov 17 '18 at 8:56




What is your question? And don’t tell me it’s “how to do it?” because that is way to broad.
– Joakim Danielson
Nov 17 '18 at 8:56












my problem is that I can not connect to HTML page and do what I want.I'm looking for the problem of my code not the way to implement it.
– Farid Amyaie
Nov 17 '18 at 9:51




my problem is that I can not connect to HTML page and do what I want.I'm looking for the problem of my code not the way to implement it.
– Farid Amyaie
Nov 17 '18 at 9:51












Update your question with your problem then including any errors and what part of the code that gives you trouble.
– Joakim Danielson
Nov 17 '18 at 11:36




Update your question with your problem then including any errors and what part of the code that gives you trouble.
– Joakim Danielson
Nov 17 '18 at 11:36












1 Answer
1






active

oldest

votes


















0














When scrapping web pages you need a basic understanding how the web technologies (http/html) are working and you also need some java/programming knowledge. At least it is really helpful to be able to find problems in your programs.



At first your code produces a class cast exception because the input fields are text areas and not input text controls. At second you have to click the right button (your code clicks the 'reset' button). And finally you got a new page if the button is clicked. Your result is on the new page.



Hope that helps....



String url = "https://www.socscistatistics.com/tests/signedranks/Default2.aspx";                                   

try (final WebClient webClient = new WebClient()) {
HtmlPage page = webClient.getPage(url);

// Inputs
HtmlTextArea treatment1 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox1");
HtmlTextArea treatment2 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox2");

// Significance Level:
HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");
s1.setChecked(true);
s2.setChecked(false);

// 1 or 2-tailed hypothesis?:
HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");
t1.setChecked(true);
t2.setChecked(false);

// Fill in Inputs
treatment1.type("4n3n2n5n5n3");
treatment2.type("1n2n3n0n0n2");

// click Calculate creates a new page
HtmlSubmitInput calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button1");
page = calculate.click();

// Result Span
HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");

// Printing the Output
System.out.println(result.asText());
}





share|improve this answer























    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53349504%2fscapping-web-page-using-htmlunit%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0














    When scrapping web pages you need a basic understanding how the web technologies (http/html) are working and you also need some java/programming knowledge. At least it is really helpful to be able to find problems in your programs.



    At first your code produces a class cast exception because the input fields are text areas and not input text controls. At second you have to click the right button (your code clicks the 'reset' button). And finally you got a new page if the button is clicked. Your result is on the new page.



    Hope that helps....



    String url = "https://www.socscistatistics.com/tests/signedranks/Default2.aspx";                                   

    try (final WebClient webClient = new WebClient()) {
    HtmlPage page = webClient.getPage(url);

    // Inputs
    HtmlTextArea treatment1 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox1");
    HtmlTextArea treatment2 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox2");

    // Significance Level:
    HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
    HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");
    s1.setChecked(true);
    s2.setChecked(false);

    // 1 or 2-tailed hypothesis?:
    HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
    HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");
    t1.setChecked(true);
    t2.setChecked(false);

    // Fill in Inputs
    treatment1.type("4n3n2n5n5n3");
    treatment2.type("1n2n3n0n0n2");

    // click Calculate creates a new page
    HtmlSubmitInput calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button1");
    page = calculate.click();

    // Result Span
    HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");

    // Printing the Output
    System.out.println(result.asText());
    }





    share|improve this answer




























      0














      When scrapping web pages you need a basic understanding how the web technologies (http/html) are working and you also need some java/programming knowledge. At least it is really helpful to be able to find problems in your programs.



      At first your code produces a class cast exception because the input fields are text areas and not input text controls. At second you have to click the right button (your code clicks the 'reset' button). And finally you got a new page if the button is clicked. Your result is on the new page.



      Hope that helps....



      String url = "https://www.socscistatistics.com/tests/signedranks/Default2.aspx";                                   

      try (final WebClient webClient = new WebClient()) {
      HtmlPage page = webClient.getPage(url);

      // Inputs
      HtmlTextArea treatment1 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox1");
      HtmlTextArea treatment2 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox2");

      // Significance Level:
      HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
      HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");
      s1.setChecked(true);
      s2.setChecked(false);

      // 1 or 2-tailed hypothesis?:
      HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
      HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");
      t1.setChecked(true);
      t2.setChecked(false);

      // Fill in Inputs
      treatment1.type("4n3n2n5n5n3");
      treatment2.type("1n2n3n0n0n2");

      // click Calculate creates a new page
      HtmlSubmitInput calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button1");
      page = calculate.click();

      // Result Span
      HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");

      // Printing the Output
      System.out.println(result.asText());
      }





      share|improve this answer


























        0












        0








        0






        When scrapping web pages you need a basic understanding how the web technologies (http/html) are working and you also need some java/programming knowledge. At least it is really helpful to be able to find problems in your programs.



        At first your code produces a class cast exception because the input fields are text areas and not input text controls. At second you have to click the right button (your code clicks the 'reset' button). And finally you got a new page if the button is clicked. Your result is on the new page.



        Hope that helps....



        String url = "https://www.socscistatistics.com/tests/signedranks/Default2.aspx";                                   

        try (final WebClient webClient = new WebClient()) {
        HtmlPage page = webClient.getPage(url);

        // Inputs
        HtmlTextArea treatment1 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox1");
        HtmlTextArea treatment2 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox2");

        // Significance Level:
        HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
        HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");
        s1.setChecked(true);
        s2.setChecked(false);

        // 1 or 2-tailed hypothesis?:
        HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
        HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");
        t1.setChecked(true);
        t2.setChecked(false);

        // Fill in Inputs
        treatment1.type("4n3n2n5n5n3");
        treatment2.type("1n2n3n0n0n2");

        // click Calculate creates a new page
        HtmlSubmitInput calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button1");
        page = calculate.click();

        // Result Span
        HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");

        // Printing the Output
        System.out.println(result.asText());
        }





        share|improve this answer














        When scrapping web pages you need a basic understanding how the web technologies (http/html) are working and you also need some java/programming knowledge. At least it is really helpful to be able to find problems in your programs.



        At first your code produces a class cast exception because the input fields are text areas and not input text controls. At second you have to click the right button (your code clicks the 'reset' button). And finally you got a new page if the button is clicked. Your result is on the new page.



        Hope that helps....



        String url = "https://www.socscistatistics.com/tests/signedranks/Default2.aspx";                                   

        try (final WebClient webClient = new WebClient()) {
        HtmlPage page = webClient.getPage(url);

        // Inputs
        HtmlTextArea treatment1 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox1");
        HtmlTextArea treatment2 = (HtmlTextArea) page.getElementById("ctl00_MainContent_TextBox2");

        // Significance Level:
        HtmlRadioButtonInput s1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_0");
        HtmlRadioButtonInput s2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList1_1");
        s1.setChecked(true);
        s2.setChecked(false);

        // 1 or 2-tailed hypothesis?:
        HtmlRadioButtonInput t1= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_0");
        HtmlRadioButtonInput t2= (HtmlRadioButtonInput) page.getElementById("ctl00_MainContent_RadioButtonList2_1");
        t1.setChecked(true);
        t2.setChecked(false);

        // Fill in Inputs
        treatment1.type("4n3n2n5n5n3");
        treatment2.type("1n2n3n0n0n2");

        // click Calculate creates a new page
        HtmlSubmitInput calculate= (HtmlSubmitInput) page.getElementById("ctl00_MainContent_Button1");
        page = calculate.click();

        // Result Span
        HtmlSpan result = (HtmlSpan) page.getElementById("ctl00_MainContent_Label9");

        // Printing the Output
        System.out.println(result.asText());
        }






        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited Nov 18 '18 at 21:09

























        answered Nov 17 '18 at 19:22









        RBRi

        1,241269




        1,241269






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.





            Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


            Please pay close attention to the following guidance:


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53349504%2fscapping-web-page-using-htmlunit%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Biblatex bibliography style without URLs when DOI exists (in Overleaf with Zotero bibliography)

            ComboBox Display Member on multiple fields

            Is it possible to collect Nectar points via Trainline?