Create and Excel Document using Selenium
Hi I am new to Python and am currently working through the book Automate the boring stuff with Python. I am trying to scrape a website that requires a login. The below code does work to a point and saves the required information in 'pricingtable'. I need to get this into an Excel or .txt file is there a way to do this using selenium?
Being new and not fulling understanding is there a better way to do this with requests and bs4? It seems silly to open a browser to do this but i don't know another way.
Thank you in advance.
from selenium import webdriver
import openpyxl, requests, bs4
import time
# opens website in Firefox
browser = webdriver.Firefox()
browser.get('https://somewebsite/somepage)
# find a username element and passes the password
emailElem = browser.find_element_by_id('Enter user name')
emailElem.send_keys('*username*')
# finds the password element and passes the password
passwordElem = browser.find_element_by_name('passwd')
passwordElem.send_keys('*password*')
passwordElem.submit()
#allows the browser to open but may not be required
time.sleep(2)
# selects a new webpage
browser.get('https://somewebsite/pricing')
# on the new webpage there is an element table
pricetable = browser.find_element_by_id('Pricetable')
python selenium beautifulsoup python-requests save
asked Nov 21 '18 at 23:12
Sloth87Sloth87
122
add a comment |
Hi I am new to Python and am currently working through the book Automate the boring stuff with Python. I am trying to scrape a website that requires a login. The below code does work to a point and saves the required information in 'pricingtable'. I need to get this into an Excel or .txt file is there a way to do this using selenium?
Being new and not fulling understanding is there a better way to do this with requests and bs4? It seems silly to open a browser to do this but i don't know another way.
Thank you in advance.
from selenium import webdriver
import openpyxl, requests, bs4
import time
# opens website in Firefox
browser = webdriver.Firefox()
browser.get('https://somewebsite/somepage)
# find a username element and passes the password
emailElem = browser.find_element_by_id('Enter user name')
emailElem.send_keys('*username*')
# finds the password element and passes the password
passwordElem = browser.find_element_by_name('passwd')
passwordElem.send_keys('*password*')
passwordElem.submit()
#allows the browser to open but may not be required
time.sleep(2)
# selects a new webpage
browser.get('https://somewebsite/pricing')
# on the new webpage there is an element table
pricetable = browser.find_element_by_id('Pricetable')
python selenium beautifulsoup python-requests save
asked Nov 21 '18 at 23:12
Sloth87Sloth87
122
How many columns?
– Kamikaze_goldfish
Nov 21 '18 at 23:20
I'm not sure I believe the table has 6 columns
– Sloth87
Nov 22 '18 at 22:17
add a comment |
Hi I am new to Python and am currently working through the book Automate the boring stuff with Python. I am trying to scrape a website that requires a login. The below code does work to a point and saves the required information in 'pricingtable'. I need to get this into an Excel or .txt file is there a way to do this using selenium?
Being new and not fulling understanding is there a better way to do this with requests and bs4? It seems silly to open a browser to do this but i don't know another way.
Thank you in advance.
from selenium import webdriver
import openpyxl, requests, bs4
import time
# opens website in Firefox
browser = webdriver.Firefox()
browser.get('https://somewebsite/somepage)
# find a username element and passes the password
emailElem = browser.find_element_by_id('Enter user name')
emailElem.send_keys('*username*')
# finds the password element and passes the password
passwordElem = browser.find_element_by_name('passwd')
passwordElem.send_keys('*password*')
passwordElem.submit()
#allows the browser to open but may not be required
time.sleep(2)
# selects a new webpage
browser.get('https://somewebsite/pricing')
# on the new webpage there is an element table
pricetable = browser.find_element_by_id('Pricetable')
python selenium beautifulsoup python-requests save
asked Nov 21 '18 at 23:12
Sloth87Sloth87
122
Hi I am new to Python and am currently working through the book Automate the boring stuff with Python. I am trying to scrape a website that requires a login. The below code does work to a point and saves the required information in 'pricingtable'. I need to get this into an Excel or .txt file is there a way to do this using selenium?
Being new and not fulling understanding is there a better way to do this with requests and bs4? It seems silly to open a browser to do this but i don't know another way.
Thank you in advance.
from selenium import webdriver
import openpyxl, requests, bs4
import time
# opens website in Firefox
browser = webdriver.Firefox()
browser.get('https://somewebsite/somepage)
# find a username element and passes the password
emailElem = browser.find_element_by_id('Enter user name')
emailElem.send_keys('*username*')
# finds the password element and passes the password
passwordElem = browser.find_element_by_name('passwd')
passwordElem.send_keys('*password*')
passwordElem.submit()
#allows the browser to open but may not be required
time.sleep(2)
# selects a new webpage
browser.get('https://somewebsite/pricing')
# on the new webpage there is an element table
pricetable = browser.find_element_by_id('Pricetable')
python selenium beautifulsoup python-requests save
python selenium beautifulsoup python-requests save
asked Nov 21 '18 at 23:12
Sloth87Sloth87
122
asked Nov 21 '18 at 23:12
Sloth87Sloth87
122
asked Nov 21 '18 at 23:12
Sloth87Sloth87
122
asked Nov 21 '18 at 23:12
Sloth87Sloth87
122
asked Nov 21 '18 at 23:12
Sloth87Sloth87
122
122
How many columns?
– Kamikaze_goldfish
Nov 21 '18 at 23:20
I'm not sure I believe the table has 6 columns
– Sloth87
Nov 22 '18 at 22:17
add a comment |
How many columns?
– Kamikaze_goldfish
Nov 21 '18 at 23:20
I'm not sure I believe the table has 6 columns
– Sloth87
Nov 22 '18 at 22:17
How many columns?
– Kamikaze_goldfish
Nov 21 '18 at 23:20
How many columns?
– Kamikaze_goldfish
Nov 21 '18 at 23:20
I'm not sure I believe the table has 6 columns
– Sloth87
Nov 22 '18 at 22:17
I'm not sure I believe the table has 6 columns
– Sloth87
Nov 22 '18 at 22:17
add a comment |
1 Answer
1
active
oldest
votes
You can save the table easily in a CSV,
CSV files are file formats that contain plain text values separated by commas. CSV files can be opened by any spreadsheet program: Microsoft Excel, Open Office, Google Sheets, etc.
import csv
#....
pricetable = browser.find_element_by_id('Pricetable')
with open('pricetable.csv', 'a') as f:
wr = csv.writer(f)
for row in pricetable.find_elements_by_css_selector('tr'):
wr.writerow([d.text for d in row.find_elements_by_css_selector('td')])
answered Nov 21 '18 at 23:51
tooTiredtooTired
1697
thanks @too Tired but I get the error : AttributeError: 'FirefoxWebElement' object has no attribute 'to_csv'
– Sloth87
Nov 22 '18 at 11:29
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53421752%2fcreate-and-excel-document-using-selenium%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You can save the table easily in a CSV,
CSV files are file formats that contain plain text values separated by commas. CSV files can be opened by any spreadsheet program: Microsoft Excel, Open Office, Google Sheets, etc.
import csv
#....
pricetable = browser.find_element_by_id('Pricetable')
with open('pricetable.csv', 'a') as f:
wr = csv.writer(f)
for row in pricetable.find_elements_by_css_selector('tr'):
wr.writerow([d.text for d in row.find_elements_by_css_selector('td')])
answered Nov 21 '18 at 23:51
tooTiredtooTired
1697
thanks @too Tired but I get the error : AttributeError: 'FirefoxWebElement' object has no attribute 'to_csv'
– Sloth87
Nov 22 '18 at 11:29
add a comment |
You can save the table easily in a CSV,
CSV files are file formats that contain plain text values separated by commas. CSV files can be opened by any spreadsheet program: Microsoft Excel, Open Office, Google Sheets, etc.
import csv
#....
pricetable = browser.find_element_by_id('Pricetable')
with open('pricetable.csv', 'a') as f:
wr = csv.writer(f)
for row in pricetable.find_elements_by_css_selector('tr'):
wr.writerow([d.text for d in row.find_elements_by_css_selector('td')])
answered Nov 21 '18 at 23:51
tooTiredtooTired
1697
thanks @too Tired but I get the error : AttributeError: 'FirefoxWebElement' object has no attribute 'to_csv'
– Sloth87
Nov 22 '18 at 11:29
add a comment |
You can save the table easily in a CSV,
CSV files are file formats that contain plain text values separated by commas. CSV files can be opened by any spreadsheet program: Microsoft Excel, Open Office, Google Sheets, etc.
import csv
#....
pricetable = browser.find_element_by_id('Pricetable')
with open('pricetable.csv', 'a') as f:
wr = csv.writer(f)
for row in pricetable.find_elements_by_css_selector('tr'):
wr.writerow([d.text for d in row.find_elements_by_css_selector('td')])
answered Nov 21 '18 at 23:51
tooTiredtooTired
1697
You can save the table easily in a CSV,
CSV files are file formats that contain plain text values separated by commas. CSV files can be opened by any spreadsheet program: Microsoft Excel, Open Office, Google Sheets, etc.
import csv
#....
pricetable = browser.find_element_by_id('Pricetable')
with open('pricetable.csv', 'a') as f:
wr = csv.writer(f)
for row in pricetable.find_elements_by_css_selector('tr'):
wr.writerow([d.text for d in row.find_elements_by_css_selector('td')])
answered Nov 21 '18 at 23:51
tooTiredtooTired
1697
edited Nov 23 '18 at 2:10
answered Nov 21 '18 at 23:51
tooTiredtooTired
1697
answered Nov 21 '18 at 23:51
tooTiredtooTired
1697
answered Nov 21 '18 at 23:51
tooTiredtooTired
1697
1697
thanks @too Tired but I get the error : AttributeError: 'FirefoxWebElement' object has no attribute 'to_csv'
– Sloth87
Nov 22 '18 at 11:29
add a comment |
thanks @too Tired but I get the error : AttributeError: 'FirefoxWebElement' object has no attribute 'to_csv'
– Sloth87
Nov 22 '18 at 11:29
thanks @too Tired but I get the error : AttributeError: 'FirefoxWebElement' object has no attribute 'to_csv'
– Sloth87
Nov 22 '18 at 11:29
thanks @too Tired but I get the error : AttributeError: 'FirefoxWebElement' object has no attribute 'to_csv'
– Sloth87
Nov 22 '18 at 11:29
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53421752%2fcreate-and-excel-document-using-selenium%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
How many columns?
– Kamikaze_goldfish
Nov 21 '18 at 23:20
I'm not sure I believe the table has 6 columns
– Sloth87
Nov 22 '18 at 22:17