Python: HTTPS using a proxy server with requests
up vote
2
down vote
favorite
I'm using MyIPHide. I downloaded their client software, installed it and have the service turned on.
I can access https
websites fine with a browser but I cannot use requests
to get the pages
This works:
import requests
IP=requests.get('http://api.ipify.org').text
proxyDict = { "http" : IP,
"https" : IP
}
url='http://www.cnn.com'
r=requests.get(url,proxies=proxyDict)
This doesn't:
url='https://www.cnn.com'
r=requests.get(url,proxies=proxyDict)
only difference is http
vs https
here is the traceback:
File "C:Python27libsite-packagesrequestsadapters.py", line 502, in send
raise ProxyError(e, request=request)
ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(10053, 'An established connection was aborted by the software in your host machine')))
I've tried other https
websites, they all don't work.
I have also emailed support at MyIPHide. They said all proxies support https
, which is true when I use a browser only.
One work around that works is if I use Selenium
and get the page, then use driver.page_source
for text.
It's not a proxy server problem because I have bought a private proxy server address through sslprivateproxy.com and put in the IP and port and I still get the same errors.
I'm using Python 2.7.15 and requests 2.20.1. Non proxy use of requests works, ie:
import requests
url='https://www.cnn.com'
r=requests.get(url)
>>> r
<Response [200]>
>>>
Also tried python 3.6 with requests 2.20.1 --> same results.
python https proxy python-requests
|
show 1 more comment
up vote
2
down vote
favorite
I'm using MyIPHide. I downloaded their client software, installed it and have the service turned on.
I can access https
websites fine with a browser but I cannot use requests
to get the pages
This works:
import requests
IP=requests.get('http://api.ipify.org').text
proxyDict = { "http" : IP,
"https" : IP
}
url='http://www.cnn.com'
r=requests.get(url,proxies=proxyDict)
This doesn't:
url='https://www.cnn.com'
r=requests.get(url,proxies=proxyDict)
only difference is http
vs https
here is the traceback:
File "C:Python27libsite-packagesrequestsadapters.py", line 502, in send
raise ProxyError(e, request=request)
ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(10053, 'An established connection was aborted by the software in your host machine')))
I've tried other https
websites, they all don't work.
I have also emailed support at MyIPHide. They said all proxies support https
, which is true when I use a browser only.
One work around that works is if I use Selenium
and get the page, then use driver.page_source
for text.
It's not a proxy server problem because I have bought a private proxy server address through sslprivateproxy.com and put in the IP and port and I still get the same errors.
I'm using Python 2.7.15 and requests 2.20.1. Non proxy use of requests works, ie:
import requests
url='https://www.cnn.com'
r=requests.get(url)
>>> r
<Response [200]>
>>>
Also tried python 3.6 with requests 2.20.1 --> same results.
python https proxy python-requests
did you add the port? {"https":"ip:port"}. Or try{"https":"183.88.219.163:57457"}
– kcorlidy
Nov 15 at 4:11
i don't think you need to. there is no port specified.
– jason
Nov 15 at 4:14
If my proxy work for you that means the issue occured on your proxy
– kcorlidy
Nov 15 at 4:15
@kcorlidy. Thanks for the proxy. It looks like it also doesn't work.ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0000000003CD9208>: Failed to establish a new connection: [Errno 10061] No connection could be made because the target machine actively refused it',)))
– jason
Nov 15 at 14:24
yes, not all nodes work for me too. But also you can find one on free-proxy-list.net. If you still go wrong. Tell me your platform,Python version and requests version, in order to i can get same error as you then find a solution.
– kcorlidy
Nov 15 at 14:35
|
show 1 more comment
up vote
2
down vote
favorite
up vote
2
down vote
favorite
I'm using MyIPHide. I downloaded their client software, installed it and have the service turned on.
I can access https
websites fine with a browser but I cannot use requests
to get the pages
This works:
import requests
IP=requests.get('http://api.ipify.org').text
proxyDict = { "http" : IP,
"https" : IP
}
url='http://www.cnn.com'
r=requests.get(url,proxies=proxyDict)
This doesn't:
url='https://www.cnn.com'
r=requests.get(url,proxies=proxyDict)
only difference is http
vs https
here is the traceback:
File "C:Python27libsite-packagesrequestsadapters.py", line 502, in send
raise ProxyError(e, request=request)
ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(10053, 'An established connection was aborted by the software in your host machine')))
I've tried other https
websites, they all don't work.
I have also emailed support at MyIPHide. They said all proxies support https
, which is true when I use a browser only.
One work around that works is if I use Selenium
and get the page, then use driver.page_source
for text.
It's not a proxy server problem because I have bought a private proxy server address through sslprivateproxy.com and put in the IP and port and I still get the same errors.
I'm using Python 2.7.15 and requests 2.20.1. Non proxy use of requests works, ie:
import requests
url='https://www.cnn.com'
r=requests.get(url)
>>> r
<Response [200]>
>>>
Also tried python 3.6 with requests 2.20.1 --> same results.
python https proxy python-requests
I'm using MyIPHide. I downloaded their client software, installed it and have the service turned on.
I can access https
websites fine with a browser but I cannot use requests
to get the pages
This works:
import requests
IP=requests.get('http://api.ipify.org').text
proxyDict = { "http" : IP,
"https" : IP
}
url='http://www.cnn.com'
r=requests.get(url,proxies=proxyDict)
This doesn't:
url='https://www.cnn.com'
r=requests.get(url,proxies=proxyDict)
only difference is http
vs https
here is the traceback:
File "C:Python27libsite-packagesrequestsadapters.py", line 502, in send
raise ProxyError(e, request=request)
ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(10053, 'An established connection was aborted by the software in your host machine')))
I've tried other https
websites, they all don't work.
I have also emailed support at MyIPHide. They said all proxies support https
, which is true when I use a browser only.
One work around that works is if I use Selenium
and get the page, then use driver.page_source
for text.
It's not a proxy server problem because I have bought a private proxy server address through sslprivateproxy.com and put in the IP and port and I still get the same errors.
I'm using Python 2.7.15 and requests 2.20.1. Non proxy use of requests works, ie:
import requests
url='https://www.cnn.com'
r=requests.get(url)
>>> r
<Response [200]>
>>>
Also tried python 3.6 with requests 2.20.1 --> same results.
python https proxy python-requests
python https proxy python-requests
edited Nov 16 at 19:06
asked Nov 15 at 1:27
jason
8253686
8253686
did you add the port? {"https":"ip:port"}. Or try{"https":"183.88.219.163:57457"}
– kcorlidy
Nov 15 at 4:11
i don't think you need to. there is no port specified.
– jason
Nov 15 at 4:14
If my proxy work for you that means the issue occured on your proxy
– kcorlidy
Nov 15 at 4:15
@kcorlidy. Thanks for the proxy. It looks like it also doesn't work.ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0000000003CD9208>: Failed to establish a new connection: [Errno 10061] No connection could be made because the target machine actively refused it',)))
– jason
Nov 15 at 14:24
yes, not all nodes work for me too. But also you can find one on free-proxy-list.net. If you still go wrong. Tell me your platform,Python version and requests version, in order to i can get same error as you then find a solution.
– kcorlidy
Nov 15 at 14:35
|
show 1 more comment
did you add the port? {"https":"ip:port"}. Or try{"https":"183.88.219.163:57457"}
– kcorlidy
Nov 15 at 4:11
i don't think you need to. there is no port specified.
– jason
Nov 15 at 4:14
If my proxy work for you that means the issue occured on your proxy
– kcorlidy
Nov 15 at 4:15
@kcorlidy. Thanks for the proxy. It looks like it also doesn't work.ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0000000003CD9208>: Failed to establish a new connection: [Errno 10061] No connection could be made because the target machine actively refused it',)))
– jason
Nov 15 at 14:24
yes, not all nodes work for me too. But also you can find one on free-proxy-list.net. If you still go wrong. Tell me your platform,Python version and requests version, in order to i can get same error as you then find a solution.
– kcorlidy
Nov 15 at 14:35
did you add the port? {"https":"ip:port"}. Or try
{"https":"183.88.219.163:57457"}
– kcorlidy
Nov 15 at 4:11
did you add the port? {"https":"ip:port"}. Or try
{"https":"183.88.219.163:57457"}
– kcorlidy
Nov 15 at 4:11
i don't think you need to. there is no port specified.
– jason
Nov 15 at 4:14
i don't think you need to. there is no port specified.
– jason
Nov 15 at 4:14
If my proxy work for you that means the issue occured on your proxy
– kcorlidy
Nov 15 at 4:15
If my proxy work for you that means the issue occured on your proxy
– kcorlidy
Nov 15 at 4:15
@kcorlidy. Thanks for the proxy. It looks like it also doesn't work.
ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0000000003CD9208>: Failed to establish a new connection: [Errno 10061] No connection could be made because the target machine actively refused it',)))
– jason
Nov 15 at 14:24
@kcorlidy. Thanks for the proxy. It looks like it also doesn't work.
ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0000000003CD9208>: Failed to establish a new connection: [Errno 10061] No connection could be made because the target machine actively refused it',)))
– jason
Nov 15 at 14:24
yes, not all nodes work for me too. But also you can find one on free-proxy-list.net. If you still go wrong. Tell me your platform,Python version and requests version, in order to i can get same error as you then find a solution.
– kcorlidy
Nov 15 at 14:35
yes, not all nodes work for me too. But also you can find one on free-proxy-list.net. If you still go wrong. Tell me your platform,Python version and requests version, in order to i can get same error as you then find a solution.
– kcorlidy
Nov 15 at 14:35
|
show 1 more comment
1 Answer
1
active
oldest
votes
up vote
1
down vote
On requests it based on PoolManager size, so if you want to have more connection in pool you can reset the size of HTTPAdapter()
. The way like that(to ensure if it works you can set to 0).
import requests
from requests.adapters import HTTPAdapter
proxy = {"http":"118.174.233.31:51726",
"https":"43.254.132.86:50659"} # free proxy,often need to modify
with requests.Session() as se:
# pool_connections=pool_maxsize=0 -> pool closed
se.mount('https://', HTTPAdapter(pool_connections=100,pool_maxsize=100))
se.mount('http://', HTTPAdapter(pool_connections=100,pool_maxsize=100))
print(se.adapters["https://"]._pool_maxsize)
#print(se.get("https://github.com"))
print(se.get('http://api.ipify.org',proxies=proxy).text)
#100
#118.174.233.31
Ran on Windows10, Python27, requests-2.20.1
– kcorlidy
Nov 15 at 15:41
Thanks for trying, but I still get the same error when I make a page request.url='https://www.cnn.com' r=se.get(url,proxies=proxy)
gives meProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(10053, 'An established connection was aborted by the software in your host machine')))
– jason
Nov 15 at 16:02
what about your requests_version, and can you access website without https proxy?
– kcorlidy
Nov 15 at 16:16
yes I can.requests == 2.18.4
. I just upgraded to2.20.1
. Same error. I'm running python 2.7.15
– jason
Nov 15 at 16:32
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
On requests it based on PoolManager size, so if you want to have more connection in pool you can reset the size of HTTPAdapter()
. The way like that(to ensure if it works you can set to 0).
import requests
from requests.adapters import HTTPAdapter
proxy = {"http":"118.174.233.31:51726",
"https":"43.254.132.86:50659"} # free proxy,often need to modify
with requests.Session() as se:
# pool_connections=pool_maxsize=0 -> pool closed
se.mount('https://', HTTPAdapter(pool_connections=100,pool_maxsize=100))
se.mount('http://', HTTPAdapter(pool_connections=100,pool_maxsize=100))
print(se.adapters["https://"]._pool_maxsize)
#print(se.get("https://github.com"))
print(se.get('http://api.ipify.org',proxies=proxy).text)
#100
#118.174.233.31
Ran on Windows10, Python27, requests-2.20.1
– kcorlidy
Nov 15 at 15:41
Thanks for trying, but I still get the same error when I make a page request.url='https://www.cnn.com' r=se.get(url,proxies=proxy)
gives meProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(10053, 'An established connection was aborted by the software in your host machine')))
– jason
Nov 15 at 16:02
what about your requests_version, and can you access website without https proxy?
– kcorlidy
Nov 15 at 16:16
yes I can.requests == 2.18.4
. I just upgraded to2.20.1
. Same error. I'm running python 2.7.15
– jason
Nov 15 at 16:32
add a comment |
up vote
1
down vote
On requests it based on PoolManager size, so if you want to have more connection in pool you can reset the size of HTTPAdapter()
. The way like that(to ensure if it works you can set to 0).
import requests
from requests.adapters import HTTPAdapter
proxy = {"http":"118.174.233.31:51726",
"https":"43.254.132.86:50659"} # free proxy,often need to modify
with requests.Session() as se:
# pool_connections=pool_maxsize=0 -> pool closed
se.mount('https://', HTTPAdapter(pool_connections=100,pool_maxsize=100))
se.mount('http://', HTTPAdapter(pool_connections=100,pool_maxsize=100))
print(se.adapters["https://"]._pool_maxsize)
#print(se.get("https://github.com"))
print(se.get('http://api.ipify.org',proxies=proxy).text)
#100
#118.174.233.31
Ran on Windows10, Python27, requests-2.20.1
– kcorlidy
Nov 15 at 15:41
Thanks for trying, but I still get the same error when I make a page request.url='https://www.cnn.com' r=se.get(url,proxies=proxy)
gives meProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(10053, 'An established connection was aborted by the software in your host machine')))
– jason
Nov 15 at 16:02
what about your requests_version, and can you access website without https proxy?
– kcorlidy
Nov 15 at 16:16
yes I can.requests == 2.18.4
. I just upgraded to2.20.1
. Same error. I'm running python 2.7.15
– jason
Nov 15 at 16:32
add a comment |
up vote
1
down vote
up vote
1
down vote
On requests it based on PoolManager size, so if you want to have more connection in pool you can reset the size of HTTPAdapter()
. The way like that(to ensure if it works you can set to 0).
import requests
from requests.adapters import HTTPAdapter
proxy = {"http":"118.174.233.31:51726",
"https":"43.254.132.86:50659"} # free proxy,often need to modify
with requests.Session() as se:
# pool_connections=pool_maxsize=0 -> pool closed
se.mount('https://', HTTPAdapter(pool_connections=100,pool_maxsize=100))
se.mount('http://', HTTPAdapter(pool_connections=100,pool_maxsize=100))
print(se.adapters["https://"]._pool_maxsize)
#print(se.get("https://github.com"))
print(se.get('http://api.ipify.org',proxies=proxy).text)
#100
#118.174.233.31
On requests it based on PoolManager size, so if you want to have more connection in pool you can reset the size of HTTPAdapter()
. The way like that(to ensure if it works you can set to 0).
import requests
from requests.adapters import HTTPAdapter
proxy = {"http":"118.174.233.31:51726",
"https":"43.254.132.86:50659"} # free proxy,often need to modify
with requests.Session() as se:
# pool_connections=pool_maxsize=0 -> pool closed
se.mount('https://', HTTPAdapter(pool_connections=100,pool_maxsize=100))
se.mount('http://', HTTPAdapter(pool_connections=100,pool_maxsize=100))
print(se.adapters["https://"]._pool_maxsize)
#print(se.get("https://github.com"))
print(se.get('http://api.ipify.org',proxies=proxy).text)
#100
#118.174.233.31
answered Nov 15 at 15:35
kcorlidy
1,699317
1,699317
Ran on Windows10, Python27, requests-2.20.1
– kcorlidy
Nov 15 at 15:41
Thanks for trying, but I still get the same error when I make a page request.url='https://www.cnn.com' r=se.get(url,proxies=proxy)
gives meProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(10053, 'An established connection was aborted by the software in your host machine')))
– jason
Nov 15 at 16:02
what about your requests_version, and can you access website without https proxy?
– kcorlidy
Nov 15 at 16:16
yes I can.requests == 2.18.4
. I just upgraded to2.20.1
. Same error. I'm running python 2.7.15
– jason
Nov 15 at 16:32
add a comment |
Ran on Windows10, Python27, requests-2.20.1
– kcorlidy
Nov 15 at 15:41
Thanks for trying, but I still get the same error when I make a page request.url='https://www.cnn.com' r=se.get(url,proxies=proxy)
gives meProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(10053, 'An established connection was aborted by the software in your host machine')))
– jason
Nov 15 at 16:02
what about your requests_version, and can you access website without https proxy?
– kcorlidy
Nov 15 at 16:16
yes I can.requests == 2.18.4
. I just upgraded to2.20.1
. Same error. I'm running python 2.7.15
– jason
Nov 15 at 16:32
Ran on Windows10, Python27, requests-2.20.1
– kcorlidy
Nov 15 at 15:41
Ran on Windows10, Python27, requests-2.20.1
– kcorlidy
Nov 15 at 15:41
Thanks for trying, but I still get the same error when I make a page request.
url='https://www.cnn.com' r=se.get(url,proxies=proxy)
gives me ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(10053, 'An established connection was aborted by the software in your host machine')))
– jason
Nov 15 at 16:02
Thanks for trying, but I still get the same error when I make a page request.
url='https://www.cnn.com' r=se.get(url,proxies=proxy)
gives me ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(10053, 'An established connection was aborted by the software in your host machine')))
– jason
Nov 15 at 16:02
what about your requests_version, and can you access website without https proxy?
– kcorlidy
Nov 15 at 16:16
what about your requests_version, and can you access website without https proxy?
– kcorlidy
Nov 15 at 16:16
yes I can.
requests == 2.18.4
. I just upgraded to 2.20.1
. Same error. I'm running python 2.7.15– jason
Nov 15 at 16:32
yes I can.
requests == 2.18.4
. I just upgraded to 2.20.1
. Same error. I'm running python 2.7.15– jason
Nov 15 at 16:32
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53311168%2fpython-https-using-a-proxy-server-with-requests%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
did you add the port? {"https":"ip:port"}. Or try
{"https":"183.88.219.163:57457"}
– kcorlidy
Nov 15 at 4:11
i don't think you need to. there is no port specified.
– jason
Nov 15 at 4:14
If my proxy work for you that means the issue occured on your proxy
– kcorlidy
Nov 15 at 4:15
@kcorlidy. Thanks for the proxy. It looks like it also doesn't work.
ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0000000003CD9208>: Failed to establish a new connection: [Errno 10061] No connection could be made because the target machine actively refused it',)))
– jason
Nov 15 at 14:24
yes, not all nodes work for me too. But also you can find one on free-proxy-list.net. If you still go wrong. Tell me your platform,Python version and requests version, in order to i can get same error as you then find a solution.
– kcorlidy
Nov 15 at 14:35