Python request response is empty
My code below successfully pulled data from https://permit.pcta.org/application/
import requests
from tabulate import tabulate
import time
'''outputs '''
PERMITS =
'''input'''
url = 'https://permit.pcta.org/manage/ajax/onload-ajax.php'
headers = {
'User-Agent': 'Mozilla/5.0',
'X-Requested-With': 'XMLHttpRequest'
}
r = requests.get(url, headers=headers)
print(r.content)
However, the response is empty. I am positive I didn't change anything in my code, so I can't explain why the website's response is now different.
As a sanity check below are snippets of the element and response from my browser.
html element
response
python python-3.x python-requests
edited Nov 26 '18 at 23:13
Spacedman
73.2k995169
asked Nov 21 '18 at 1:56
chacoschacos
42
|
show 2 more comments
My code below successfully pulled data from https://permit.pcta.org/application/
import requests
from tabulate import tabulate
import time
'''outputs '''
PERMITS =
'''input'''
url = 'https://permit.pcta.org/manage/ajax/onload-ajax.php'
headers = {
'User-Agent': 'Mozilla/5.0',
'X-Requested-With': 'XMLHttpRequest'
}
r = requests.get(url, headers=headers)
print(r.content)
However, the response is empty. I am positive I didn't change anything in my code, so I can't explain why the website's response is now different.
As a sanity check below are snippets of the element and response from my browser.
html element
response
python python-3.x python-requests
edited Nov 26 '18 at 23:13
Spacedman
73.2k995169
asked Nov 21 '18 at 1:56
chacoschacos
42
What is your question? Hitting that URL returns a 404 for me.
– Matt Morgan
Nov 21 '18 at 2:01
why the code I copied is not working anymore. The URL works, just tried it.
– chacos
Nov 21 '18 at 2:13
Did you tried to specify an Accept header with Application/JSON value?
– Jacobo Tapia
Nov 21 '18 at 2:42
Just tried that. I get the same error mentioned above.
– chacos
Nov 21 '18 at 3:13
One reason is you may use post method to get data. And the another reason should be login required before access
– kcorlidy
Nov 21 '18 at 6:30
|
show 2 more comments
My code below successfully pulled data from https://permit.pcta.org/application/
import requests
from tabulate import tabulate
import time
'''outputs '''
PERMITS =
'''input'''
url = 'https://permit.pcta.org/manage/ajax/onload-ajax.php'
headers = {
'User-Agent': 'Mozilla/5.0',
'X-Requested-With': 'XMLHttpRequest'
}
r = requests.get(url, headers=headers)
print(r.content)
However, the response is empty. I am positive I didn't change anything in my code, so I can't explain why the website's response is now different.
As a sanity check below are snippets of the element and response from my browser.
html element
response
python python-3.x python-requests
edited Nov 26 '18 at 23:13
Spacedman
73.2k995169
asked Nov 21 '18 at 1:56
chacoschacos
42
My code below successfully pulled data from https://permit.pcta.org/application/
import requests
from tabulate import tabulate
import time
'''outputs '''
PERMITS =
'''input'''
url = 'https://permit.pcta.org/manage/ajax/onload-ajax.php'
headers = {
'User-Agent': 'Mozilla/5.0',
'X-Requested-With': 'XMLHttpRequest'
}
r = requests.get(url, headers=headers)
print(r.content)
However, the response is empty. I am positive I didn't change anything in my code, so I can't explain why the website's response is now different.
As a sanity check below are snippets of the element and response from my browser.
html element
response
python python-3.x python-requests
python python-3.x python-requests
edited Nov 26 '18 at 23:13
Spacedman
73.2k995169
asked Nov 21 '18 at 1:56
chacoschacos
42
edited Nov 26 '18 at 23:13
Spacedman
73.2k995169
asked Nov 21 '18 at 1:56
chacoschacos
42
edited Nov 26 '18 at 23:13
Spacedman
73.2k995169
edited Nov 26 '18 at 23:13
Spacedman
73.2k995169
edited Nov 26 '18 at 23:13
Spacedman
73.2k995169
73.2k995169
asked Nov 21 '18 at 1:56
chacoschacos
42
asked Nov 21 '18 at 1:56
chacoschacos
42
asked Nov 21 '18 at 1:56
chacoschacos
42
42
What is your question? Hitting that URL returns a 404 for me.
– Matt Morgan
Nov 21 '18 at 2:01
why the code I copied is not working anymore. The URL works, just tried it.
– chacos
Nov 21 '18 at 2:13
Did you tried to specify an Accept header with Application/JSON value?
– Jacobo Tapia
Nov 21 '18 at 2:42
Just tried that. I get the same error mentioned above.
– chacos
Nov 21 '18 at 3:13
One reason is you may use post method to get data. And the another reason should be login required before access
– kcorlidy
Nov 21 '18 at 6:30
|
show 2 more comments
What is your question? Hitting that URL returns a 404 for me.
– Matt Morgan
Nov 21 '18 at 2:01
why the code I copied is not working anymore. The URL works, just tried it.
– chacos
Nov 21 '18 at 2:13
Did you tried to specify an Accept header with Application/JSON value?
– Jacobo Tapia
Nov 21 '18 at 2:42
Just tried that. I get the same error mentioned above.
– chacos
Nov 21 '18 at 3:13
One reason is you may use post method to get data. And the another reason should be login required before access
– kcorlidy
Nov 21 '18 at 6:30
What is your question? Hitting that URL returns a 404 for me.
– Matt Morgan
Nov 21 '18 at 2:01
What is your question? Hitting that URL returns a 404 for me.
– Matt Morgan
Nov 21 '18 at 2:01
why the code I copied is not working anymore. The URL works, just tried it.
– chacos
Nov 21 '18 at 2:13
why the code I copied is not working anymore. The URL works, just tried it.
– chacos
Nov 21 '18 at 2:13
Did you tried to specify an Accept header with Application/JSON value?
– Jacobo Tapia
Nov 21 '18 at 2:42
Did you tried to specify an Accept header with Application/JSON value?
– Jacobo Tapia
Nov 21 '18 at 2:42
Just tried that. I get the same error mentioned above.
– chacos
Nov 21 '18 at 3:13
Just tried that. I get the same error mentioned above.
– chacos
Nov 21 '18 at 3:13
One reason is you may use post method to get data. And the another reason should be login required before access
– kcorlidy
Nov 21 '18 at 6:30
One reason is you may use post method to get data. And the another reason should be login required before access
– kcorlidy
Nov 21 '18 at 6:30
|
show 2 more comments
1 Answer
1
active
oldest
votes
It looks like you are missing the Referer in the headers, I added Referer to your code and it worked:
headers = {
'User-Agent': 'Mozilla/5.0',
'X-Requested-With': 'XMLHttpRequest',
'Referer': 'https://permit.pcta.org/application/'
}
Not sure why it worked without it before, probably an api has changed, or you were passing Referer somewhere else in your code.
answered Nov 26 '18 at 3:05
svetlannnkasvetlannnka
814
Thank you! I found this out earlier by copying every header my browser used to make the request. Still not sure how it was working before without the Referer header.
– chacos
Nov 26 '18 at 23:14
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53404286%2fpython-request-response-is-empty%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
It looks like you are missing the Referer in the headers, I added Referer to your code and it worked:
headers = {
'User-Agent': 'Mozilla/5.0',
'X-Requested-With': 'XMLHttpRequest',
'Referer': 'https://permit.pcta.org/application/'
}
Not sure why it worked without it before, probably an api has changed, or you were passing Referer somewhere else in your code.
answered Nov 26 '18 at 3:05
svetlannnkasvetlannnka
814
Thank you! I found this out earlier by copying every header my browser used to make the request. Still not sure how it was working before without the Referer header.
– chacos
Nov 26 '18 at 23:14
add a comment |
It looks like you are missing the Referer in the headers, I added Referer to your code and it worked:
headers = {
'User-Agent': 'Mozilla/5.0',
'X-Requested-With': 'XMLHttpRequest',
'Referer': 'https://permit.pcta.org/application/'
}
Not sure why it worked without it before, probably an api has changed, or you were passing Referer somewhere else in your code.
answered Nov 26 '18 at 3:05
svetlannnkasvetlannnka
814
Thank you! I found this out earlier by copying every header my browser used to make the request. Still not sure how it was working before without the Referer header.
– chacos
Nov 26 '18 at 23:14
add a comment |
It looks like you are missing the Referer in the headers, I added Referer to your code and it worked:
headers = {
'User-Agent': 'Mozilla/5.0',
'X-Requested-With': 'XMLHttpRequest',
'Referer': 'https://permit.pcta.org/application/'
}
Not sure why it worked without it before, probably an api has changed, or you were passing Referer somewhere else in your code.
answered Nov 26 '18 at 3:05
svetlannnkasvetlannnka
814
It looks like you are missing the Referer in the headers, I added Referer to your code and it worked:
headers = {
'User-Agent': 'Mozilla/5.0',
'X-Requested-With': 'XMLHttpRequest',
'Referer': 'https://permit.pcta.org/application/'
}
Not sure why it worked without it before, probably an api has changed, or you were passing Referer somewhere else in your code.
answered Nov 26 '18 at 3:05
svetlannnkasvetlannnka
814
answered Nov 26 '18 at 3:05
svetlannnkasvetlannnka
814
answered Nov 26 '18 at 3:05
svetlannnkasvetlannnka
814
answered Nov 26 '18 at 3:05
svetlannnkasvetlannnka
814
814
Thank you! I found this out earlier by copying every header my browser used to make the request. Still not sure how it was working before without the Referer header.
– chacos
Nov 26 '18 at 23:14
add a comment |
Thank you! I found this out earlier by copying every header my browser used to make the request. Still not sure how it was working before without the Referer header.
– chacos
Nov 26 '18 at 23:14
Thank you! I found this out earlier by copying every header my browser used to make the request. Still not sure how it was working before without the Referer header.
– chacos
Nov 26 '18 at 23:14
Thank you! I found this out earlier by copying every header my browser used to make the request. Still not sure how it was working before without the Referer header.
– chacos
Nov 26 '18 at 23:14
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53404286%2fpython-request-response-is-empty%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What is your question? Hitting that URL returns a 404 for me.
– Matt Morgan
Nov 21 '18 at 2:01
why the code I copied is not working anymore. The URL works, just tried it.
– chacos
Nov 21 '18 at 2:13
Did you tried to specify an Accept header with Application/JSON value?
– Jacobo Tapia
Nov 21 '18 at 2:42
Just tried that. I get the same error mentioned above.
– chacos
Nov 21 '18 at 3:13
One reason is you may use post method to get data. And the another reason should be login required before access
– kcorlidy
Nov 21 '18 at 6:30