In GCP, is there a canonical way to scrape data from an API?
I'm building an application that will periodically pull data from several APIs and write them to cloud storage for later processing by Dataflow. There are many different ways to do this so I wanted to sanity check before I jumped in.
My plan is this:
- For each API, Cloud Scheduler will hit an endpoint for an App Engine app
- The app will create a Compute Engine VM instance with a startup script that pulls the data from the API and writes it to storage
- When done, the VM will hit another endpoint on the App Engine app that shuts down the VM.
Is this a reasonable way to perform this sort of action? Is there a simpler or more straight-forward method? Thank you in advance for the replies.
google-app-engine google-cloud-platform
asked Nov 21 '18 at 17:27
littlebenlittlelittlebenlittle
518
add a comment |
I'm building an application that will periodically pull data from several APIs and write them to cloud storage for later processing by Dataflow. There are many different ways to do this so I wanted to sanity check before I jumped in.
My plan is this:
- For each API, Cloud Scheduler will hit an endpoint for an App Engine app
- The app will create a Compute Engine VM instance with a startup script that pulls the data from the API and writes it to storage
- When done, the VM will hit another endpoint on the App Engine app that shuts down the VM.
Is this a reasonable way to perform this sort of action? Is there a simpler or more straight-forward method? Thank you in advance for the replies.
google-app-engine google-cloud-platform
asked Nov 21 '18 at 17:27
littlebenlittlelittlebenlittle
518
1
I'm not sure if it will cover all your needs but Cloud Scheduler can schedule Compute Engine without App Engine however it seems that you cannot create and delete the VM with this method.
– oakinlaja
Nov 21 '18 at 19:04
This seems like a more standard way of doing things. Thank you for the suggestion!
– littlebenlittle
Nov 22 '18 at 1:23
add a comment |
I'm building an application that will periodically pull data from several APIs and write them to cloud storage for later processing by Dataflow. There are many different ways to do this so I wanted to sanity check before I jumped in.
My plan is this:
- For each API, Cloud Scheduler will hit an endpoint for an App Engine app
- The app will create a Compute Engine VM instance with a startup script that pulls the data from the API and writes it to storage
- When done, the VM will hit another endpoint on the App Engine app that shuts down the VM.
Is this a reasonable way to perform this sort of action? Is there a simpler or more straight-forward method? Thank you in advance for the replies.
google-app-engine google-cloud-platform
asked Nov 21 '18 at 17:27
littlebenlittlelittlebenlittle
518
I'm building an application that will periodically pull data from several APIs and write them to cloud storage for later processing by Dataflow. There are many different ways to do this so I wanted to sanity check before I jumped in.
My plan is this:
- For each API, Cloud Scheduler will hit an endpoint for an App Engine app
- The app will create a Compute Engine VM instance with a startup script that pulls the data from the API and writes it to storage
- When done, the VM will hit another endpoint on the App Engine app that shuts down the VM.
Is this a reasonable way to perform this sort of action? Is there a simpler or more straight-forward method? Thank you in advance for the replies.
google-app-engine google-cloud-platform
google-app-engine google-cloud-platform
asked Nov 21 '18 at 17:27
littlebenlittlelittlebenlittle
518
asked Nov 21 '18 at 17:27
littlebenlittlelittlebenlittle
518
asked Nov 21 '18 at 17:27
littlebenlittlelittlebenlittle
518
asked Nov 21 '18 at 17:27
littlebenlittlelittlebenlittle
518
asked Nov 21 '18 at 17:27
littlebenlittlelittlebenlittle
518
518
1
I'm not sure if it will cover all your needs but Cloud Scheduler can schedule Compute Engine without App Engine however it seems that you cannot create and delete the VM with this method.
– oakinlaja
Nov 21 '18 at 19:04
This seems like a more standard way of doing things. Thank you for the suggestion!
– littlebenlittle
Nov 22 '18 at 1:23
add a comment |
1
I'm not sure if it will cover all your needs but Cloud Scheduler can schedule Compute Engine without App Engine however it seems that you cannot create and delete the VM with this method.
– oakinlaja
Nov 21 '18 at 19:04
This seems like a more standard way of doing things. Thank you for the suggestion!
– littlebenlittle
Nov 22 '18 at 1:23
1
1
I'm not sure if it will cover all your needs but Cloud Scheduler can schedule Compute Engine without App Engine however it seems that you cannot create and delete the VM with this method.
– oakinlaja
Nov 21 '18 at 19:04
I'm not sure if it will cover all your needs but Cloud Scheduler can schedule Compute Engine without App Engine however it seems that you cannot create and delete the VM with this method.
– oakinlaja
Nov 21 '18 at 19:04
This seems like a more standard way of doing things. Thank you for the suggestion!
– littlebenlittle
Nov 22 '18 at 1:23
This seems like a more standard way of doing things. Thank you for the suggestion!
– littlebenlittle
Nov 22 '18 at 1:23
add a comment |
2 Answers
2
active
oldest
votes
Cloud Scheduler can schedule Compute Engine without App Engine however it seems that you cannot create and delete the VM with this method.
answered Nov 22 '18 at 19:10
oakinlajaoakinlaja
1816
add a comment |
You can just use App Engine cron jobs to schedule the tasks. Your App Engine app cron handler can simply run the script that pulls data from the APIs. Maybe I am missing something, why do you need to use a Compute Engine instance to run the script?
answered Nov 22 '18 at 0:26
Ying LiYing Li
2,8162936
My thinking was that having a huge chunk of code in app engine is a less "cloud-y" way of solving the problem. Also VMs are cheaper than the app engine--particularly for this use-case where micro vm instances can do the job for essentially free (you get 720 hours a month which is way more than I need).
– littlebenlittle
Nov 22 '18 at 1:31
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53417592%2fin-gcp-is-there-a-canonical-way-to-scrape-data-from-an-api%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Cloud Scheduler can schedule Compute Engine without App Engine however it seems that you cannot create and delete the VM with this method.
answered Nov 22 '18 at 19:10
oakinlajaoakinlaja
1816
add a comment |
Cloud Scheduler can schedule Compute Engine without App Engine however it seems that you cannot create and delete the VM with this method.
answered Nov 22 '18 at 19:10
oakinlajaoakinlaja
1816
add a comment |
Cloud Scheduler can schedule Compute Engine without App Engine however it seems that you cannot create and delete the VM with this method.
answered Nov 22 '18 at 19:10
oakinlajaoakinlaja
1816
Cloud Scheduler can schedule Compute Engine without App Engine however it seems that you cannot create and delete the VM with this method.
answered Nov 22 '18 at 19:10
oakinlajaoakinlaja
1816
answered Nov 22 '18 at 19:10
oakinlajaoakinlaja
1816
answered Nov 22 '18 at 19:10
oakinlajaoakinlaja
1816
answered Nov 22 '18 at 19:10
oakinlajaoakinlaja
1816
1816
add a comment |
add a comment |
You can just use App Engine cron jobs to schedule the tasks. Your App Engine app cron handler can simply run the script that pulls data from the APIs. Maybe I am missing something, why do you need to use a Compute Engine instance to run the script?
answered Nov 22 '18 at 0:26
Ying LiYing Li
2,8162936
My thinking was that having a huge chunk of code in app engine is a less "cloud-y" way of solving the problem. Also VMs are cheaper than the app engine--particularly for this use-case where micro vm instances can do the job for essentially free (you get 720 hours a month which is way more than I need).
– littlebenlittle
Nov 22 '18 at 1:31
add a comment |
You can just use App Engine cron jobs to schedule the tasks. Your App Engine app cron handler can simply run the script that pulls data from the APIs. Maybe I am missing something, why do you need to use a Compute Engine instance to run the script?
answered Nov 22 '18 at 0:26
Ying LiYing Li
2,8162936
My thinking was that having a huge chunk of code in app engine is a less "cloud-y" way of solving the problem. Also VMs are cheaper than the app engine--particularly for this use-case where micro vm instances can do the job for essentially free (you get 720 hours a month which is way more than I need).
– littlebenlittle
Nov 22 '18 at 1:31
add a comment |
You can just use App Engine cron jobs to schedule the tasks. Your App Engine app cron handler can simply run the script that pulls data from the APIs. Maybe I am missing something, why do you need to use a Compute Engine instance to run the script?
answered Nov 22 '18 at 0:26
Ying LiYing Li
2,8162936
You can just use App Engine cron jobs to schedule the tasks. Your App Engine app cron handler can simply run the script that pulls data from the APIs. Maybe I am missing something, why do you need to use a Compute Engine instance to run the script?
answered Nov 22 '18 at 0:26
Ying LiYing Li
2,8162936
answered Nov 22 '18 at 0:26
Ying LiYing Li
2,8162936
answered Nov 22 '18 at 0:26
Ying LiYing Li
2,8162936
answered Nov 22 '18 at 0:26
Ying LiYing Li
2,8162936
2,8162936
My thinking was that having a huge chunk of code in app engine is a less "cloud-y" way of solving the problem. Also VMs are cheaper than the app engine--particularly for this use-case where micro vm instances can do the job for essentially free (you get 720 hours a month which is way more than I need).
– littlebenlittle
Nov 22 '18 at 1:31
add a comment |
My thinking was that having a huge chunk of code in app engine is a less "cloud-y" way of solving the problem. Also VMs are cheaper than the app engine--particularly for this use-case where micro vm instances can do the job for essentially free (you get 720 hours a month which is way more than I need).
– littlebenlittle
Nov 22 '18 at 1:31
My thinking was that having a huge chunk of code in app engine is a less "cloud-y" way of solving the problem. Also VMs are cheaper than the app engine--particularly for this use-case where micro vm instances can do the job for essentially free (you get 720 hours a month which is way more than I need).
– littlebenlittle
Nov 22 '18 at 1:31
My thinking was that having a huge chunk of code in app engine is a less "cloud-y" way of solving the problem. Also VMs are cheaper than the app engine--particularly for this use-case where micro vm instances can do the job for essentially free (you get 720 hours a month which is way more than I need).
– littlebenlittle
Nov 22 '18 at 1:31
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53417592%2fin-gcp-is-there-a-canonical-way-to-scrape-data-from-an-api%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
I'm not sure if it will cover all your needs but Cloud Scheduler can schedule Compute Engine without App Engine however it seems that you cannot create and delete the VM with this method.
– oakinlaja
Nov 21 '18 at 19:04
This seems like a more standard way of doing things. Thank you for the suggestion!
– littlebenlittle
Nov 22 '18 at 1:23