How do I load a file from Cloud Storage into memory
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I have end users that are going to be uploading a csv file into a bucket which will then be loaded to BigQuery.
The issue is the content of the data is unreliable.
i.e. it contains fields with free text that may contain linefeeds,extra commas, invalid date formats e.t.c. e.t.c.
I have a python script that will pre-process the file and write out a new one with all errors corrected.
I need to be able to automate this into the cloud.
I was thinking I could load the contents of the file (it's only small) into memory and process the records then write it back out to the Bucket.
I do not want to process the file locally.
Despite extensive searching I can't find how to load a file in a bucket into memory and then write it back out again.
Can anyone help ?
python-3.x google-cloud-storage
asked Nov 23 '18 at 8:17
mez63mez63
4510
add a comment |
I have end users that are going to be uploading a csv file into a bucket which will then be loaded to BigQuery.
The issue is the content of the data is unreliable.
i.e. it contains fields with free text that may contain linefeeds,extra commas, invalid date formats e.t.c. e.t.c.
I have a python script that will pre-process the file and write out a new one with all errors corrected.
I need to be able to automate this into the cloud.
I was thinking I could load the contents of the file (it's only small) into memory and process the records then write it back out to the Bucket.
I do not want to process the file locally.
Despite extensive searching I can't find how to load a file in a bucket into memory and then write it back out again.
Can anyone help ?
python-3.x google-cloud-storage
asked Nov 23 '18 at 8:17
mez63mez63
4510
You can do this usngdownload_as_stringdetailed answer here: stackoverflow.com/a/48279267/6039925
– Gaurav Taneja
Nov 29 '18 at 8:23
add a comment |
I have end users that are going to be uploading a csv file into a bucket which will then be loaded to BigQuery.
The issue is the content of the data is unreliable.
i.e. it contains fields with free text that may contain linefeeds,extra commas, invalid date formats e.t.c. e.t.c.
I have a python script that will pre-process the file and write out a new one with all errors corrected.
I need to be able to automate this into the cloud.
I was thinking I could load the contents of the file (it's only small) into memory and process the records then write it back out to the Bucket.
I do not want to process the file locally.
Despite extensive searching I can't find how to load a file in a bucket into memory and then write it back out again.
Can anyone help ?
python-3.x google-cloud-storage
asked Nov 23 '18 at 8:17
mez63mez63
4510
I have end users that are going to be uploading a csv file into a bucket which will then be loaded to BigQuery.
The issue is the content of the data is unreliable.
i.e. it contains fields with free text that may contain linefeeds,extra commas, invalid date formats e.t.c. e.t.c.
I have a python script that will pre-process the file and write out a new one with all errors corrected.
I need to be able to automate this into the cloud.
I was thinking I could load the contents of the file (it's only small) into memory and process the records then write it back out to the Bucket.
I do not want to process the file locally.
Despite extensive searching I can't find how to load a file in a bucket into memory and then write it back out again.
Can anyone help ?
python-3.x google-cloud-storage
python-3.x google-cloud-storage
asked Nov 23 '18 at 8:17
mez63mez63
4510
asked Nov 23 '18 at 8:17
mez63mez63
4510
asked Nov 23 '18 at 8:17
mez63mez63
4510
asked Nov 23 '18 at 8:17
mez63mez63
4510
asked Nov 23 '18 at 8:17
mez63mez63
4510
4510
You can do this usngdownload_as_stringdetailed answer here: stackoverflow.com/a/48279267/6039925
– Gaurav Taneja
Nov 29 '18 at 8:23
add a comment |
You can do this usngdownload_as_stringdetailed answer here: stackoverflow.com/a/48279267/6039925
– Gaurav Taneja
Nov 29 '18 at 8:23
You can do this usng
download_as_string detailed answer here: stackoverflow.com/a/48279267/6039925– Gaurav Taneja
Nov 29 '18 at 8:23
You can do this usng
download_as_string detailed answer here: stackoverflow.com/a/48279267/6039925– Gaurav Taneja
Nov 29 '18 at 8:23
add a comment |
1 Answer
1
active
oldest
votes
I believe what you’re looking for is Google Cloud Functions. You can set a Cloud Function to be triggered by an upload to the GCS bucket, and use your Python code in the same Cloud Function to process the .csv and upload it to BigQuery, however, please bear in mind that Python 3.7.1 support for Cloud Functions is currently in a Beta state of development.
answered Nov 23 '18 at 11:38
MaximMaxim
1,573211
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53442893%2fhow-do-i-load-a-file-from-cloud-storage-into-memory%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I believe what you’re looking for is Google Cloud Functions. You can set a Cloud Function to be triggered by an upload to the GCS bucket, and use your Python code in the same Cloud Function to process the .csv and upload it to BigQuery, however, please bear in mind that Python 3.7.1 support for Cloud Functions is currently in a Beta state of development.
answered Nov 23 '18 at 11:38
MaximMaxim
1,573211
add a comment |
I believe what you’re looking for is Google Cloud Functions. You can set a Cloud Function to be triggered by an upload to the GCS bucket, and use your Python code in the same Cloud Function to process the .csv and upload it to BigQuery, however, please bear in mind that Python 3.7.1 support for Cloud Functions is currently in a Beta state of development.
answered Nov 23 '18 at 11:38
MaximMaxim
1,573211
add a comment |
I believe what you’re looking for is Google Cloud Functions. You can set a Cloud Function to be triggered by an upload to the GCS bucket, and use your Python code in the same Cloud Function to process the .csv and upload it to BigQuery, however, please bear in mind that Python 3.7.1 support for Cloud Functions is currently in a Beta state of development.
answered Nov 23 '18 at 11:38
MaximMaxim
1,573211
I believe what you’re looking for is Google Cloud Functions. You can set a Cloud Function to be triggered by an upload to the GCS bucket, and use your Python code in the same Cloud Function to process the .csv and upload it to BigQuery, however, please bear in mind that Python 3.7.1 support for Cloud Functions is currently in a Beta state of development.
answered Nov 23 '18 at 11:38
MaximMaxim
1,573211
edited Nov 25 '18 at 9:02
answered Nov 23 '18 at 11:38
MaximMaxim
1,573211
answered Nov 23 '18 at 11:38
MaximMaxim
1,573211
answered Nov 23 '18 at 11:38
MaximMaxim
1,573211
1,573211
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53442893%2fhow-do-i-load-a-file-from-cloud-storage-into-memory%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
You can do this usng
download_as_stringdetailed answer here: stackoverflow.com/a/48279267/6039925– Gaurav Taneja
Nov 29 '18 at 8:23