vocab size versus vector size in word2vec
I have a data with 6200 sentences(which are triplets of form "sign_or_symptoms diagnoses Pathologic_function"), however the unique words(vocabulary) in these sentence is 181, what would be the appropriate vector size to train a model on the sentences with such low vocabulary. Is there any resource or research on appropriate vector size depending on vocabulary size?
word2vec word-embedding
asked Nov 20 '18 at 5:44
Arshad ShaikArshad Shaik
54
add a comment |
I have a data with 6200 sentences(which are triplets of form "sign_or_symptoms diagnoses Pathologic_function"), however the unique words(vocabulary) in these sentence is 181, what would be the appropriate vector size to train a model on the sentences with such low vocabulary. Is there any resource or research on appropriate vector size depending on vocabulary size?
word2vec word-embedding
asked Nov 20 '18 at 5:44
Arshad ShaikArshad Shaik
54
add a comment |
I have a data with 6200 sentences(which are triplets of form "sign_or_symptoms diagnoses Pathologic_function"), however the unique words(vocabulary) in these sentence is 181, what would be the appropriate vector size to train a model on the sentences with such low vocabulary. Is there any resource or research on appropriate vector size depending on vocabulary size?
word2vec word-embedding
asked Nov 20 '18 at 5:44
Arshad ShaikArshad Shaik
54
I have a data with 6200 sentences(which are triplets of form "sign_or_symptoms diagnoses Pathologic_function"), however the unique words(vocabulary) in these sentence is 181, what would be the appropriate vector size to train a model on the sentences with such low vocabulary. Is there any resource or research on appropriate vector size depending on vocabulary size?
word2vec word-embedding
word2vec word-embedding
asked Nov 20 '18 at 5:44
Arshad ShaikArshad Shaik
54
asked Nov 20 '18 at 5:44
Arshad ShaikArshad Shaik
54
edited Nov 21 '18 at 21:22
Arshad Shaik
asked Nov 20 '18 at 5:44
Arshad ShaikArshad Shaik
54
asked Nov 20 '18 at 5:44
Arshad ShaikArshad Shaik
54
asked Nov 20 '18 at 5:44
Arshad ShaikArshad Shaik
54
54
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
The best practice is to test it against your true end-task.
That's an incredibly small corpus and vocabulary-size for word2vec. It might not be appropriate at all, as it gets its power from large, varied training sets.
But on the bright side, you can run lots of trials with different parameters very quickly!
You absolutely can't use a vector dimensionality as large as your vocabulary (181), or even really very close. In such a case, the model is certain to 'overfit' – just memorizing the effects of each word in isolation, with none of the necessary trading-off 'tug-of-war', forcing words to be nearer/farther to each other, that creates the special value/generality of word2vec models.
My very loose rule-of-thumb would be to investigate dimensionalities around the square-root of the vocabulary size. And, multiples-of-4 tend to work best in the underlying array routines (at least when performance is critical, which it might not be with such a tiny data set). So I'd try 12 or 16 dimensions first, and then explore other lower/higher values based on some quantitative quality evaluation on your real task.
But again, you're working with a dataset so tiny, unless your 'sentences' are actually really long, word2vec may be a very weak technique for you without more data.
answered Nov 20 '18 at 21:58
gojomogojomo
19.6k64467
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53386911%2fvocab-size-versus-vector-size-in-word2vec%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The best practice is to test it against your true end-task.
That's an incredibly small corpus and vocabulary-size for word2vec. It might not be appropriate at all, as it gets its power from large, varied training sets.
But on the bright side, you can run lots of trials with different parameters very quickly!
You absolutely can't use a vector dimensionality as large as your vocabulary (181), or even really very close. In such a case, the model is certain to 'overfit' – just memorizing the effects of each word in isolation, with none of the necessary trading-off 'tug-of-war', forcing words to be nearer/farther to each other, that creates the special value/generality of word2vec models.
My very loose rule-of-thumb would be to investigate dimensionalities around the square-root of the vocabulary size. And, multiples-of-4 tend to work best in the underlying array routines (at least when performance is critical, which it might not be with such a tiny data set). So I'd try 12 or 16 dimensions first, and then explore other lower/higher values based on some quantitative quality evaluation on your real task.
But again, you're working with a dataset so tiny, unless your 'sentences' are actually really long, word2vec may be a very weak technique for you without more data.
answered Nov 20 '18 at 21:58
gojomogojomo
19.6k64467
add a comment |
The best practice is to test it against your true end-task.
That's an incredibly small corpus and vocabulary-size for word2vec. It might not be appropriate at all, as it gets its power from large, varied training sets.
But on the bright side, you can run lots of trials with different parameters very quickly!
You absolutely can't use a vector dimensionality as large as your vocabulary (181), or even really very close. In such a case, the model is certain to 'overfit' – just memorizing the effects of each word in isolation, with none of the necessary trading-off 'tug-of-war', forcing words to be nearer/farther to each other, that creates the special value/generality of word2vec models.
My very loose rule-of-thumb would be to investigate dimensionalities around the square-root of the vocabulary size. And, multiples-of-4 tend to work best in the underlying array routines (at least when performance is critical, which it might not be with such a tiny data set). So I'd try 12 or 16 dimensions first, and then explore other lower/higher values based on some quantitative quality evaluation on your real task.
But again, you're working with a dataset so tiny, unless your 'sentences' are actually really long, word2vec may be a very weak technique for you without more data.
answered Nov 20 '18 at 21:58
gojomogojomo
19.6k64467
add a comment |
The best practice is to test it against your true end-task.
That's an incredibly small corpus and vocabulary-size for word2vec. It might not be appropriate at all, as it gets its power from large, varied training sets.
But on the bright side, you can run lots of trials with different parameters very quickly!
You absolutely can't use a vector dimensionality as large as your vocabulary (181), or even really very close. In such a case, the model is certain to 'overfit' – just memorizing the effects of each word in isolation, with none of the necessary trading-off 'tug-of-war', forcing words to be nearer/farther to each other, that creates the special value/generality of word2vec models.
My very loose rule-of-thumb would be to investigate dimensionalities around the square-root of the vocabulary size. And, multiples-of-4 tend to work best in the underlying array routines (at least when performance is critical, which it might not be with such a tiny data set). So I'd try 12 or 16 dimensions first, and then explore other lower/higher values based on some quantitative quality evaluation on your real task.
But again, you're working with a dataset so tiny, unless your 'sentences' are actually really long, word2vec may be a very weak technique for you without more data.
answered Nov 20 '18 at 21:58
gojomogojomo
19.6k64467
The best practice is to test it against your true end-task.
That's an incredibly small corpus and vocabulary-size for word2vec. It might not be appropriate at all, as it gets its power from large, varied training sets.
But on the bright side, you can run lots of trials with different parameters very quickly!
You absolutely can't use a vector dimensionality as large as your vocabulary (181), or even really very close. In such a case, the model is certain to 'overfit' – just memorizing the effects of each word in isolation, with none of the necessary trading-off 'tug-of-war', forcing words to be nearer/farther to each other, that creates the special value/generality of word2vec models.
My very loose rule-of-thumb would be to investigate dimensionalities around the square-root of the vocabulary size. And, multiples-of-4 tend to work best in the underlying array routines (at least when performance is critical, which it might not be with such a tiny data set). So I'd try 12 or 16 dimensions first, and then explore other lower/higher values based on some quantitative quality evaluation on your real task.
But again, you're working with a dataset so tiny, unless your 'sentences' are actually really long, word2vec may be a very weak technique for you without more data.
answered Nov 20 '18 at 21:58
gojomogojomo
19.6k64467
answered Nov 20 '18 at 21:58
gojomogojomo
19.6k64467
answered Nov 20 '18 at 21:58
gojomogojomo
19.6k64467
answered Nov 20 '18 at 21:58
gojomogojomo
19.6k64467
19.6k64467
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53386911%2fvocab-size-versus-vector-size-in-word2vec%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
