Couchbase Disk Usage
I'm using Couchbase, with .net - Latest CouchbaseNetClient.
Couchbase Serer 4.6.2 Enterprise running.
I have an app where It performs 5k/ops.
The problem I am facing - is as soon as I start this app - then the CPU usage jumps to 100% and Disk Usage jumps to 100% also (as noted on the ctrl+alt+del ->Performance->Disk 0 (C:)
I have tested across a few PC.s - All are SSD (Samsung 850 PRO) and a dell M.2. Both have excellent r/w capabilities which I would have thought would swallow 5k ops.
The machine with the 850 PRO is a 6c/12t 980x CPU.
The machine with the m2.dell is a 4c/8t i76700hq.
I'm using a standard Upsert as per couchbase doco's. When I comment out this line, CPU goes right down to expected levels.
Is there some form of tweaking/performance/indexing mechanism I am missing entirely here??
Regards,
Chud
EDIT #1
So I uninstalled Couchbase on the Dell XPS and reinstalled it as memcached server only. Funny thing is, the CPU goes to 100% and now disk stays at 0% (expected of course). I have commented out the lines that actually do the Upsert, and CPU usage remains virtually unchanged to what it was before I started the app, even though the app itself is still doing a fair bit of crunching, it's just not entering the data into the (now) memcached database.
How can the upsert be so so so expensive? It's only a small Json file...
EDIT #2
So here is some samples that may get you going. I have changed the models around a some what to keep confidentiality in what I am doing, so it is "untested" so to speak. So if there are bugs in this I apologise, I literally just hacked it together.
Json Sample Below - I guess put this in a loop and let it rain, see if you can produce the same load.
{"Position":10,"Type":0,"Type2":5,"Value":0.0,"Type3":1,"Location":"Bla","Type":"Ex","Id":"11111"}
public class RootObject
{
public int Position { get; set; }
public MyFirstType Type { get; set; }
public MySecondType Type2 { get; set; }
public double Value { get; set; }
public MyThirdType Type3 { get; set; }
public string Location { get; set; }
public string Id { get; set; }
}
public enum MyFirstType
{
A,
B
}
public enum MySecondType
{
C,
D,
E,
F,
G,
H
}
public enum MyThirdType
{
I,
J
}
public class CouchBaseInstanceTest
{
private Cluster CouchClient;
private IBucket Bucket;
public CouchBaseInstanceTest()
{
CouchClient = new Cluster(new ClientConfiguration
{
Servers = new List<Uri> {new Uri("http://localhost:8091")}
});
// I think CB v 5.0 and over, or 5.5 needs auth against the bucket
//CouchClient.Authenticate("admin", "password");
Bucket = CouchClient.OpenBucket("default");
}
public bool Upsert(string key, dynamic value, double expiry)
{
var Document = new Document<dynamic>()
{
Id = key,
Content = value,
Expiry = TimeSpan.FromHours(expiry).ToTtl()
};
return Bucket.Upsert(Document).Success;
}
}
So as soon as that Bucket.Upsert is hit - the CPU/Disk maxes out.... or in just the memcached config....the CPU maxes out....
To be honest though, the code here is really irrelevant I think. When I replaced the document values with a blank string "" - The problem was still the same
c# .net couchbase
add a comment |
I'm using Couchbase, with .net - Latest CouchbaseNetClient.
Couchbase Serer 4.6.2 Enterprise running.
I have an app where It performs 5k/ops.
The problem I am facing - is as soon as I start this app - then the CPU usage jumps to 100% and Disk Usage jumps to 100% also (as noted on the ctrl+alt+del ->Performance->Disk 0 (C:)
I have tested across a few PC.s - All are SSD (Samsung 850 PRO) and a dell M.2. Both have excellent r/w capabilities which I would have thought would swallow 5k ops.
The machine with the 850 PRO is a 6c/12t 980x CPU.
The machine with the m2.dell is a 4c/8t i76700hq.
I'm using a standard Upsert as per couchbase doco's. When I comment out this line, CPU goes right down to expected levels.
Is there some form of tweaking/performance/indexing mechanism I am missing entirely here??
Regards,
Chud
EDIT #1
So I uninstalled Couchbase on the Dell XPS and reinstalled it as memcached server only. Funny thing is, the CPU goes to 100% and now disk stays at 0% (expected of course). I have commented out the lines that actually do the Upsert, and CPU usage remains virtually unchanged to what it was before I started the app, even though the app itself is still doing a fair bit of crunching, it's just not entering the data into the (now) memcached database.
How can the upsert be so so so expensive? It's only a small Json file...
EDIT #2
So here is some samples that may get you going. I have changed the models around a some what to keep confidentiality in what I am doing, so it is "untested" so to speak. So if there are bugs in this I apologise, I literally just hacked it together.
Json Sample Below - I guess put this in a loop and let it rain, see if you can produce the same load.
{"Position":10,"Type":0,"Type2":5,"Value":0.0,"Type3":1,"Location":"Bla","Type":"Ex","Id":"11111"}
public class RootObject
{
public int Position { get; set; }
public MyFirstType Type { get; set; }
public MySecondType Type2 { get; set; }
public double Value { get; set; }
public MyThirdType Type3 { get; set; }
public string Location { get; set; }
public string Id { get; set; }
}
public enum MyFirstType
{
A,
B
}
public enum MySecondType
{
C,
D,
E,
F,
G,
H
}
public enum MyThirdType
{
I,
J
}
public class CouchBaseInstanceTest
{
private Cluster CouchClient;
private IBucket Bucket;
public CouchBaseInstanceTest()
{
CouchClient = new Cluster(new ClientConfiguration
{
Servers = new List<Uri> {new Uri("http://localhost:8091")}
});
// I think CB v 5.0 and over, or 5.5 needs auth against the bucket
//CouchClient.Authenticate("admin", "password");
Bucket = CouchClient.OpenBucket("default");
}
public bool Upsert(string key, dynamic value, double expiry)
{
var Document = new Document<dynamic>()
{
Id = key,
Content = value,
Expiry = TimeSpan.FromHours(expiry).ToTtl()
};
return Bucket.Upsert(Document).Success;
}
}
So as soon as that Bucket.Upsert is hit - the CPU/Disk maxes out.... or in just the memcached config....the CPU maxes out....
To be honest though, the code here is really irrelevant I think. When I replaced the document values with a blank string "" - The problem was still the same
c# .net couchbase
Sounds like you should ask this on github or where ever this is located.
– TheGeneral
Nov 19 '18 at 6:28
Could you post the minimum code to recreate this? It's possible that you're creating too many connections or something like that.
– Matthew Groves
Nov 20 '18 at 15:01
Hi Matthew, I actually use a DI Container to establish the connection. This is only done once on program startup, and then the only thing called is an Upsert Function. I have even tried to use the Upsert(Dictionary<string,dynamic> "bulk" type function, but it makes no difference.
– The_Chud
Nov 20 '18 at 22:31
add a comment |
I'm using Couchbase, with .net - Latest CouchbaseNetClient.
Couchbase Serer 4.6.2 Enterprise running.
I have an app where It performs 5k/ops.
The problem I am facing - is as soon as I start this app - then the CPU usage jumps to 100% and Disk Usage jumps to 100% also (as noted on the ctrl+alt+del ->Performance->Disk 0 (C:)
I have tested across a few PC.s - All are SSD (Samsung 850 PRO) and a dell M.2. Both have excellent r/w capabilities which I would have thought would swallow 5k ops.
The machine with the 850 PRO is a 6c/12t 980x CPU.
The machine with the m2.dell is a 4c/8t i76700hq.
I'm using a standard Upsert as per couchbase doco's. When I comment out this line, CPU goes right down to expected levels.
Is there some form of tweaking/performance/indexing mechanism I am missing entirely here??
Regards,
Chud
EDIT #1
So I uninstalled Couchbase on the Dell XPS and reinstalled it as memcached server only. Funny thing is, the CPU goes to 100% and now disk stays at 0% (expected of course). I have commented out the lines that actually do the Upsert, and CPU usage remains virtually unchanged to what it was before I started the app, even though the app itself is still doing a fair bit of crunching, it's just not entering the data into the (now) memcached database.
How can the upsert be so so so expensive? It's only a small Json file...
EDIT #2
So here is some samples that may get you going. I have changed the models around a some what to keep confidentiality in what I am doing, so it is "untested" so to speak. So if there are bugs in this I apologise, I literally just hacked it together.
Json Sample Below - I guess put this in a loop and let it rain, see if you can produce the same load.
{"Position":10,"Type":0,"Type2":5,"Value":0.0,"Type3":1,"Location":"Bla","Type":"Ex","Id":"11111"}
public class RootObject
{
public int Position { get; set; }
public MyFirstType Type { get; set; }
public MySecondType Type2 { get; set; }
public double Value { get; set; }
public MyThirdType Type3 { get; set; }
public string Location { get; set; }
public string Id { get; set; }
}
public enum MyFirstType
{
A,
B
}
public enum MySecondType
{
C,
D,
E,
F,
G,
H
}
public enum MyThirdType
{
I,
J
}
public class CouchBaseInstanceTest
{
private Cluster CouchClient;
private IBucket Bucket;
public CouchBaseInstanceTest()
{
CouchClient = new Cluster(new ClientConfiguration
{
Servers = new List<Uri> {new Uri("http://localhost:8091")}
});
// I think CB v 5.0 and over, or 5.5 needs auth against the bucket
//CouchClient.Authenticate("admin", "password");
Bucket = CouchClient.OpenBucket("default");
}
public bool Upsert(string key, dynamic value, double expiry)
{
var Document = new Document<dynamic>()
{
Id = key,
Content = value,
Expiry = TimeSpan.FromHours(expiry).ToTtl()
};
return Bucket.Upsert(Document).Success;
}
}
So as soon as that Bucket.Upsert is hit - the CPU/Disk maxes out.... or in just the memcached config....the CPU maxes out....
To be honest though, the code here is really irrelevant I think. When I replaced the document values with a blank string "" - The problem was still the same
c# .net couchbase
I'm using Couchbase, with .net - Latest CouchbaseNetClient.
Couchbase Serer 4.6.2 Enterprise running.
I have an app where It performs 5k/ops.
The problem I am facing - is as soon as I start this app - then the CPU usage jumps to 100% and Disk Usage jumps to 100% also (as noted on the ctrl+alt+del ->Performance->Disk 0 (C:)
I have tested across a few PC.s - All are SSD (Samsung 850 PRO) and a dell M.2. Both have excellent r/w capabilities which I would have thought would swallow 5k ops.
The machine with the 850 PRO is a 6c/12t 980x CPU.
The machine with the m2.dell is a 4c/8t i76700hq.
I'm using a standard Upsert as per couchbase doco's. When I comment out this line, CPU goes right down to expected levels.
Is there some form of tweaking/performance/indexing mechanism I am missing entirely here??
Regards,
Chud
EDIT #1
So I uninstalled Couchbase on the Dell XPS and reinstalled it as memcached server only. Funny thing is, the CPU goes to 100% and now disk stays at 0% (expected of course). I have commented out the lines that actually do the Upsert, and CPU usage remains virtually unchanged to what it was before I started the app, even though the app itself is still doing a fair bit of crunching, it's just not entering the data into the (now) memcached database.
How can the upsert be so so so expensive? It's only a small Json file...
EDIT #2
So here is some samples that may get you going. I have changed the models around a some what to keep confidentiality in what I am doing, so it is "untested" so to speak. So if there are bugs in this I apologise, I literally just hacked it together.
Json Sample Below - I guess put this in a loop and let it rain, see if you can produce the same load.
{"Position":10,"Type":0,"Type2":5,"Value":0.0,"Type3":1,"Location":"Bla","Type":"Ex","Id":"11111"}
public class RootObject
{
public int Position { get; set; }
public MyFirstType Type { get; set; }
public MySecondType Type2 { get; set; }
public double Value { get; set; }
public MyThirdType Type3 { get; set; }
public string Location { get; set; }
public string Id { get; set; }
}
public enum MyFirstType
{
A,
B
}
public enum MySecondType
{
C,
D,
E,
F,
G,
H
}
public enum MyThirdType
{
I,
J
}
public class CouchBaseInstanceTest
{
private Cluster CouchClient;
private IBucket Bucket;
public CouchBaseInstanceTest()
{
CouchClient = new Cluster(new ClientConfiguration
{
Servers = new List<Uri> {new Uri("http://localhost:8091")}
});
// I think CB v 5.0 and over, or 5.5 needs auth against the bucket
//CouchClient.Authenticate("admin", "password");
Bucket = CouchClient.OpenBucket("default");
}
public bool Upsert(string key, dynamic value, double expiry)
{
var Document = new Document<dynamic>()
{
Id = key,
Content = value,
Expiry = TimeSpan.FromHours(expiry).ToTtl()
};
return Bucket.Upsert(Document).Success;
}
}
So as soon as that Bucket.Upsert is hit - the CPU/Disk maxes out.... or in just the memcached config....the CPU maxes out....
To be honest though, the code here is really irrelevant I think. When I replaced the document values with a blank string "" - The problem was still the same
c# .net couchbase
c# .net couchbase
edited Nov 20 '18 at 23:17
The_Chud
asked Nov 19 '18 at 6:17
The_ChudThe_Chud
195110
195110
Sounds like you should ask this on github or where ever this is located.
– TheGeneral
Nov 19 '18 at 6:28
Could you post the minimum code to recreate this? It's possible that you're creating too many connections or something like that.
– Matthew Groves
Nov 20 '18 at 15:01
Hi Matthew, I actually use a DI Container to establish the connection. This is only done once on program startup, and then the only thing called is an Upsert Function. I have even tried to use the Upsert(Dictionary<string,dynamic> "bulk" type function, but it makes no difference.
– The_Chud
Nov 20 '18 at 22:31
add a comment |
Sounds like you should ask this on github or where ever this is located.
– TheGeneral
Nov 19 '18 at 6:28
Could you post the minimum code to recreate this? It's possible that you're creating too many connections or something like that.
– Matthew Groves
Nov 20 '18 at 15:01
Hi Matthew, I actually use a DI Container to establish the connection. This is only done once on program startup, and then the only thing called is an Upsert Function. I have even tried to use the Upsert(Dictionary<string,dynamic> "bulk" type function, but it makes no difference.
– The_Chud
Nov 20 '18 at 22:31
Sounds like you should ask this on github or where ever this is located.
– TheGeneral
Nov 19 '18 at 6:28
Sounds like you should ask this on github or where ever this is located.
– TheGeneral
Nov 19 '18 at 6:28
Could you post the minimum code to recreate this? It's possible that you're creating too many connections or something like that.
– Matthew Groves
Nov 20 '18 at 15:01
Could you post the minimum code to recreate this? It's possible that you're creating too many connections or something like that.
– Matthew Groves
Nov 20 '18 at 15:01
Hi Matthew, I actually use a DI Container to establish the connection. This is only done once on program startup, and then the only thing called is an Upsert Function. I have even tried to use the Upsert(Dictionary<string,dynamic> "bulk" type function, but it makes no difference.
– The_Chud
Nov 20 '18 at 22:31
Hi Matthew, I actually use a DI Container to establish the connection. This is only done once on program startup, and then the only thing called is an Upsert Function. I have even tried to use the Upsert(Dictionary<string,dynamic> "bulk" type function, but it makes no difference.
– The_Chud
Nov 20 '18 at 22:31
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53369249%2fcouchbase-disk-usage%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53369249%2fcouchbase-disk-usage%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Sounds like you should ask this on github or where ever this is located.
– TheGeneral
Nov 19 '18 at 6:28
Could you post the minimum code to recreate this? It's possible that you're creating too many connections or something like that.
– Matthew Groves
Nov 20 '18 at 15:01
Hi Matthew, I actually use a DI Container to establish the connection. This is only done once on program startup, and then the only thing called is an Upsert Function. I have even tried to use the Upsert(Dictionary<string,dynamic> "bulk" type function, but it makes no difference.
– The_Chud
Nov 20 '18 at 22:31