Scikit-Learn DBSCAN clustering yielding no clusters
up vote
-1
down vote
favorite
I have a data set with a dozen dimensions (columns) and about 200 observations (rows). This dataset has been normalized using quantile_transform_normalize. (Edit: I tried running the clustering without normalization, but still no luck, so I don't believe this is the cause.) Now I want to cluster the data into several clusters. Until now I had been using KMeans, but I have read that it may not be accurate in higher dimensions and doesn't handle outliers well, so I wanted to compare to DBSCAN to see if I get a different result.
However, when I try to cluster the data with DBSCAN using the Mahalanobis distance metric, every item is clustered into -1. According to the documentation:
Noisy samples are given the label -1.
I'm not really sure what this means, but I was getting some OK clusters with KMeans so I know there is something there to cluster -- it's not just random.
Here is the code I am using for clustering:
covariance = np.cov(data.values.astype("float32"), rowvar=False)
clusterer = sklearn.cluster.DBSCAN(min_samples=6, metric="mahalanobis", metric_params={"V": covariance})
clusterer.fit(data)
And that's all. I know for certain that data is a numeric Pandas DataFrame as I have inspected it in the debugger.
What could be causing this issue?
python machine-learning scikit-learn cluster-analysis dbscan
asked Nov 12 at 17:24
Ian
1,81631944
add a comment |
up vote
-1
down vote
favorite
I have a data set with a dozen dimensions (columns) and about 200 observations (rows). This dataset has been normalized using quantile_transform_normalize. (Edit: I tried running the clustering without normalization, but still no luck, so I don't believe this is the cause.) Now I want to cluster the data into several clusters. Until now I had been using KMeans, but I have read that it may not be accurate in higher dimensions and doesn't handle outliers well, so I wanted to compare to DBSCAN to see if I get a different result.
However, when I try to cluster the data with DBSCAN using the Mahalanobis distance metric, every item is clustered into -1. According to the documentation:
Noisy samples are given the label -1.
I'm not really sure what this means, but I was getting some OK clusters with KMeans so I know there is something there to cluster -- it's not just random.
Here is the code I am using for clustering:
covariance = np.cov(data.values.astype("float32"), rowvar=False)
clusterer = sklearn.cluster.DBSCAN(min_samples=6, metric="mahalanobis", metric_params={"V": covariance})
clusterer.fit(data)
And that's all. I know for certain that data is a numeric Pandas DataFrame as I have inspected it in the debugger.
What could be causing this issue?
python machine-learning scikit-learn cluster-analysis dbscan
asked Nov 12 at 17:24
Ian
1,81631944
Maybe your data is just not "dense" enough for DBScan. Have you tried to use another metric and to adjust the min_samples value?
– Digital-Thinking
Nov 12 at 23:23
add a comment |
up vote
-1
down vote
favorite
up vote
-1
down vote
favorite
I have a data set with a dozen dimensions (columns) and about 200 observations (rows). This dataset has been normalized using quantile_transform_normalize. (Edit: I tried running the clustering without normalization, but still no luck, so I don't believe this is the cause.) Now I want to cluster the data into several clusters. Until now I had been using KMeans, but I have read that it may not be accurate in higher dimensions and doesn't handle outliers well, so I wanted to compare to DBSCAN to see if I get a different result.
However, when I try to cluster the data with DBSCAN using the Mahalanobis distance metric, every item is clustered into -1. According to the documentation:
Noisy samples are given the label -1.
I'm not really sure what this means, but I was getting some OK clusters with KMeans so I know there is something there to cluster -- it's not just random.
Here is the code I am using for clustering:
covariance = np.cov(data.values.astype("float32"), rowvar=False)
clusterer = sklearn.cluster.DBSCAN(min_samples=6, metric="mahalanobis", metric_params={"V": covariance})
clusterer.fit(data)
And that's all. I know for certain that data is a numeric Pandas DataFrame as I have inspected it in the debugger.
What could be causing this issue?
python machine-learning scikit-learn cluster-analysis dbscan
asked Nov 12 at 17:24
Ian
1,81631944
I have a data set with a dozen dimensions (columns) and about 200 observations (rows). This dataset has been normalized using quantile_transform_normalize. (Edit: I tried running the clustering without normalization, but still no luck, so I don't believe this is the cause.) Now I want to cluster the data into several clusters. Until now I had been using KMeans, but I have read that it may not be accurate in higher dimensions and doesn't handle outliers well, so I wanted to compare to DBSCAN to see if I get a different result.
However, when I try to cluster the data with DBSCAN using the Mahalanobis distance metric, every item is clustered into -1. According to the documentation:
Noisy samples are given the label -1.
I'm not really sure what this means, but I was getting some OK clusters with KMeans so I know there is something there to cluster -- it's not just random.
Here is the code I am using for clustering:
covariance = np.cov(data.values.astype("float32"), rowvar=False)
clusterer = sklearn.cluster.DBSCAN(min_samples=6, metric="mahalanobis", metric_params={"V": covariance})
clusterer.fit(data)
And that's all. I know for certain that data is a numeric Pandas DataFrame as I have inspected it in the debugger.
What could be causing this issue?
python machine-learning scikit-learn cluster-analysis dbscan
python machine-learning scikit-learn cluster-analysis dbscan
asked Nov 12 at 17:24
Ian
1,81631944
asked Nov 12 at 17:24
Ian
1,81631944
edited Nov 12 at 17:33
asked Nov 12 at 17:24
Ian
1,81631944
asked Nov 12 at 17:24
Ian
1,81631944
asked Nov 12 at 17:24
Ian
1,81631944
1,81631944
Maybe your data is just not "dense" enough for DBScan. Have you tried to use another metric and to adjust the min_samples value?
– Digital-Thinking
Nov 12 at 23:23
add a comment |
Maybe your data is just not "dense" enough for DBScan. Have you tried to use another metric and to adjust the min_samples value?
– Digital-Thinking
Nov 12 at 23:23
Maybe your data is just not "dense" enough for DBScan. Have you tried to use another metric and to adjust the min_samples value?
– Digital-Thinking
Nov 12 at 23:23
Maybe your data is just not "dense" enough for DBScan. Have you tried to use another metric and to adjust the min_samples value?
– Digital-Thinking
Nov 12 at 23:23
add a comment |
1 Answer
1
active
oldest
votes
up vote
1
down vote
accepted
You need to choose the parameter eps, too.
DBSCAN results depend on this parameter very much. You can find some methods for estimating it in literature.
IMHO, sklearn should not provide a default for this parameter, because it rarely ever works (on normalized toy data it is usually okay, but that's about it).
200 instances probably is too small to reliably measure density, in particular with a dozen variables.
answered Nov 13 at 8:50
Anony-Mousse
56.3k794158
How is the defaultepsdetermined? I was a little suspicious of it but I couldn't find any documentation, so I figured it must be an acceptable default.
– Ian
Nov 14 at 15:49
Someone at some point chose a default value. There is no data dependent process. Which is why it most often does not work. File a bug that you find the default value misleading.
– Anony-Mousse
2 days ago
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
accepted
You need to choose the parameter eps, too.
DBSCAN results depend on this parameter very much. You can find some methods for estimating it in literature.
IMHO, sklearn should not provide a default for this parameter, because it rarely ever works (on normalized toy data it is usually okay, but that's about it).
200 instances probably is too small to reliably measure density, in particular with a dozen variables.
answered Nov 13 at 8:50
Anony-Mousse
56.3k794158
How is the defaultepsdetermined? I was a little suspicious of it but I couldn't find any documentation, so I figured it must be an acceptable default.
– Ian
Nov 14 at 15:49
Someone at some point chose a default value. There is no data dependent process. Which is why it most often does not work. File a bug that you find the default value misleading.
– Anony-Mousse
2 days ago
add a comment |
up vote
1
down vote
accepted
You need to choose the parameter eps, too.
DBSCAN results depend on this parameter very much. You can find some methods for estimating it in literature.
IMHO, sklearn should not provide a default for this parameter, because it rarely ever works (on normalized toy data it is usually okay, but that's about it).
200 instances probably is too small to reliably measure density, in particular with a dozen variables.
answered Nov 13 at 8:50
Anony-Mousse
56.3k794158
How is the defaultepsdetermined? I was a little suspicious of it but I couldn't find any documentation, so I figured it must be an acceptable default.
– Ian
Nov 14 at 15:49
Someone at some point chose a default value. There is no data dependent process. Which is why it most often does not work. File a bug that you find the default value misleading.
– Anony-Mousse
2 days ago
add a comment |
up vote
1
down vote
accepted
up vote
1
down vote
accepted
You need to choose the parameter eps, too.
DBSCAN results depend on this parameter very much. You can find some methods for estimating it in literature.
IMHO, sklearn should not provide a default for this parameter, because it rarely ever works (on normalized toy data it is usually okay, but that's about it).
200 instances probably is too small to reliably measure density, in particular with a dozen variables.
answered Nov 13 at 8:50
Anony-Mousse
56.3k794158
You need to choose the parameter eps, too.
DBSCAN results depend on this parameter very much. You can find some methods for estimating it in literature.
IMHO, sklearn should not provide a default for this parameter, because it rarely ever works (on normalized toy data it is usually okay, but that's about it).
200 instances probably is too small to reliably measure density, in particular with a dozen variables.
answered Nov 13 at 8:50
Anony-Mousse
56.3k794158
answered Nov 13 at 8:50
Anony-Mousse
56.3k794158
answered Nov 13 at 8:50
Anony-Mousse
56.3k794158
answered Nov 13 at 8:50
Anony-Mousse
56.3k794158
56.3k794158
How is the defaultepsdetermined? I was a little suspicious of it but I couldn't find any documentation, so I figured it must be an acceptable default.
– Ian
Nov 14 at 15:49
Someone at some point chose a default value. There is no data dependent process. Which is why it most often does not work. File a bug that you find the default value misleading.
– Anony-Mousse
2 days ago
add a comment |
How is the defaultepsdetermined? I was a little suspicious of it but I couldn't find any documentation, so I figured it must be an acceptable default.
– Ian
Nov 14 at 15:49
Someone at some point chose a default value. There is no data dependent process. Which is why it most often does not work. File a bug that you find the default value misleading.
– Anony-Mousse
2 days ago
How is the default
eps determined? I was a little suspicious of it but I couldn't find any documentation, so I figured it must be an acceptable default.– Ian
Nov 14 at 15:49
How is the default
eps determined? I was a little suspicious of it but I couldn't find any documentation, so I figured it must be an acceptable default.– Ian
Nov 14 at 15:49
Someone at some point chose a default value. There is no data dependent process. Which is why it most often does not work. File a bug that you find the default value misleading.
– Anony-Mousse
2 days ago
Someone at some point chose a default value. There is no data dependent process. Which is why it most often does not work. File a bug that you find the default value misleading.
– Anony-Mousse
2 days ago
add a comment |
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53267148%2fscikit-learn-dbscan-clustering-yielding-no-clusters%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Maybe your data is just not "dense" enough for DBScan. Have you tried to use another metric and to adjust the min_samples value?
– Digital-Thinking
Nov 12 at 23:23