Clustering measurement/ clustering error

[mkcedward]

Clustering measurement/ clustering error

[nicodv]

Basically, after training there is a .cost_ attribute on the clusterer available for this. This is compatible with how scikit-learn does it, and I'd like to stick to that.

This is documented in the KPrototypes class: https://github.com/nicodv/kmodes/blob/master/kmodes/kprototypes.py#L365

For finding optimal number of clusters, simply run the algorithm in a loop with varying k.

[mkcedward]

Refer to finding an optimal number of clusters. The definition of ".cost_" is "sum distance of all points to their respective cluster centroids". I am thinking how can I find the optimal number of cluster base on ".cost_"

For example (an extreme case for easier explanation ), I have 10 data points and want to know whether I should cluster it to 2 clusters or 10 clusters.
For 2 Clusters. Total distance is 5 for Cluster A while it is 3 for Cluster B. So the best cost is 3. For 10 Clusters. As I only have 10 data points, the cost of all centroid should be 0? So the optimal number of cluster is 10

[nicodv]

Cost is defined as the sum of dissimilarities of all points with their closest centroids, not as a cost per cluster. (You have to look at the total picture instead of cluster-by-cluster, because if you leave 1 cluster out, the cost of the other changes because the data points would need to get re-assigned.)

In you example, you would get cost=8 in the first scenario, cost=0 in the second. Simply re-run the algorithm across a range of k and see what minimizes cost.