The Gap Statistic: Intuitive Determination of Cluster Counts by Sofie Netteberg '20

The Gap Statistic: Intuitive Determination of Cluster Counts by Sofie Netteberg ’20, Wednesday, November 20, Statistics Colloquium, 1:10 – 1:50 pm, Stetson Court Classroom 105

Abstract:  Cluster analysis is a pivotal tool for “unsupervised” learning to finding groups within a dataset without the use of a formal response variable.  But once one has already decided how to cluster data by a measure of observation similarity, how do they determine what number of clusters is most informative? Statistical folklore has recommended the informal “elbow” method, but the gap statistic has the goal of formalizing a function for the most conservatively informative number of clusters.  The gap statistic is applicable to virtually any clustering method and this talk will explore its relative merits in interpretability with applications to real-world data.