Hierarchical Clustering of Sub-Populations with a dissimilarity based on the likelihood ratio statis

  1. Ciampi, A. 1
  2. Lechevallier, Y. 3
  3. Limas, M.C. 2
  4. Marcos, A.G. 2
  1. 1 McGill University
    info

    McGill University

    Montreal, Canadá

    GRID grid.14709.3b

  2. 2 Universidad de León
    info

    Universidad de León

    León, España

    GRID grid.4807.b

  3. 3 French Institute for Research in Computer Science and Automation
    info

    French Institute for Research in Computer Science and Automation

    Le Chesnay, Francia

    GRID grid.5328.c

Journal:
Pattern Analysis and Applications

ISSN: 1433-7541

Year of publication: 2008

Volume: 11

Issue: 2

Pages: 199-220

Type: Article

Export: RIS
DOI: 10.1007/s10044-007-0088-4 SCOPUS: 2-s2.0-44449157741 GOOGLE SCHOLAR

Abstract

The problem of clustering subpopulations on the basis of samples is considered within a statistical framework: a distribution for the variables is assumed for each subpopulation and the dissimilarity between any two populations is defined as the likelihood ratio statistic which compares the hypothesis that the two subpopulations differ in the parameter of their distributions to the hypothesis that they do not. A general algorithm for the construction of a hierarchical classification is described which has the important property of not having inversions in the dendrogram. The essential elements of the algorithm are specified for the case of well-known distributions (normal, multinomial and Poisson) and an outline of the general parametric case is also discussed. Several applications are discussed, the main one being a novel approach to dealing with massive data in the context of a two-step approach. After clustering the data in a reasonable number of 'bins' by a fast algorithm such as k-Means, we apply a version of our algorithm to the resulting bins. Multivariate normality for the means calculated on each bin is assumed: this is justified by the central limit theorem and the assumption that each bin contains a large number of units, an assumption generally justified when dealing with truly massive data such as currently found in modern data analysis. However, no assumption is made about the data generating distribution. © 2007 Springer-Verlag London Limited.