Deep hybrid manifold for image set classification

The exponential growth of the data volume of image sets, which contain more information than a single image, has attracted increasing attention from researchers. Image set data are often described as covariance matrices or linear subspaces, and the unique geometries they span are symmetric positive definite (SPD) manifolds and Grassmann manifolds, respectively. Image set data are often described as covariance matrices or linear subspaces, and the distinctive geometries they span are symmetric positive definite (SPD) manifold and Grassmann manifold, respectively. However, most studies focus on a single manifold and ignore the useful information of the another manifold. Based on this, we propose a new Deep Hybrid Manifold Network (DHMNet).The DHMNet consists of backbone network, stackable Hybrid Manifold AutoEncoder (HMAE) and,Maximum Fusion Module (MFM). The image set data is modeled through SPD manifold and Grassmann manifold. The modeled data is input into the backbone network composed of SPDNet and GrNet for initial feature extraction, and the output manifold data are input into HMAEs. The HMAE effectively extracts and hybridizes complementary information from different manifolds and has the ability to generate deep representations with rich structural semantic information. For the three image datasets used, DHMNet with two HMAEs improves the classification accuracy by 3.83–5.76% over the classical SPDNet, and even reaches the best when compared to other models, with the best performance on the First Person Hand Action (FPHA) dataset for skeleton-based hand action recognition.