Modulated flexural edge waves in a plate with its free edge structured by an array of grooves

•The coupled mode theory in diffraction optics and acoustics is extended to solve dispersion curves of FEWs.•High order flexural waveguide modes are revealed and adopted for a plate waveguide.•The first-order (symmetric mode) and second-order (antisymmetric mode) modulated FEWs are analyzed.•Rainbow trapping and mode conversion (to the bulk wave) of the modulated FEWs are demonstrated by FE simulations and experiments.