Modulational Instability in the Ostrovsky Equation and Related Models

We study the modulational instability of small-amplitude periodic traveling wave solutions in a generalized Ostrovsky equation. Specifically, we investigate the invertibility of the associated linearized operator in the vicinity of the origin and derive a modulational instability index that depends on the dispersion and nonlinearity. We then show that the small-amplitude periodic traveling waves of the Ostrovsky equation exhibit modulational instability if the wavenumber is greater than a critical value which agrees with previously found numerical results in \cite{Whitfield2014Rotation-inducedWaves} both qualitatively and quantitatively. We also study the effects of surface tension on modulational instability using the index.