Vector Modulation Instability in Highly Birefringent Fibers With Circularly Polarized Eigenmodes

We report a detailed study of vector modulation instability (VMI) in highly birefringent fibers with circularly polarized modes in the normal dispersion regime. We show that because of suppression of coherent terms, the VMI in circularly birefringent fibers is governed by one set of coupled-mode nonlinear Schrödinger equations regardless of the fiber birefringence. In consequence, the VMI sidebands are polarized linearly and orthogonally to the pump up to the birefringence level of 10 ^-5 , similarly like in isotropic fibers. For greater birefringence the polarization states of the sidebands become elliptical with opposite handedness while the azimuth angle deviates from orthogonality to the pump. We also point on the dependence of the critical power beyond which the VMI cannot exist upon ellipticity angle θ of the eigenmodes. We show that the critical power gradually increases with the ellipticity angle and for θ > 17.6° the VMI gain is not limited, in contrast to linearly birefringent fibers. Our findings were confirmed experimentally by observation of the isotropic-like VMI in the spun side-hole fiber with nearly circularly polarized eigenmodes, in spite of relatively high birefringence of the order of 2 × 10 ^-6 .