Quantum and thermal phase transitions in a bosonic atom-molecule mixture in a two-dimensional optical lattice

We study the ground state and the thermal phase diagram of a two-species Bose-Hubbard model, with U(1) x Z(2) symmetry, describing atoms and molecules on a two-dimensional optical lattice interacting via a Feshbach resonance. Using quantum Monte Carlo simulations andmean-field theory, we show that the conversion between the two species, coherently coupling the atomic and molecular states, has a crucial impact on the Mott-superfluid transition and stabilizes an insulating phase with a gap controlled by the conversion term-the Feshbach insulator-instead of a standard Mott-insulating phase. Depending on the detuning between atoms and molecules, this model exhibits three phases: the Feshbach insulator, a molecular condensate coexisting with noncondensed atoms, and a mixed atomic-molecular condensate. Employing finite-size scaling analysis, we observe three-dimensional (3D) XY (3D Ising) transition when U(1) (Z(2)) symmetry is broken, whereas the transition is first order when both U(1) and Z(2) symmetries are spontaneously broken. The finite-temperature phase diagram is also discussed. The thermal disappearance of the molecular superfluid leads to a BerezinskiiKosterlitz- Thouless transition with unusual universal jump in the superfluid density. The loss of the quasilong- range coherence of the mixed atomic and molecular superfluid is more subtle since only atoms exhibit conventional Berezinskii-Kosterlitz-Thouless criticality. We also observe a signal compatible with a classical first-order transition between the mixed superfluid and the normal Bose liquid at low temperature.