Evidence for a Phase Transition in the Quantum Spin Liquid State of a Kitaev Candidate alpha-RuCl3

The Kitaev quantum spin liquid (QSL) on the two-dimensional honeycomb lattice epitomizes an entangled topological state, where the spins fractionalize into Majorana fermions. This state has aroused tremendous interest because it harbors non-Abelian anyons. The half-integer quantized thermal Hall (HIQTH) conductance observed in alpha-RuCl3 is a key signature of the non-Abelian topological order. However, the fate of this topologically nontrivial state at intense fields remains largely elusive. Here, we report the thermal conductivity kappa and specific heat C of alpha-RuCl3 with in-plane magnetic fields H. For the field direction perpendicular to the Ru-Ru bond, where the HIQTH effect is observed, we find clear anomalies in kappa(H) and C(H) at mu H-0* approximate to 11 T, indicating the emergence of a phase transition between low- and high-field spin liquid phases. We point out that this phase transition is likely to be weak first-order. We also find that even well above H*, kappa(H) is highly anisotropic with respect to H-direction. This indicates that the high field phase is not in a simple spin-polarized state and spin-fractionalization may be retained. Intriguingly, the thermal Hall conductance of the same crystal deviates from the half-integer plateau near H*, suggesting that the topological properties are affected by this transition.