Magnetic properties and spin dynamics in a spin-orbit driven Jeff= 1/2 triangular lattice antiferromagnet

Frustration-induced strong quantum fluctuations accompanied by spin-orbit coupling and crystal electric field can give rise to rich and diverse magnetic phenomena associated with unconventional low-energy excitations in rare-earth based quantum magnets. Herein, we present crystal structure, magnetic susceptibility, specific heat, muon spin relaxation(muSR), and electron spin resonance (ESR) studies on the polycrystalline samples of Ba6Yb2Ti4O17 in which Yb3+ ions constitute a perfect triangular lattice in ab-plane without detectable anti-site disorder between atomic sites. The Curie-Weiss fit of low-temperature magnetic susceptibility data suggest the spin-orbit entangled Jeff = 1/2 degrees of freedom of Yb3+ spin with weak antiferromagnetic exchange interactions in the Kramers doublet ground state. The zero-field specific heat data reveal the presence of long-range magnetic order at TN = 77 mK which is suppressed in a magnetic field 1 T. The broad maximum in specific heat is attributed to the Schottky anomaly implying the Zeeman splitting of the Kramers doublet ground state. The ESR measurements suggest the presence of anisotropic exchange interaction between the moments of Yb3+ spins and the well separated Kramers doublet state. muSR experiments reveal a fluctuating state of Yb3+ spins in the temperature range 0.1 K-100 K owing to depopulation of crystal electric field levels, which suggests that the Kramers doublets are well separated consistent with thermodynamic and ESR results. In addition to the intraplane nearest-neighbor superexchange interaction, the interplane exchange interaction and anisotropy are expected to stabilize the long-range ordered state in this triangular lattice antiferromagnet.