(a) The population of molecules ρ<sub><em>b</em></sub> versus the transition strength <em>J</em> for the ground and first excited states with different interactions

<p><strong>Figure 1.</strong> (a) The population of molecules ρ<sub><em>b</em></sub> versus the transition strength <em>J</em> for the ground and first excited states with different interactions. The parameter is δ<sub><em>b</em></sub> = −2<em>g</em>. (b) The mean-field phase diagram of the ground state in the parameter space of δ<sub><em>b</em></sub> and <em>J</em>. The pure molecule (PM) phase and the mixed atom–molecule (AM) phase are separated by a critical line (marked with the blue solid line).</p> <p><strong>Abstract</strong></p> <p>The quantum phase transition in an atom–molecule conversion system with atomic transition between two hyperfine states is studied. In the mean-field approximation, we give the phase diagram in which the phase boundary only depends on the atomic transition strength and the atom–molecule energy detuning, but not on the atomic interactions. Such a phase boundary is further confirmed by calculating the fidelity of the ground state and the energy gap between the first excited state and the ground state. As a comparison to the mean-field results, we also study the quantum phase transition with the full quantum method where the phase boundary depends on the particle number. Analysing the finite-size scaling behaviours of the energy gap, the fidelity susceptibility and the first-order derivative of entanglement entropy with respect to the atomic transition strength, we obtain their critical exponents by numerical calculation and show that in the thermodynamic limit, one can obtain the same phase boundary as in the mean-field approximation. Our results show a new way to manipulate the quantum phase transition by regulating the atomic transition strength with the intensity of the laser.</p>