Multi-robot collaborative navigation is currently widely used in search and rescue, logistics, and other fields. Cooperative strategy and target navigation are the main challenges faced by multi-robot collaborative navigation. To improve the cooperative navigation ability of multiple mobile robots in an unknown environment, this study proposes a new hierarchical control cooperative navigation (HCCN) strategy. The high-level target decision layer and low-level target navigation layer are applied to assign a target point to each robot, and the global path planning and local path planning algorithms are adopted to guide the agent to reach the assigned target point without collision. Experimental verification is carried out on the Gazebo platform. The results show that the proposed method can effectively solve the sparse reward problem in cooperative navigation, and the training speed can be improved by at least 16.6%. It has better robustness in different scenarios. It is expected to provide theoretical guidance for further research on multi-robot cooperative navigation and be applied to more real scenarios.