Because of the best-effort service of the Internet, the VoIP system has QoS problem. In order to improve QoS, most systems use application-layer routing mechanism. One of the key issues of the mechanism is how to search for relay node. Combining the P2P technology, this paper proposes a topology-aware based P2P relay search mechanism. First, in order to simulate the real Internet topology, a Cluster Overlay network was built. In the Cluster Overlay network, a topology-aware relay search algorithm was designed. Finally, this program was achieved in the SIP terminal. A simulation on the Cluster Overlay network and relay search mechanism was carried out on the NS2 platform. The results show that the similarity between the Cluster Overlay network and the real Internet topology is very high. The relay search algorithm can find the optimal relay node, thus to improve the quality of the path.