In view of the low convergence efficiency and high communication overhead caused by dynamic heterogeneity among nodes and unreasonable communication resources in decentralized federated learning (DFL), this study proposes a hierarchical device-to-device (D2D) communication optimization scheme based on deep reinforcement learning. First, a hierarchical topology-driven federated learning framework is constructed. Specifically, the bottom-layer devices achieve local model aggregation via the Gossip protocol, while the top-layer representative nodes perform cross-region global collaboration, thus reducing communication frequency and mitigating the parameter inconsistency under dynamic networks. Second, a dynamic resource joint optimization algorithm based on deep deterministic policy gradient (DDPG) is designed to jointly optimize node collaboration strategies and spectrum resource allocation, adaptively balancing communication costs and model accuracy while suppressing the global model displacement caused by transmission disparities among devices. Experimental results on the MNIST and CIFAR-10 datasets demonstrate that the proposed scheme improves average model prediction accuracy by 2.61% and reduces communication energy consumption by 64.1%, compared to traditional decentralized federated learning methods.