High-throughput X-ray diffraction (XRD) analysis plays a crucial role in accelerating material discovery. However, traditional methods often rely heavily on manual interpretation and tend to overlook low-intensity peak information when processing complex XRD data, thus limiting the potential for accuracy improvement. To address this issue, a large language model (LLM) agent framework for multi-modal crystal structure prediction is proposed. The framework integrates a GPT-4-driven intelligent agent with multi-modal voting models based on XRD and pair distribution functions, enabling autonomous crystal structure and space group prediction tasks. In addition, the reasoning capability of the LLM is enhanced through the introduction of knowledge graphs, which aid in understanding the relationships between crystal features, thus improving both prediction accuracy and reasoning performance. Experimental results demonstrate that the accuracy of this framework in crystal structure prediction and space group prediction tasks reaches 97.5% and 98.7%, respectively. This design significantly enhances the accuracy and efficiency of high-throughput analysis, with the potential to advance materials science research and provide valuable insights for addressing other highly interrelated multi-task problems.