The smart grid (SG) constitutes a technological evolution of the traditional grid by introducing information and communication technology (ICT) services. Although using of ICT has advantages, it poses some serious challenges to security. In this study, we propose a security architecture of the smart grid intrusion detection system and a novel intrusion detection system (IDS) based on unsupervised learning. We design block-training architecture which can not only reduce the computing burden in the data center but also train the characteristics of local traffic. We also propose a variational autoencoder based on recursive feature elimination with cross-validation (RFECV-VAE). The RFECV-VAE is a combination of RFECV (for feature selection) and VAE model (for anomaly detection) and can detect large-scale and high-dimensional data with high accuracy. Finally, we choose deep autoencoder (DAE), deep autoencoding Gaussian mixture model (DAGMM), one-class support vector machine (OCSVM), isolation forest (IF), and VAE as comparison algorithms and accuracy, ROC_AUC, F1_score, and training duration for performance evaluation. The experimental results show that RFECV-VAE outperforms the comparison algorithms.