Abstract:Detecting outliers is crucial for practical applications in large and high-dimensional datasets. Outlier detection is the process of identifying data points that deviate from the typical data distribution. This process primarily involves density estimation. Substantial advancements are achieved by models like the deep autoencoder Gaussian mixture model, which initially reduces dimensionality and subsequently estimates density. However, it introduces noise into the low-dimensional latent space and faces limitations in optimizing the density estimation module, such as the requirement to ensure positive definiteness of the covariance matrix. To overcome these constraints, this study introduces the deep autoencoder normalizing flow (DANF) for unsupervised outlier detection. The model employs deep autoencoders to produce low-dimensional latent space representations and reconstruction errors for individual input samples. These outputs are subsequently fed into a normalizing flow (NF) for transformation into a Gaussian distribution. Experimental results on several widely recognized benchmark datasets reveal that the DANF model consistently surpasses state-of-the-art outlier detection methods. The most notable improvement is a remarkable 26.43% increase in the F1-score evaluation metric.