This study summarizes the current research on semantic-based video retrieval to help future researchers understand the technologies available in this field, and video retrieval systems are created to find the video that users want to query in a large number of video data collections on the Internet or in databases. This study introduces and discusses the semantic-based video retrieval process and also summarizes the relevant techniques to solve the main problem of a semantic gap in this process. The semantic gap is induced by the difference between the low-level features extracted from video content and the user’s cognition of these features in the real world. It is a highly concerned research topic to transform the low-level features of video content into high-level semantic concepts.