With the rapid growth of Internet and Web service technology, the number of Web services having the same function is getting more and more. When constructing service-oriented applications, Quality of Service(QoS) as the non-functional properties of services is attracting more and more attention from users. To recommend high quality services to users, first we need to predict the quality of services. Now there are many research works on Web service QoS prediction, but most of these works focus on the optimization of modeling methods, which ignore the impact of auxiliary features. This paper emphatically analyzes auxiliary features' impact on QoS prediction such as service category and user location. To achieve this goal, in this paper designs and builds up a unified QoS prediction framework based on Factorization Machines, which can incorporate multiple auxiliary features easily and conveniently. Combined with service category and user location, this paper develops a QoS prediction method, which is proved to be advantageous via experiments on real-life data sets.