Outsourcing software development to the community developers is a promising model to help reduce software development cost and improve development efficiency. In this paper, we present a method to evaluate the quality of service in the managing such community-based software outsourcing process. In the community-based software outsourcing service, a customer (e.g., a software company) firstly releases the requirement and design specifications of a software system to the community, then the community helps to decompose the whole development tasks into a set of fine-grained tasks (including programming, designing test cases, testing, etc) and allocate them to community developers (programmers, testers, project managers, etc). These service providers work to fulfill the tasks and submit results to the community. In this service, quality is quite important and it is necessary to evaluate the quality of both final submitted software entities and various development activities, to ensure that all the initial requirements have been completely and correctly accomplished. In our quality evaluation method, there are three types of objects whose service quality need to be evaluated, i.e., products, behaviors and people. Specifically speaking, they are the deliverables (codes, testcases, test records) submitted by each service provider, the development process, and various community developers, respectively. For each type of the objects, we designed five dimensions of quality indicators, i.e. time and efficiency, price and cost, quality of service content, resources and conditions, reputation and risk. A set of refined quality indicators is designed for each of the five dimensions. Aiming at each quality indicator, we put forward the corresponding measurement method, i.e., quantitatively calculating the value of each quality indicator based on the original data automatically collected from the community platform and some subjective evaluation opinions from customers. Then, traditional AHP method is adopted to calculate the total quality of each service object (products, behavior and people) by accumulating all the quality indicators together. A prototype is developed to support above evaluation process and exhibit the results of quality evaluation. Results of the quality evaluation will help to: (1) monitor the execution of community-based software outsourcing service and to obtain up-to-date quality information; (2) find quality deficiencies timely and take remedial measures against them; (3) guide better selection of community developers ased on their historical quality records.