The transformation between the SV set and non-SV set is analyzed during the process of incremental SVM learning. Considering the initial non-SV set and new samples which will influence the accuracy of classification, it improves the KKT rule and error-driven rule. With these rules the new error-driven incremental SVM learning algorithm based on KKT conditions is proposed. With this algorithm, the useful information of original sample can be preserved as much as possible, the useless information of new samples can be removed accurately without affecting the processing speed. Experimental results show that this new algorithm has a good effect on both optimizing classifier and improving classification performance.