For the shortage of current serial device server which almost lacks security and functionality, based on arm9 and Linux, we have designed a secure serial device server using SSL protocol and multi-user management. The overall design of hardware and software were introduced. Including U-boot, kernel, root file system and a variety of tools, embedded Linux system has been emigrated, we discussed the framework of the software system, focusing on data process framework, user management and the implementation of SSL. The application shows that this device runs stably and meets the needs of secure communication.