Guided, Deep Testing of X.509 Certificate Validation via Coverage Transfer Graphs

SSL and TLS are two secure protocols for creating secure connections over the Internet. X.509 certificate validation is important for security and needs to be performed before an SSL/TLS connection is established. However, state-of-the-art testing techniques, such as frankencert and mucert, have revealed, through randomly mutating Internet accessible certificates, that there exist unexpected, sometimes critical, validation differences among different SSL/TLS implementations. Despite these strong efforts, certificate validation is still not thoroughly tested and more effective techniques are needed as this work shows.To this end, this paper introduces transcert, a novel approach for effectively guiding fuzzing to perform deep testing of X.509 certificate validation. The goal of transcert is to generate certificates that trigger diverse executions; it achieves this goal by introducing the concept of a coverage transfer graph to efficiently, precisely abstract program executions. In particular, it records the execution of how a given certificate is validated by a reference SSL/TLS implementation. It then constructs a coverage transfer graph to model the coverage transfer from a test certificate (seed) to its mutated certificates (mutants), and explores the coverage transfer graph by iteratively sampling and mutating certificates.We have implemented transcert and evaluated it against frankencert and mucert on four state-of-the-art SSL/TLS implementations. The evaluation results clearly show the strengths of transcert-during 10,000 iterations, transcert has revealed 3,469 validation differences, 8× as many as those revealed by frankencert and mucert. We have identified 11 root causes of validation differences, all of which have been confirmed and five have never been reported previously. We also found that the primary goal of certificate chain validation is stated ambiguously in the widely-adopted PKI standard RFC 5280.