Signature size for RSA 3072 and 4096 bits is too small

[mitresthen]

When signing a PDF, the library calculates how much space to reserve for the signature based on Certificate.SignatureAlgorithm. I believe this is the algorithm the CA used to sign the certificate, not the algorithm or key size of the certificate itself.
This means if you have an RSA-4096 key, but your certificate was signed by a CA using RSA-2048, the library only reserves space for a 256-byte signature when RSA-4096 actually produces 512-byte signatures.
The signature ends up being too large for the reserved space, triggering the retry mechanism. The retry mechanism itself also has bugs that can corrupt the PDF.
I discovered this when trying to use a 3072 bit certificate for signing, and ended up with corrupt pdf's. Then after changing the signature space allocation function and rebuilding pdfsign, it worked fine.

This bug can be reproduced with the below test:

func TestSignPDFWithRSA3072Key(t *testing.T) {
	// Generate a 3072-bit RSA key
	privateKey, err := rsa.GenerateKey(rand.Reader, 3072)
	if err != nil {
		t.Fatalf("failed to generate RSA-3072 key: %v", err)
	}

	// Create a self-signed certificate with SHA256-RSA signature algorithm
	// This is the key part: the certificate is signed with SHA256-RSA,
	// but the public key is RSA-3072 which produces 384-byte signatures
	template := &x509.Certificate{
		SerialNumber: big.NewInt(1),
		Subject: pkix.Name{
			CommonName:   "RSA-3072 Test Certificate",
			Organization: []string{"Test"},
		},
		NotBefore:             time.Now(),
		NotAfter:              time.Now().Add(365 * 24 * time.Hour),
		KeyUsage:              x509.KeyUsageDigitalSignature,
		ExtKeyUsage:           []x509.ExtKeyUsage{x509.ExtKeyUsageAny},
		BasicConstraintsValid: true,
	}

	certDER, err := x509.CreateCertificate(rand.Reader, template, template, &privateKey.PublicKey, privateKey)
	if err != nil {
		t.Fatalf("failed to create certificate: %v", err)
	}

	cert, err := x509.ParseCertificate(certDER)
	if err != nil {
		t.Fatalf("failed to parse certificate: %v", err)
	}

	// Verify our test setup: certificate should be signed with SHA256-RSA
	// but have an RSA-3072 public key
	if cert.SignatureAlgorithm != x509.SHA256WithRSA {
		t.Fatalf("expected SHA256WithRSA signature algorithm, got %v", cert.SignatureAlgorithm)
	}
	if privateKey.Size() != 384 { // 3072 bits = 384 bytes
		t.Fatalf("expected 384-byte key size, got %d", privateKey.Size())
	}

	// Now try to sign a PDF with this certificate
	inputFilePath := "../testfiles/testfile20.pdf"
	tmpfile, err := os.CreateTemp("", "rsa3072_test_*.pdf")
	if err != nil {
		t.Fatalf("failed to create temp file: %v", err)
	}
	defer os.Remove(tmpfile.Name())

	err = SignFile(inputFilePath, tmpfile.Name(), SignData{
		Signature: SignDataSignature{
			Info: SignDataSignatureInfo{
				Name:        "RSA-3072 Test",
				Location:    "Test Location",
				Reason:      "Testing RSA-3072 signature buffer size",
				ContactInfo: "test@example.com",
				Date:        time.Now().Local(),
			},
			CertType:   CertificationSignature,
			DocMDPPerm: AllowFillingExistingFormFieldsAndSignaturesPerms,
		},
		Signer:             privateKey,
		DigestAlgorithm:    crypto.SHA256,
		Certificate:        cert,
		RevocationData:     revocation.InfoArchival{},
		RevocationFunction: DefaultEmbedRevocationStatusFunction,
	})
	if err != nil {
		t.Fatalf("failed to sign PDF with RSA-3072 key: %v", err)
	}

	// Verify the signed PDF
	verifySignedFile(t, tmpfile, "rsa3072_test.pdf")
}```

[vanbroup]

Thanks for reporting this! I’ll test it against the refactored version using the new API, which I plan to push as a pre-release in the next few days.

[mitresthen]

Great, thank you! Currently I'm just using my fork that i made the PR from as a workaround, but would be very nice to use the official version.