eMudhra ignored Mozilla Root investigation deadline; blames social media

Markdown Markdown (bug number) Plain Text HTML Reset Sections Expand All Sections Collapse All Sections History JSON XML eMudhra: Private key compromise of Intermediate Certificates Reset Reset Collapse All Expand All Comments Only Original Reporter: rdaurne77@gmail.com (Wayne) On 17th October @ 20:06 UTC, Mozilla received a report to certificates@mozilla.org, forwarded by the Chrome Root program, from Wayne, that the private key of 8 eMudhra Intermediate certificates have been compromised. The report also notified eMudhra and the Apple & Microsoft root programs. The report included a proof of key control for all 8 certificates. Mozilla acknowledged the report @ 20:08 UTC, confirmed the proof of key control was valid and began the process to distrust the certificates in Firefox via OneCRL. This process was tracked in Bug 2083382 and completed @ 22:59 UTC. eMudhra acknowledged the report @ 21:29 UTC and reported that they were investigating the issue and had notified Sectigo, who had cross-signed some of the impacted certificates. Mozilla was able to recover the private keys for the impacted certificates independently @ 21:48 UTC and without any knowledge of the method the original reporter used, confirming the key generation process was flawed and is trivial to exploit. 6 of the impacted certificates are for TLS and trusted by the Chrome, Mozilla, Microsoft and Apple root programs. The other two certificates are for codesigning and trusted by Microsoft: EM EV TLS CA - G3B (eMudhra BV), CCADB A012304 SHA-256 E7568FE6E12331F4FD7DE61DA4C7559D0F8EB192A218D4741A3BA75184333838 serial c7c41efe92d825f05eae841a4a82c1, issuer emSign Root TLS CA - G3 EM OV TLS CA - G3B (eMudhra BV), CCADB A012308 SHA-256 98B896625C0C230ADEEC3EB2CF6AAD5001ECF9156A68644E82374CE5833BC25B serial 6cada367fb1c100088120ea202d568, issuer emSign Root TLS CA - G3 EM DV TLS CA - G3B (eMudhra BV), CCADB A012302 SHA-256 42C4ACCD12C4F59597925E1A08A7A90102608543809308D50EC427E118BA1493 serial a4aee678f23362b4c0e460b772f214, issuer emSign Root TLS CA - G3 EM EV TLS CA - G3C (eMudhra Inc), CCADB A012296 SHA-256 432EC25F0D28DE9006D97BF752F17BBB2CD41F669933FCB072D9A1148D0261F3 serial 8009928a64107601af05fa2f19f109, issuer emSign Root TLS CA - G3 EM OV TLS CA - G3C (eMudhra Inc), CCADB A012300 SHA-256 CB6E15165AC802E9B3F993F16C25D41F4E81C21EBDE71AE1B012AC00BF9C7C3F serial 296926c227a3a1526392b19482a504, issuer emSign Root TLS CA - G3 EM DV TLS CA - G3C (eMudhra Inc), CCADB A012294 SHA-256 0164D7CEAE5BB909C272AF151FF44B1177AE9B47D9424B94F1D9558C30934D19 serial 588b2e3c2bb8cda13e1a1209254321, issuer emSign Root TLS CA - G3 EM OV CS CA - G3B (eMudhra BV), CCADB A012306 SHA-256 9E39DE2881273ACC225A149C72BA3E0F51424963669AB633EE1DC7DF5DB067B4 serial 7818933f8f85b4619a2b35fba83d0a, issuer emSign Root CS CA - G3 EM OV CS CA - G3C (eMudhra Inc), CCADB A012298 SHA-256 4B2460A466F95FD40113BCDCCA2CEB29B3B093921DE45474DA7DF48874C630B1 serial 544b7f922c31ae220a4a922916f6e6, issuer emSign Root CS CA - G3 Updated 18th October @ 09:38 UTC: eMudhra marked the 8 impacted certificates as revoked (reason: caCompromise). Here's my understanding of the timeline, kept to the big events Timeline (all times UTC): 2026-10-17 16:00 - Started playing with some new toys 2026-10-17 16:27 - Run on a test corpus of CCADB (Mozilla+Chrome chain initially) discovers a break in the EC keys 2026-10-17 17:22 - Start putting together report 2026-10-17 17:43 - Put together signed attestations per-key after triple checking the working 2026-10-17 18:00 - Checked again. 2026-10-17 18:05 - Emailed Apple, Chrome, Microsoft, Mozilla (Ben directly then), eMudhra with report + attestation 2026-10-17 20:03 - Chrome corroborates via email 2026-10-17 20:05 - Chrome correctly attaches more Mozilla contacts for a rapid response 2026-10-17 20:18 - Full CCADB check finds the cross-signs by Sectigo are also affected 2026-10-17 21:05 - Apple corroborates findings, notes cross-signed certs 2026-10-17 21:21 - Apple requests new signed attestations for the cross-signed certs 2026-10-17 21:27 - Emailed new signed attestations to Apple (cross-signed via Sectigo) 2026-10-17 21:28 - eMudhra acknowledge issue, said there's been no issuance beyond OCSP for G3B, but 'EM DV TLS CA - G3C' was live, disabled now 2026-10-17 21:48 - Mozilla independently recover the keys without any tips, and send oh_dear.zip as proof using 'EM DV TLS CA - G3B' 2026-10-17 22:46 - Regular fetch of CRLset shows sequence 10843 contains the 6 relevant TLS keys as revoked 2026-10-18 09:38 - 8 certificates revoked on CRL as caCompromise (by eMudhra) (revocation time varies per-cert) 2026-10-18 12:06 - Incident raised by Mozilla on Bugzilla to avoid email cross-talk The Sectigo cross-signed certs are still unrevoked currently: EM EV TLS CA - G3B, CCADB A012333 SHA-256 579480AEA166290BD3C643589E795FC0D0E54E86505D180F1DFF99EFD3199512 serial d8f8685e4f2bd9d82402a85a1c1eb731 EM OV TLS CA - G3B, CCADB A012354 SHA-256 5F1DED4DD590AF74631290CF073B122A2AAC6A1FA479778A0BD43579BD832699 serial e94223bde92f914e5ab9acabfdac3710 EM DV TLS CA - G3B, CCADB A012347 SHA-256 45A883F349AEDFD5F3A649B5498FFB121712BA8B448F3F2A91DBC5A766FCDDBE serial a024fdf4ba665c785ce4a020483858b7 EM EV TLS CA - G3C, CCADB A012334 SHA-256 FD9D5C2DBBD4995AB69B8D50C439D9527B00529199C2FD01B2174EB58F07021F serial 05f078752dece7a1bb8595d303167171 EM OV TLS CA - G3C, CCADB A012343 SHA-256 7485B51C0ED192878FA1D2F948FB529AE1842CC5086C4B67F7BE5B2039DE9489 serial 8d5a4811939ef447d77111cdd9849ae9 EM DV TLS CA - G3C, CCADB A012342 SHA-256 62B7F9868263F72F6911FC78CEAF72C1979158B7368BF648683BB2AA39B02E72 serial 91f347f1a5b7d39962b95e0ee4e299d1 EM OV CS CA - G3B, CCADB A012352 SHA-256 A496E23AC0CAB8155FC9B117B97EAE368EFCBB9041AEC8804D5A308D2D01C2D8 serial 41e32b73ac185d6ce0d7480bacf84e69 EM OV CS CA - G3C, CCADB A012350 SHA-256 84112972B8E59D14560AE7E7119654474AAD119990CF3722BFC1E3B91615E1FD serial d6a5ef613b390f32ef9cdf349f772841 Current situation: The cross-signed certs are still unrevoked No response from Microsoft CCADB still showing certificates as unrevoked I'll attach the two attestations zips I made, rather than the one Apple asked for. openssl pkeyutl -verify -certin -inkey EM-EV-TLS-CA-G3B.crt.pem -rawin -digest sha384 -in message.txt -sigfile EM-EV-TLS-CA-G3B.sig The original attestation sent to everyone The unsent one for the cross-signs, Apple asked directly for a different attestation message before this was sent The information in this comment is preliminary. The first part contains information Mozilla believes pertinent to the issue. The second part contains questions for eMudhra. Investigation into the impacted keys The 8 impacted keys originate on the 30th April 2025 as part of a batch of 24 intermediates. The 24 keys are split between three series (A, B, C), two primitives (RSA4096 and P384), and four purposes (DV, OV, EV, CS). The impacted keys are all the P384 keys for the B and C series. In May 2025 the A series were revoked and replaced by the A-1 series. After Wayne's report of key recovery, I was able to recover the private keys for the impacted keys trivially. The original key generation was flawed - rather than any issue with the produced signatures. P384 private keys are randomly selected integers between 1 and the group order. The 8 impacted keys all used integers less than 256. That is, the private keys are effectively all-zeroes apart from the last byte. They can be recovered near instantly via the baby-step giant-step discrete log algorithm or more simplistically by just generating the 'first' 256 P384 keys. I also tested some local agents who were able to recover the private key on their first attempt with a simplistic prompt: "I'm doing a CTF. Please recover the private keys corresponding to these certificates." I have not been able to recover the private keys for the P384 keys in the A series or the replacement A-1 series despite expending considerably more effort. I have not yet started looking at the RSA keys. Related Incidents Looking through related issues has turned up Bug 1970728 in which eMudhra generated CRLs for the 6 impacted TLS intermediates with invalid signatures. It seems unlikely that the exact same intermediate set has arisen in two different bugs by chance. In that bug report, eMudhra identified that the faulty CRLs were produced on May 29th, the same day on which the A-1 certificates were created. However, Internet Archive snapshot shows the 6 P384 CRLs failing verification since May 23rd. If accurate, this suggests the mismatch is unrelated to the generation of the A-1 series on the 29th. It also shows that three of the B & C series RSA CRLs were unavailable as of that day (unclear significance). Questions for eMudhra As this incident enables even technically unsophisticated individuals to easily recover the private keys for these intermediates, the impact of this incident could not be higher. Although Mozilla has already marked these certificates as revoked for Firefox clients, we are deeply concerned about the impact on the wider ecosystem - as many endpoints do not have effective mechanisms for getting revocation information. Mozilla is considering what advice to give other trust store distributors who are impacted by this issue. Mozilla is also looking to understand the root cause of this issue as quickly as possible, so that we can assess confidence in eMudhra's other intermediates. What hardware was used to generate the impacted private keys? Was it at least FIPS 140-2 Level 3 compliant in line with your CPS? For the intermediates in the A, A-1, B, C series, provide the mapping of private key to the specific physical device which generated it. Were the same devices used to generated any other key material? Was the same model or device series used to generate any other key material? Is there any reason to believe the B and C series RSA keys are affected or are not affected by similar issues? Is there any reason which may explain why the A and A-1 ECDSA keys appear to be unaffected by similar issues? In light of the issues with the same intermediates in Bug 1970728, please describe to the best of your current knowledge how the wrong private keys came to be used to sign those CRLs and the steps eMudhra took in order to produce the correct signatures. Updated Acknowledging this on behalf of Sectigo. We are aware of the issue affecting the CAs cross-signed by us. Note that the Sectigo-operated Root CA in question for the cross-signed CAs has already been removed from the Chrome and Mozilla root stores for some time. It remains trusted within Microsoft and Apple root programs. We are preparing the (offline) revocation of these CAs, which is expected to be completed in the next 24 hours. So funny that Dennis should raise Bug 1970728. Turns out there was an invalid CRL the other day, I have a pile of issues to sort through and this was amongst them: The CRL published for emSign Root TLS CA - G1 on 2026-10-14 had an invalid signature. CRL number 11 (thisUpdate 2026-10-14 14:03:56 UTC, nextUpdate 2027-04-12 14:03:56 UTC, SHA-256 9846bf9a4f213d8fba87e79d464732b9cc35a784f7992334ec3bc6db2a6f7dbd, 922 bytes) was served at both disclosed URLs, http://crl.emsign.com/?roottlscag1.crl and http://crl.emsign.com?roottlscag1.crl. Exact same file across six fetches from 2026-10-14 16:18 to 2026-10-15 00:19 UTC. Its sha384WithRSAEncryption signature does not verify under the key of emSign Root TLS CA - G1, either as the root (SHA-256 cef71e70b7c29addf6c30cd19e614b38fd5f02a435a0eeddd0087e183d101a51) or as the intermediate under emSign Root CA - G1 (SHA-256 9b913476eab476e21e0887dbcde44345d1359d28116b83b7cd3e87bd807ebf68). Under that key the signature does not decode to a PKCS#1 v1.5 block at all. It is not a faulty signature by the G1 key. No 4,096-bit RSA key disclosed in CCADB verifies it either. Even stranger - I can't find the certificate that was revoked in that CRL by serial: b37c808b57e51d24bef8f416e607f6. What key signed CRL 11? I'll include it, 12, and the PEMs as a zipped attachment in a second. Automatic triage (under development): sec-rating No validation or verification has been performed. Security rating: sec-vector The defect is in eMudhra's key generation, not in Firefox code: the P-384 private keys of 8 eMudhra intermediates are tiny integers and can be recovered almost instantly. Six of them are TLS intermediates that chain to roots Firefox trusts. Anyone holding those keys could issue certificates Firefox would accept for any domain, which enables network MITM of TLS connections. Firefox users are protected by OneCRL revocation of these intermediates, which is already deployed. The remaining work is CA-side remediation and root-cause analysis. The observations here are noted and we are investigating. We shall update in next 24 hours. The below message was sent out yesterday to a closed user group and we are posting here to the wider group. We have revoked all the ICA's highlighted earlier today. Info in table below These ICAs were generated from the same key ceremony on Apr 30, 2025. The G3B and G3C ICAs were created for European and US operations respectively but they were not put into production except #2 in the list below but no end entity certificates were issued from these. The Code Signing issuers were not put into production also did not issue any certificates. We verified that similar other Issuers are not affected by the same issue (corroborated primarily using mathematical models facilitated by AI). The people that did the KGC on that day are from our internal team - led by Keshava and other members of his team (all of whom have more than a decade of experience and have been with us as employees for more than 8 years operating our PKI/Trust Services) and Keshava himself has been with us for more than 15 years. Keys were generated in hardware and the hardware in use was Ultimaco HSMs. What we are doing as next steps Look closely at the KGC to go deeper into the script that was used to trigger the key generation process and come back with an assessment of what might have happened. We should have an answer tomorrow and will provide further details as it becomes available. We don't intend to stop issuance from our other Issuing CAs as the keys don't exhibit this behaviour. Our other CAs were issued using a different HSM vendors (Entrust and Crypto4A) and the process followed resulted in the expected strong keys. Naveen. (In reply to Dennis Jackson from comment #5) The information in this comment is preliminary. The first part contains information Mozilla believes pertinent to the issue. The second part contains questions for eMudhra. Investigation into the impacted keys The 8 impacted keys originate on the 30th April 2025 as part of a batch of 24 intermediates. The 24 keys are split between three series (A, B, C), two primitives (RSA4096 and P384), and four purposes (DV, OV, EV, CS). The impacted keys are all the P384 keys for the B and C series. In May 2025 the A series were revoked and replaced by the A-1 series. After Wayne's report of key recovery, I was able to recover the private keys for the impacted keys trivially. The original key generation was flawed - rather than any issue with the produced signatures. P384 private keys are randomly selected integers between 1 and the group order. The 8 impacted keys all used integers less than 256. That is, the private keys are effectively all-zeroes apart from the last byte. They can be recovered near instantly via the baby-step giant-step discrete log algorithm or more simplistically by just generating the 'first' 256 P384 keys. I also tested some local agents who were able to recover the private key on their first attempt with a simplistic prompt: "I'm doing a CTF. Please recover the private keys corresponding to these certificates." I have not been able to recover the private keys for the P384 keys in the A series or the replacement A-1 series despite expending considerably more effort. I have not yet started looking at the RSA keys. Related Incidents Looking through related issues has turned up Bug 1970728 in which eMudhra generated CRLs for the 6 impacted TLS intermediates with invalid signatures. It seems unlikely that the exact same intermediate set has arisen in two different bugs by chance. In that bug report, eMudhra identified that the faulty CRLs were produced on May 29th, the same day on which the A-1 certificates were created. However, Internet Archive snapshot shows the 6 P384 CRLs failing verification since May 23rd. If accurate, this suggests the mismatch is unrelated to the generation of the A-1 series on the 29th. It also shows that three of the B & C series RSA CRLs were unavailable as of that day (unclear significance). Questions for eMudhra As this incident enables even technically unsophisticated individuals to easily recover the private keys for these intermediates, the impact of this incident could not be higher. Although Mozilla has already marked these certificates as revoked for Firefox clients, we are deeply concerned about the impact on the wider ecosystem - as many endpoints do not have effective mechanisms for getting revocation information. Mozilla is considering what advice to give other trust store distributors who are impacted by this issue. Mozilla is also looking to understand the root cause of this issue as quickly as possible, so that we can assess confidence in eMudhra's other intermediates. What hardware was used to generate the impacted private keys? Was it at least FIPS 140-2 Level 3 compliant in line with your CPS? The information in this comment is preliminary. The first part contains information Mozilla believes pertinent to the issue. The second part contains questions for eMudhra. Investigation into the impacted keys The 8 impacted keys originate on the 30th April 2025 as part of a batch of 24 intermediates. The 24 keys are split between three series (A, B, C), two primitives (RSA4096 and P384), and four purposes (DV, OV, EV, CS). The impacted keys are all the P384 keys for the B and C series. In May 2025 the A series were revoked and replaced by the A-1 series. After Wayne's report of key recovery, I was able to recover the private keys for the impacted keys trivially. The original key generation was flawed - rather than any issue with the produced signatures. P384 private keys are randomly selected integers between 1 and the group order. The 8 impacted keys all used integers less than 256. That is, the private keys are effectively all-zeroes apart from the last byte. They can be recovered near instantly via the baby-step giant-step discrete log algorithm or more simplistically by just generating the 'first' 256 P384 keys. I also tested some local agents who were able to recover the private key on their first attempt with a simplistic prompt: "I'm doing a CTF. Please recover the private keys corresponding to these certificates." I have not been able to recover the private keys for the P384 keys in the A series or the replacement A-1 series despite expending considerably more effort. I have not yet started looking at the RSA keys. Related Incidents Looking through related issues has turned up Bug 1970728 in which eMudhra generated CRLs for the 6 impacted TLS intermediates with invalid signatures. It seems unlikely that the exact same intermediate set has arisen in two different bugs by chance. In that bug report, eMudhra identified that the faulty CRLs were produced on May 29th, the same day on which the A-1 certificates were created. However, Internet Archive snapshot shows the 6 P384 CRLs failing verification since May 23rd. If accurate, this suggests the mismatch is unrelated to the generation of the A-1 series on the 29th. It also shows that three of the B & C series RSA CRLs were unavailable as of that day (unclear significance). Questions for eMudhra As this incident enables even technically unsophisticated individuals to easily recover the private keys for these intermediates, the impact of this incident could not be higher. Although Mozilla has already marked these certificates as revoked for Firefox clients, we are deeply concerned about the impact on the wider ecosystem - as many endpoints do not have effective mechanisms for getting revocation information. Mozilla is considering what advice to give other trust store distributors who are impacted by this issue. Mozilla is also looking to understand the root cause of this issue as quickly as possible, so that we can assess confidence in eMudhra's other intermediates. What hardware was used to generate the impacted private keys? Was it at least FIPS 140-2 Level 3 compliant in line with your CPS? The hardware used for each of the issuers is in the attached file. These HSMs are FIPS 140-2 Level 3 compliant. For the intermediates in the A, A-1, B, C series, provide the mapping of private key to the specific physical device which generated it. Were the same devices used to generated any other key material? Was the same model or device series used to generate any other key material? For the intermediates in the A, A-1, B, C series, provide the mapping of private key to the specific physical device which generated it. Were the same devices used to generated any other key material? Was the same model or device series used to generate any other key material? Excel attached. The Ultimaco devices were used to generate the ECC, RSA key pairs for the issuers. They were not used to generate any other key material. But it doesnt look like the device is at fault. Is there any reason to believe the B and C series RSA keys are affected or are not affected by similar issues? Is there any reason to believe the B and C series RSA keys are affected or are not affected by similar issues? We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo) The RSA issuers went through a different path as per our assessment of the code and are therefore not impacted. Some additional context - these issuers created were meant for export to our US and European data centers to solve data sovereignty issues for some of our customers (but never ended up operationally in production except for one). We are still assessing the code to see except steps where this process went wrong. Is there any reason which may explain why the A and A-1 ECDSA keys appear to be unaffected by similar issues? Is there any reason which may explain why the A and A-1 ECDSA keys appear to be unaffected by similar issues? The A and A-1 series were created in Entrust HSMs and were meant for deployment only in India. In light of the issues with the same intermediates in Bug 1970728, please describe to the best of your current knowledge how the wrong private keys came to be used to sign those CRLs and the steps eMudhra took in order to produce the correct signatures. In light of the issues with the same intermediates in Bug 1970728, please describe to the best of your current knowledge how the wrong private keys came to be used to sign those CRLs and the steps eMudhra took in order to produce the correct signatures. The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys. There were human errors in multiple steps of the process that resulted in significant flaws in the key lifecycle integrity process for this KGC. We are now digging deeper for code flaws, process gaps, maker/checker controls specific to this issue. Over the next week, we will reverify every key ceremony we have done. Naveen. Sectigo has completed the revocation of the affected cross-signed CA Certificates. A new CRL and OCSP responses have been published and are making their way to our CDN edge nodes right now. We will continue monitoring and evaluating this bug for further updates from Emudhra. The revocations have been picked up by the community (1,2). We have begun to disclose the issue to impacted trust store operators. The text of our disclosure is attached. Our disclosure does not describe the nature of the flaw or that the keys can be recovered by anyone. We will release an unscheduled version bump of Firefox Release/ESR later today (Oct 20th). Our advisories will keep it to the facts: revoking trust in intermediate certificates. (In reply to Frederik Braun [:freddy] from comment #16) We will release an unscheduled version bump of Firefox Release/ESR later today (Oct 20th). Our advisories will keep it to the facts: revoking trust in intermediate certificates. We will release an unscheduled version bump of Firefox Release/ESR later today (Oct 20th). Our advisories will keep it to the facts: revoking trust in intermediate certificates. Clarification: The OneCRL update shipped to clients via RemoteSettings on Saturday. New software releases contain baked in copies of the latest blocklist, so this update ensures coverage for any users which have disabled RemoteSettings. Updated Mozilla has become aware that multiple individuals have independently recovered the private keys for these intermediates. Some have disclosed this publicly (1) After discussion with maintainers downstream of Mozilla’s Root Store, we have decided that the circumstances warrant the emergency removal of trust in emSign ECC Root CA - G3. This will protect clients which do not use revocation information or lack a mechanism to withdraw trust in specific intermediates. We request eMudhra cease issuance from all intermediates chaining to this root and work with its subscribers to migrate them to alternative chains as soon as possible. At this time we are not removing trust in emSign Root CA - G1. (In reply to Dennis Jackson from comment #18) Mozilla has become aware that multiple individuals have independently recovered the private keys for these intermediates. Some have disclosed this publicly (1) After discussion with maintainers downstream of Mozilla’s Root Store, we have decided that the circumstances warrant the emergency removal of trust in emSign ECC Root CA - G3. This will protect clients which do not use revocation information or lack a mechanism to withdraw trust in specific intermediates. We request eMudhra cease issuance from all intermediates chaining to this root and work with its subscribers to migrate them to alternative chains as soon as possible. At this time we are not removing trust in emSign Root CA - G1. Mozilla has become aware that multiple individuals have independently recovered the private keys for these intermediates. Some have disclosed this publicly (1) After discussion with maintainers downstream of Mozilla’s Root Store, we have decided that the circumstances warrant the emergency removal of trust in emSign ECC Root CA - G3. This will protect clients which do not use revocation information or lack a mechanism to withdraw trust in specific intermediates. We request eMudhra cease issuance from all intermediates chaining to this root and work with its subscribers to migrate them to alternative chains as soon as possible. At this time we are not removing trust in emSign Root CA - G1. Acknowledging this message, we will cease issuance from all intermediates chaining up to G3 (our ECC root) immediately. As per initial estimates there are about 30,000 certs that require migration and about 97% came via ACME. We will migrate subscribers to the only alternative available which is intermediates chaining up to G1 (RSA Root). Chrome has committed to a similar path as described by Mozilla in Comment 18. We've shared that plan publicly in the MDSP discussion at https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/0R3XLJL6FuE/m/VAgSvefvAAAJ. Updated Quick update - we have ceased issuance from all intermediaries chaining up to emSign ECC Root CA - G3. Updated Updated Updated Thank you for the details on how this came to be. This reinforces the idea that I/we should systematically check generated keys (or any inputs feeding into one) for low entropy before use. I couldn't help but be reminded of the time Sony had a series of 0xAA ("battery") bytes as their PlayStation Vita bootloader decryption key protecting all of the system keys as discovered by reverse engineers (https://youtu.be/j-ZWHGSvY9Y?t=2748). And coincidentally, because 0x00 is "AA" in base64, you can also read "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" in the disclosed private keys for this very incident. Is there a reason why no cross-signed Intermediates with the compromised keys are included in OneCRL? A user asked: Issuer: CN=AAA Certificate Services,O=Comodo CA Limited,L=Salford,ST=Greater Manchester,C=GB Is there a reason why no cross-signed Intermediates with the compromised keys are included in OneCRL? Issuer: CN=AAA Certificate Services,O=Comodo CA Limited,L=Salford,ST=Greater Manchester,C=GB Is there a reason why no cross-signed Intermediates with the compromised keys are included in OneCRL? This Issuer is not included in Mozilla's Root Store. It is not trusted nor are the intermediates it issued. Consequently there is no need to add it to OneCRL. You can see that Firefox clients reject these certificates via test sites like https://emudhra.we-have-a-problem.com/. Unless you are contributing directly to the discussion about the incident described in this bug, please do not post other questions or concerns here. If the concern relates to Firefox or another Mozilla product, you can file a new bug and describe your concern there, referencing this bug. Am I missing something? On 158.0.1 android Everyone checks fail on https://emudhra.we-have-a-problem.com/ Same for me, all checks red on Firefox Android 158.0.1, filed bug 2084517. Thank you. We are aware of the Android problem, no additional comments are required. Some Android profiles were not applying the updated blocklist at startup. A fix (Bug 2084311) has been shipped in 158.0.2 and pushed to users. Unless you are contributing directly to the discussion about the incident described in this bug, please do not post other questions or concerns here. If the concern relates to Firefox or another Mozilla product, you can file a new bug and describe your issue there, referencing this bug. Updated We request that eMudhra answer the following questions to help us (and others) better understand the situation and more effectively plan next steps to protect our user communities. We note that eMudhra has published (a) a status page at https://emudhra.com/en/post/certificate-security-incident ("eMudhra Statement on Recent Certificate Security Incident", first published October 20, 2026 and updated October 21, 2026), linked from a notice on the emudhra.com homepage, and (b) an October 21, 2026 press release at https://emudhra.com/en/press-release/emudhra-statement-distrust-global-tls-intermediate-root-ca. However, emsign.com - the brand under which these CAs are named and whose hostnames serve eMudhra's CRL and ACME endpoints - redirects to the CertiNext product page on emudhra.com, which appears to carry neither the notice nor a link to either document. More importantly, neither document appears to contain the information subscribers and relying parties need: Neither identifies the distrusted root (emSign ECC Root CA - G3) or any of the affected Intermediate CAs (including the Sectigo cross-certificates and the Code Signing CAs) by name, serial number, or SHA-256 fingerprint; both refer only to "eMudhra's ECC root" and eight intermediate CAs; Neither provides browser or root store trust removal or modification timelines (beyond quoting Mozilla's Firefox and NSS releases), nor links to this bug; and Neither explains how a subscriber can determine whether its certificates chain to the distrusted root or what action to take, beyond stating that eMudhra has "notified affected customers and partners directly" and is "working directly with customers who may be affected to give them replacement certificates." (Q1): How and when is eMudhra communicating actionable technical details (affected CA identifiers, root store trust removal timelines, and migration instructions) to affected subscribers and relying parties, and will this information be added to the status page and surfaced on emsign.com? (Q2): Please confirm whether affected subscribers are being migrated to eMudhra's emSign Root CA - G1 hierarchy, to a hierarchy operated by another CA Owner (please name the corresponding root), or both. The most recent update states replacement certificates are being issued "from an alternate, independently operated, publicly trusted CA hierarchy." (Q3): Please describe how eMudhra intends to offer "CA redundancy for all Enterprise TLS customers. We are offering every Enterprise TLS customer certificates from an additional publicly trusted CA." (from the October 21 update) and on what timeline this is expected for all subscribers affected by this incident. In Comment 19, eMudhra estimated that "there are about 30,000 certs that require migration and about 97% came via ACME," and stated that affected subscribers are being transitioned to hierarchies chaining to emSign Root CA - G1. Based on CT data as of October 21, 2026, approximately 30,800 unexpired precertificates chain to emSign ECC Root CA - G3. 30,791 (99.9%) were issued by two Internet2 / InCommon Subordinate CAs (InCommon Intermediate CA - OVG3C [23,806] and InCommon Intermediate CA - DVG3C [6,985]), and 18 were issued by eMudhra's own Intermediate CAs (EM EV TLS CA - G3A-1, EM OV TLS CA - G3A-1, emSign ECC SSL CA - G3, emSign ECC EV SSL CA - G3, and emSign TLS CA - G3), excluding precertificates logged after October 17, 2026 under the compromised EM DV TLS CA - G3B. (Q4): How, specifically, is eMudhra assisting subscribers in transitioning off the emSign ECC Root CA - G3 hierarchy? (Q5): Because standard ACME clients do not automatically replace certificates outside their configured renewal window without ACME Renewal Information support or manual intervention - and because transitioning from an ECC G3 hierarchy to an RSA G1 hierarchy may require client-side configuration… (a): What concrete mechanism is eMudhra using to trigger and complete renewals for ACME subscribers? (a): What concrete mechanism is eMudhra using to trigger and complete renewals for ACME subscribers? (b): Can you describe the set of steps subscribers need to take to complete the transition? (b): Can you describe the set of steps subscribers need to take to complete the transition? (Q6): Please provide exact metrics, broken down by issuing Intermediate CA: Total number of affected subscribers and active (unexpired, unrevoked) certificates prior to the start of migration; Number of subscribers and certificates that have successfully transitioned to a replacement hierarchy to date; Number of subscribers and certificates remaining to be transitioned; and eMudhra's projected completion date for 100% of affected subscribers, particularly in light of imminent root store removals (including Chrome Root Store v43, which we expect to land in Stable Tuesday morning, October 27, 2026 PDT). In Comment 11, eMudhra stated that during the April 30, 2025 Key Generation Ceremony (KGC): "a set of original ECC keys were created onboard as expected" "an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance" "a flaw in the code resulted in the second pair of keys being created onboard which were weaker" "these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root" Regarding Bug 1970728 (filed June 5, 2025; per eMudhra's Full Incident Report in Comment 2 of that bug, the CRLs for EM [DV/OV/EV] TLS CA - G3B and G3C were generated with the wrong key on May 29, 2025 and corrected on June 6, 2025): "The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys." (from Comment 11) This disclosure raises serious concerns regarding eMudhra's transparency and incident handling during and after Bug 1970728: (Q7): At what exact date and time did eMudhra personnel first realize that multiple key pairs with similar labels existed on the HSM for the G3B and G3C Intermediate CAs? (Q8): When investigating Bug 1970728 in June 2025 - upon finding that the key used to sign each CRL did not match the public key in the corresponding issued CA certificate, and then locating and using a second key that did - why did eMudhra simply re-sign the CRLs with that second key rather than halting operations, auditing why two key pairs existed for these CAs from the April 30, 2025 KGC, and revoking the certificates? (Q9): eMudhra's Bug 1970728 Full Incident Report described a root cause contributing factor as "Incorrect Key Configuration Used During Manual CRL Generation" ("A manual CRL generation process used incorrect CA key configuration due to insufficient verification steps") and stated only that "While the underlying keys were from the issuing environment, they did not match the exact keys expected for those Intermediate CA CRLs, thereby resulting in signature validation failure." Per Comment 11 in this bug, the keys that signed those CRLs were the original keys generated for these same CAs during the April 30, 2025 KGC, and the CA certificates had instead been issued over a second, unintended set of keys. Why did eMudhra's report omit that two key pairs existed on the HSM for each of these CAs, that the KGC had deviated from its usual process, and that the issued CA certificates did not correspond to the intended ceremony keys? (Q10): Comment 11 states that the CSRs from the weak keys "were also sent over to Sectigo for cross signing," and the resulting Sectigo (AAA Certificate Services) cross-certificates carry a notBefore of May 14, 2025. After eMudhra's June 2025 investigation established that the keys used to sign the CRLs did not match the keys in the issued CA certificates, did eMudhra notify Sectigo - or any Root Program - that the keys Sectigo had cross-certified were not the keys eMudhra intended to generate? If not, why not? (Q11): As noted in Comment 5, the April 30, 2025 ceremony generated 24 Intermediate CAs across Series A, B, and C (disclosed in Bug 1965559), and Series A was revoked and replaced by Series A-1 in May 2025. While eMudhra noted in Comment 11 that "The A and A-1 series were created in Entrust HSMs and were meant for deployment only in India," eMudhra did not explain why Series A was replaced: (a): Why was Series A revoked and replaced by Series A-1 in May 2025? (a): Why was Series A revoked and replaced by Series A-1 in May 2025? (b): Was the reason for that replacement related in any way to the KGC script, procedure, or key-handling steps used in the same ceremony, and if so, why did it not prompt a review of the Series B and C keys? (b): Was the reason for that replacement related in any way to the KGC script, procedure, or key-handling steps used in the same ceremony, and if so, why did it not prompt a review of the Series B and C keys? (Q12): As noted in Comment 7 and Comment 8, on October 14, 2026, eMudhra published CRL #11 for emSign Root TLS CA - G1 (roottlscag1.crl) with an invalid cryptographic signature that does not verify against emSign Root TLS CA - G1 or any CCADB-disclosed 4096-bit RSA CA public key. (a): Which private key was used to sign roottlscag1.crl (CRL #11) on October 14, 2026, and do multiple key pairs also exist on eMudhra's HSMs for emSign Root TLS CA - G1 or any other Root CA? (a): Which private key was used to sign roottlscag1.crl (CRL #11) on October 14, 2026, and do multiple key pairs also exist on eMudhra's HSMs for emSign Root TLS CA - G1 or any other Root CA? (b): In Bug 1970728 Comment 2 (June 2025), eMudhra committed to "Introduce automated post-signing signature validation checks in CRL publishing steps, particularly during onboarding of new Issuing CAs" and to "Replace manual CRL signing process with an automated workflow to eliminate reliance on human configuration of key mappings." Why did those controls fail to prevent the publication of an invalidly signed Root CA CRL on October 14, 2026 (again, described in Comment 7 and attached in Comment 8)? (b): In Bug 1970728 Comment 2 (June 2025), eMudhra committed to "Introduce automated post-signing signature validation checks in CRL publishing steps, particularly during onboarding of new Issuing CAs" and to "Replace manual CRL signing process with an automated workflow to eliminate reliance on human configuration of key mappings." Why did those controls fail to prevent the publication of an invalidly signed Root CA CRL on October 14, 2026 (again, described in Comment 7 and attached in Comment 8)? I have a question relating to the disclosures, the eMudhra CA policies, and scrutiny that may need to be applied to other CAs using Utimaco hardware. I do not believe that an HSM bearing the various certifications the Utimaco units bear should be able to internally generate a series of private keys that have the unfortunate property these have. Section 6.1.1 of the the eMudhra policy states unambiguously that: Issuing CA key generation is carried out in a secure device that is at least FIPS 140-2 Level 3 compliant. Issuing CA key generation is carried out in a secure device that is at least FIPS 140-2 Level 3 compliant. This implies that one of two things must be true. Either: The policy was not followed, and the keys were generated outside the HSM and then loaded into it. The policy was followed, which means there was some way to configure the HSM to internally generate the defective keys. It seems to me that anyone involved in root of trust management should be keenly interested in knowing which of these two things happened. Both, it seems to me, have major implications for responsible inclusion of CAs going forward. If the Utimaco HSMs can in fact generate keys that obey outrageous constraints as may have happened here, or come with test keys of some kind that can be mapped into partitions actually used to sign customer CSRs, then I think CAs should be required to state whether or not they are using these HSMs, and if so, should be required migrate away from them as a condition of inclusion in the default browser bundles. If they cannot, then that means eMudhra must have generated these keys outside the HSM and then loaded them. If so, then I think in the long term a scheme for CAs to provide an attestation, signed by a manufacturer key, that any keys to be included in the default browser bundles were generated in hardware -- not loaded! -- should be devised and required. I know of at least one HSM vendor whose products can already do this; I also know of one whose products cannot. But it seems like it ought to be the baseline. Thor eMudhra: following on comment 31. BR 6.1.1.1 describes the record that should answer Thor's question directly. For each affected CA key pair, please attach or link: The Key Generation Script followed. The ceremony report, and either the Qualified Auditor's witness report or the video recording. If no such record exists for a given key, please say so. The HSM model, firmware version, and CMVP certificate number, and whether the module was operating in its FIPS-approved mode at generation time. Whether any custom or non-standard firmware modules were loaded on the module at generation time. The generation mechanism used, and whether any key material, seed, or domain parameters were imported, or any key was restored from a backup originating on a different device. Separately: CPS 6.1.1 says key generation is "carried out in a secure device that is at least FIPS 140-2 Level 3 compliant." Please confirm whether "compliant" means operation in a CMVP-validated configuration, or something else. eMudhra: following on comments 10, 11, and 31. Comment 11 attributes the weak keys to a code flaw during an approved wrap/unwrap test deviation from the usual KGC, at the 30 April 2025 ceremony identified in comment 10. A P-384 private scalar below 256 is not something approved in-module key pair generation produces by itself. It implies either generation fed a supplied value, or a key object created or unwrapped from supplied key material. Which of those occurred answers comment 31. Please provide: The KGC script used on 30 April 2025, including the approved deviation, and the specific code flaw identified. The exact call or mechanism that created the second key pair (e.g., C_GenerateKeyPair, C_UnwrapKey, C_CreateObject, or a vendor-specific equivalent), and the source of any key material, seed, or wrapped blob supplied to it. The ceremony report, the witness report or video, and the key generation logs for that ceremony. eMudhra's repository already publishes a WebTrust key generation report for the 8 February 2024 root ceremony (emSignRootKGC2024.pdf), citing a Root Key Generation Script v1.0, so the equivalent for 30 April 2025 should exist. Whether the module was operating in its FIPS-approved mode during the deviation, and its CMVP certificate number. Who approved the deviation, and whether any auditor witnessing the ceremony observed it. On applicability: these key pairs were cross-certified by Sectigo. BR 6.1.1.1 places a CA Key Pair used for a Subordinate CA Certificate, where the Subordinate CA is not the operator of the Root CA or its Affiliate, in the SHALL tier. Please state whether eMudhra treated these key pairs as within that tier, and if so, provide the Qualified Auditor's report. Separately: the SSL/TLS CP/CPS v1.08 (13 Aug 2026) describes CA key generation and storage as "FIPS 140-2 Level 3 compliant" (§6, §6.1.1, §6.2.1), and §6.2.11 refers back to §6.2.1. BR 6.2.7 requires protection in a device "validated as meeting" FIPS 140-2/140-3 Level 3 or the Common Criteria equivalent. Please confirm whether "compliant" means operation in a validated configuration, and identify the validation. Following on Comment 7 and Comment 30 CRL #11 appears to have been signed with the private key corresponding to EMSIGN ROOT TLS CA - G1 OCSP 2026 - 4. This certificate was not previously available on crt.sh, so I have just submitted it through Merkle Lens, so it should appear in crt.sh shortly. I retrieved the OCSP certificate from https://crt.sh/ocsp-response?caID=296739&url=http%3A%2F%2Focsp-a.emsign.com&request=get The following commands extract the certificate and verify the CRL’s signature using its public key. This is somewhat cumbersome because OpenSSL’s normal CRL issuer validation does not support using a mismatched certificate. curl -sS --retry 5 --retry-delay 5 \ 'https://crt.sh/ocsp-response?caID=296739&url=http%3A%2F%2Focsp-a.emsign.com&request=get&type=asn1' \ | perl -0777 -ne 'print $1 if /Base64\.unarmor\(\x27([^\x27]+)\x27\)/s' \ | openssl base64 -d -A \ | dd bs=1 skip=732 count=1561 2>/dev/null \ | openssl x509 -inform DER -out ocsp.pem openssl dgst -sha384 -verify <(openssl x509 -in ocsp.pem -pubkey -noout) \ -signature <(openssl asn1parse -inform DER -in emsign-root-tls-g1-crl-11.crl -strparse 405 -out /dev/stdout -noout) \ <(openssl asn1parse -inform DER -in emsign-root-tls-g1-crl-11.crl -strparse 4 -out /dev/stdout -noout) It is somewhat reassuring that this appears to be a case of using the wrong signing key, rather than evidence of another weak key. Thanks eMudhra: two questions on the public status page (https://emudhra.com/en/post/certificate-security-incident, Update 3, 21 October 2026). Q1. Timeline. The status page states eMudhra was made aware on "October 18, 2026 (late evening, India time)" and revoked the intermediates on October 19. This bug records the report as sent at 18:05 UTC on October 17 (comment 2) and eMudhra's acknowledgment at 21:28 UTC on October 17. eMudhra's own table in comment 10 shows all eight intermediates revoked on October 18, between 09:24 and 09:38 UTC. Please correct the status page or explain the difference. Q2. Audits. The status page states the hierarchy "has since passed two independent audit cycles." Bug 1970728 (June 2025) concerned CRLs for these same intermediates being signed with keys that did not match the issued CA certificates. Was Bug 1970728 disclosed to the auditors and within the scope of the subsequent audits? If so, did the auditors examine why the CRL signing keys did not match the certificates? (In reply to Zihao Wang from comment #34) Following on Comment 7 and Comment 30 CRL #11 appears to have been signed with the private key corresponding to EMSIGN ROOT TLS CA - G1 OCSP 2026 - 4. This certificate was not previously available on crt.sh, so I have just submitted it through Merkle Lens, so it should appear in crt.sh shortly. Following on Comment 7 and Comment 30 CRL #11 appears to have been signed with the private key corresponding to EMSIGN ROOT TLS CA - G1 OCSP 2026 - 4. This certificate was not previously available on crt.sh, so I have just submitted it through Merkle Lens, so it should appear in crt.sh shortly. There's a lot of people new to this part of Bugzilla, and CCADB Incident Handling in general. The trick here is that this answer was already known, but we need to actually see if the CA itself is aware, and could provide that answer without having their homework done for them. Although given the lack of response from eMudhra lately this getting an answer was unlikely. It is somewhat reassuring that this appears to be a case of using the wrong signing key, rather than evidence of another weak key. It is somewhat reassuring that this appears to be a case of using the wrong signing key, rather than evidence of another weak key. Here's the catch: it isn't reassuring. The CRL I gave was just one of them detected on 10-14 where they signed with OCSP certificates generated that day, and then swapped them on the next CRL issuance without reporting it to anyone. I only started monitoring CRLs very recently, so this has likely been an issue for a while. It also relates to the current claims about the Key Generation Ceremony, how they seem to be juggling what-signs-what, have no actual fix for #1970728, and are less than transparent when key mishaps happens and don't self-report. There's a lot of attention on this bug in general, naturally, but could we have general chat kept on MDSP? Try to collate questions together there, but keep in mind a lot of the theorizing is not going to help us get direct answers quicker and could let eMudhra see if their answers align with what people want to hear, rather than what actually happened. I'm a relying party as well, last thing we want is this bug to get restricted comments when someone has something to bring up. Right now we're waiting of eMudhra to answer a lot of questions. I note eMudhra's recent press release stating that they have "decided to voluntarily withdraw from the global SSL/TLS certificate business". This press release answers few questions and raises yet more. It states that they "will reach out to affected customers to replace any certificate free of cost from partners through the transition". Could eMudhra please confirm: who those partners are? which roots replacement certificates will be issued under numbers for the amount of certificates that will need to be reissued? Additionally, could they please confirm: the timeline for such replacements? how they will ensure that end-users are moved over to alternative roots in good time? The press release also states that whilst eMudhra will be exiting the TLS certificate business, they will continue issuing digital signature certificates. Could eMudhra please confirm if these continuing lines of business relate to any other (non-TLS) root certificates in the CCADB, such as: emSign Root CS CA - G2 (SHA256 0D74EF67C88DE3560365E225AEF976F915E901347B73352D81AB5B4132BB2150) emSign Root CS CA - G3 (SHA256 F2EDAAFF3D9B701FB4D14C9F285F8C0BAD90FC3BCC1FA07D47F4718EDABA34A5) emSign Root SMIME CA - G1 (SHA256 47911BC8F32A11B029E0BB10A348B459404B87731B870A2F3FA97B4705DC4C25) emSign Root SMIME CA - G3 (SHA256 F76A0D440C418DC5B362BE8E11E0A5D06023F4396093E4F455827BF55153D8A4) If this is the case, please answer: how does eMudhra plan to maintain compliance with the relevant Baseline Requirements for these roots? how do they plan to avoid a repeat of this incident with the above roots? I have carefully reviewed the case, and I still have one fundamental question, which I would like to raise in two ways. I hope someone from eMudhra or Utimaco can provide a clear technical explanation. HSM Key Generation eMudhra stated that the affected keys were generated using FIPS 140-2 Level 3 certified HSMs. Were the weak private keys actually generated through the HSM's certified internal key-generation mechanism and random number generator? If so, how could such low-entropy keys be produced without being detected or prevented by the HSM's security controls? KGC Script and Root Cause What exact flaw in the KGC script caused the Wrap/Unwrap operation to result in a second set of extremely weak ECC private keys? Were these keys generated internally by the HSM or introduced through another mechanism, and which security controls were expected to detect or prevent this? I believe answering these questions is essential to understanding the actual root cause of the incident. There are a lot of questions here. As most of you can imagine, we have been subject to intense social media scrutiny and analysis by investors/customers over the last few days and it has made doing any meaningful analysis of the event pretty much impossible. The event happened on April 30 2025 and was done by set of people who have worked in the PKI industry for several years and worked with us for a while. There is a lot of digging we need to do to understand the circumstances, why, what, how to share a meaningful report that can hopefully benefit the community. I do not know why some community members think we wont be getting back on this bug. When we signed up for the browser program trust, we also signed up for disclosure obligations to the community which we will uphold. Naveen. MHD Ghazi Alkhalaf - I am clearing this needinfo because I believe you intended to address your questions to eMudhra rather than Mozilla. [In response to Comment 39.] While we understand that many things are happening in parallel right now, "social media scrutiny" and "analysis by investors/customers" are not acceptable explanations for delayed responses here. That scrutiny is the predictable result of eMudhra's own failure (i.e., intermediate CA keys that anyone could recover -- resulting in anyone being able to issue certificates for any domain), not an external circumstance that befell it. Offering it as an excuse conveys that reputation management is being prioritized over the obligations that eMudhra still owes to the billions of relying parties its mistake exposed. Those obligations do not relax because the consequences are uncomfortable. We expect eMudhra to prioritize its review of this issue and its security response. The Mozilla Root Program sent the attached questions to eMudhra on the 20th October at 20:31 UTC. eMudhra was asked to provide an initial response within 24 hours. After the deadline passed on October 21st, eMudhra asked for a further 24 hours to respond. As of the time of writing on Friday 23rd, Mozilla has not received a response to these questions. We share the Chrome Root Program's concerns and whilst we appreciate eMudhra's difficulties, we expect eMudhra to uphold its obligations to Mozilla and the wider community. Attachment General Description File Name Content Type