When an upstream change broke smartcard FIPS authentication – and how we fixed it

A government agency mandated smartcard authentication across their Ubuntu fleet. When they enabled FIPS mode to meet compliance requirements, smartcard authentication stopped working. Nearly 1,000 systems sat waiting for FIPS rollout.

This is the story of how Canonical’s Support team provided bug-fix support: we tracked down an upstream change in OpenSC that inadvertently broke FIPS compatibility, coordinated with upstream developers across distributions, and delivered both a hotfix and a proper universal solution.

The setup: finding what broke

When the case first opened, the error messages looked like standard authentication failures. Nothing pointed to the root cause.

Our support engineer started by replicating the issue on a Noble (24.04 LTS) test system. The pkcs11 tooling was segfaulting on Noble FIPS installations, but the same tools worked fine on Jammy (22.04 LTS). Something had changed in the OpenSC package – the open source software toolkit for using cryptographic smartcards across various operating systems, which the government agency used for authentication – between Ubuntu 22.04 LTS and 24.04 LTS. But what?

Within days, we’d narrowed the problem to changes in the upstream OpenSC codebase, ruling out conflicts with our FIPS implementation. An upstream change in how OpenSC handled provider initialization was causing crashes when it tried to interact with OpenSSL in FIPS mode.

The investigation: confirming the root cause

We filed upstream bug reports with both OpenSSL and OpenSC to see if they had insights. Then we kept debugging.

The breakthrough came quickly: the FIPS OpenSSL provider was being initialized incorrectly. OpenSC was setting the wrong OpenSSL provider in its backend, using the default provider instead of the FIPS provider, which blocked the hashing mechanisms needed for smartcard authentication.

We built a test to confirm this was the actual root cause. The customer tested it in their environment. It worked.

This is the challenge of managing upstream dependencies in enterprise Linux: OpenSC had evolved its provider handling between releases, introducing behavior that broke FIPS compatibility. An upstream development decision had unintended consequences in specific compliance contexts. 

Enterprise bug-fix support bridges this gap: identifying which upstream change caused the regression, understanding why, and coordinating a universal fix.

The fix: unblock now, solve fully later

Working with OpenSC maintainers, we reached the final diagnosis: upstream changes to OpenSC meant it now attempts to load an algorithm disallowed under FIPS policy. OpenSSL correctly returns a NULL pointer, but OpenSC still tries to use it as a valid object. Instead of gracefully handling the situation – trying different crypto options or querying what’s available – it errors out.

We shared a hotfix – a common practice to restore functionality in our enterprise bug-fix support service – to unblock the customer so they could move forward with their FIPS rollout.

But a hotfix isn’t the same as a comprehensive fix. This needed a solution that worked upstream – fixing the issue in OpenSC itself so it would work across Ubuntu and any other distribution using OpenSC in FIPS environments, not just a distro-specific patch.

This week, we agreed on the shape of the final fix. The upstream PR is waiting to merge. Once that’s done, we’ll backport it to Ubuntu. This will resolve all three customer cases we’ve seen with the same root cause.

The key ingredient: collaboration across the ecosystem

Dariusz Gadomski, a member of our Support team who worked on this case, reflected on what made it work:

Fixing this bug involved coordination across support engineering, Canonical’s security team, and the OpenSC upstream community. What appeared as an authentication failure was an upstream change to provider initialization that broke FIPS compatibility. The diagnosis demanded understanding how OpenSC, OpenSSL, and FIPS mode interact across versions, then collaborating with upstream maintainers to develop a universal solution.

Learn more about bug-fix support through Ubuntu Pro + Support

• Read more about what Linux support actually means
• Explore Ubuntu Pro + Support options
• Get in touch to discuss your specific needs