Error in mTLS in node js using PKCS11 proxy and HSM

[rahul-vrutti]

Description

I am implementing mutual TLS (mTLS) between a Node.js server and client using private keys stored in an HSM, accessed via a local PKCS#11 proxy on each machine.

Setup

• Both server and client have their own HSM + PKCS#11 proxy running locally
• Server and client are running on different machines
• Each side uses its own HSM instance for private key operations
• Private keys are accessed via PKCS#11 proxy
• Certificates are loaded from local PEM files

Architecture Diagram

        ┌──────────────────────────────┐          ┌──────────────────────────────┐
        │        Server Machine        │          │        Client Machine        │
        │                              │          │                              │
        │   +----------------------+   │          │   +----------------------+   │
        │   |        HSM           |   │          │   |        HSM           |   │
        │   |  (Private Key:      |   │          │   |  (Private Key:      |   │
        │   |   CORE0001)         |   │          │   |   EDGE0001)         |   │
        │   +----------+-----------+  │          │   +----------+-----------+  │
        │              |              │          │              |              │
        │              | PKCS#11      │          │              | PKCS#11      │
        │              |              │          │              |              │
        │   +----------v-----------+  │          │   +----------v-----------+  │
        │   |   PKCS#11 Proxy      |  │          │   |   PKCS#11 Proxy      |  │
        │   |   (Local)            |  │          │   |   (Local)            |  │
        │   +----------+-----------+  │          │   +----------+-----------+  │
        │              |              │          │              |              │
        │   +----------v-----------+  │          │   +----------v-----------+  │
        │   |    Node.js Server    |  │          │   |    Node.js Client    |  │
        │   |    (socket.io)       |  │          │   | (socket.io-client)   |  │
        │   +----------------------+  │          │   +----------------------+  │
        │                              │          │                              │
        └──────────────┬───────────────┘          └──────────────┬───────────────┘
                       │                                         │
                       │<---------- mTLS (HTTPS) --------------->│
                       │                                         │

Libraries

• Server: socket.io (^4.8.0)
• Client: socket.io-client (^4.8.0)

Server (Working)

Key TLS config:

const httpsOptions = {
  cert: serverCert,
  privateKeyEngine: "pkcs11",
  privateKeyIdentifier: PKCS11_KEY_URI,
  ca: caCert,
  requestCert: true,
  rejectUnauthorized: true,
  minVersion: "TLSv1.2",
  maxVersion: "TLSv1.2",
};

Server starts successfully and loads the private key from HSM without issues.

Client (Failing)

Key TLS setup:

const secureContext = createSecureContext({
  cert: clientCert,
  privateKeyEngine: "pkcs11",
  privateKeyIdentifier: PKCS11_KEY_URI,
  ca: caCert,
  minVersion: "TLSv1.2",
});

const agent = new HttpsAgent({
  secureContext,
  rejectUnauthorized: true,
  checkServerIdentity: () => undefined,
});

Socket.IO connection:

const socket = io(SERVER_URL, {
  agent,
  rejectUnauthorized: true,
  transports: ["websocket", "polling"],
  timeout: 20000,
});

Issue

• Client connection times out
• No TLS handshake is completed
• No useful TLS errors on client or server
• Server never logs secureConnection

Observations

• Server works correctly with PKCS#11 key
• Client fails only when using PKCS#11 private key
• Both sides use same CA
• Network connectivity is confirmed
• Each side uses its own HSM + PKCS#11 proxy locally

Expected Behavior

Client should complete TLS handshake using private key from HSM and connect successfully.

Question

Is there any limitation or required configuration when using privateKeyEngine + privateKeyIdentifier with socket.io-client (via https.Agent)?

Any guidance on debugging PKCS#11-based TLS handshake in Node.js would be helpful.

[dominiccarrington61-a11y]

Thanks for the detailed setup — this helps a lot. One thing to check: Node’s TLS client side doesn’t always fully support privateKeyEngine/PKCS#11 the same way the server does, especially through https.Agent. A couple things I’d try: Verify OpenSSL engine support is actually loaded for the client (OPENSSL_CONF, engine init, etc.) Test a plain https.request (no socket.io) to isolate whether it’s TLS vs socket.io Enable TLS debug logs (NODE_DEBUG=tls) to see if the handshake even starts Confirm the PKCS#11 proxy supports client-side signing (not just server key ops) If the plain HTTPS call fails the same way, it’s likely a Node/OpenSSL client limitation rather than socket.io.

…

On Thu, Mar 26, 2026, 10:46 PM Rahul Kaklotar ***@***.***> wrote: *rahul-vrutti* created an issue (socketio/socket.io#5480) <#5480> Description I am implementing mutual TLS (mTLS) between a Node.js server and client using private keys stored in an HSM, accessed via a local PKCS#11 proxy on each machine. Setup - Both server and client have their own HSM + PKCS#11 proxy running locally - Server and client are running on different machines - Each side uses its own HSM instance for private key operations - Private keys are accessed via PKCS#11 proxy - Certificates are loaded from local PEM files Architecture Diagram ┌──────────────────────────────┐ ┌──────────────────────────────┐ │ Server Machine │ │ Client Machine │ │ │ │ │ │ +----------------------+ │ │ +----------------------+ │ │ | HSM | │ │ | HSM | │ │ | (Private Key: | │ │ | (Private Key: | │ │ | CORE0001) | │ │ | EDGE0001) | │ │ +----------+-----------+ │ │ +----------+-----------+ │ │ | │ │ | │ │ | PKCS#11 │ │ | PKCS#11 │ │ | │ │ | │ │ +----------v-----------+ │ │ +----------v-----------+ │ │ | PKCS#11 Proxy | │ │ | PKCS#11 Proxy | │ │ | (Local) | │ │ | (Local) | │ │ +----------+-----------+ │ │ +----------+-----------+ │ │ | │ │ | │ │ +----------v-----------+ │ │ +----------v-----------+ │ │ | Node.js Server | │ │ | Node.js Client | │ │ | (socket.io) | │ │ | (socket.io-client) | │ │ +----------------------+ │ │ +----------------------+ │ │ │ │ │ └──────────────┬───────────────┘ └──────────────┬───────────────┘ │ │ │<---------- mTLS (HTTPS) --------------->│ │ │ Libraries - Server: socket.io (^4.8.0) - Client: socket.io-client (^4.8.0) Server (Working) Key TLS config: const httpsOptions = { cert: serverCert, privateKeyEngine: "pkcs11", privateKeyIdentifier: PKCS11_KEY_URI, ca: caCert, requestCert: true, rejectUnauthorized: true, minVersion: "TLSv1.2", maxVersion: "TLSv1.2",}; Server starts successfully and loads the private key from HSM without issues. Client (Failing) Key TLS setup: const secureContext = createSecureContext({ cert: clientCert, privateKeyEngine: "pkcs11", privateKeyIdentifier: PKCS11_KEY_URI, ca: caCert, minVersion: "TLSv1.2",}); const agent = new HttpsAgent({ secureContext, rejectUnauthorized: true, checkServerIdentity: () => undefined,}); Socket.IO connection: const socket = io(SERVER_URL, { agent, rejectUnauthorized: true, transports: ["websocket", "polling"], timeout: 20000,}); Issue - Client connection times out - No TLS handshake is completed - No useful TLS errors on client or server - Server never logs secureConnection Observations - Server works correctly with PKCS#11 key - Client fails only when using PKCS#11 private key - Both sides use same CA - Network connectivity is confirmed - Each side uses its own HSM + PKCS#11 proxy locally Expected Behavior Client should complete TLS handshake using private key from HSM and connect successfully. Question Is there any limitation or required configuration when using privateKeyEngine + privateKeyIdentifier with socket.io-client (via https.Agent)? Any guidance on debugging PKCS#11-based TLS handshake in Node.js would be helpful. ChatGPT.Image.Mar.27.2026.09_13_04.AM.png (view on web) <https://github.com/user-attachments/assets/98ec8e1c-5815-4baf-b415-fea4323ba4d2> — Reply to this email directly, view it on GitHub <#5480?email_source=notifications&email_token=B74GC3HJN6R6YYWSKS2X7H34SX2SRA5CNFSL4Z3JMQ5C6L3HNF2C22DVMIXUS43TOVSS6NBRGQ4TAMJVG42TPJTSMVQXG33OVJZXKYTTMNZGSYTFMSSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/B74GC3D2OS2D6VO4OHM4CDD4SX2SRAVCNFSM6AAAAACXBKXOOSVHI2DSMVQWIX3LMV43ASLTON2WKOZUGE2DSMBRGU3TKNY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[darrachequesne]

@rahul-vrutti did you find the culprit?