Most guides for connecting Spark to Amazon MSK assume the two live in the same cloud and authenticate with IAM. That covers a lot of cases. It does not cover the one that keeps showing up in large enterprises: the Kafka cluster is in AWS, the compute is Azure Databricks, IAM is off the table because the identity system is a corporate PKI, and the traffic never touches the public internet. This walks through that setup end to end. Certificates from a private CA, a private network path between the two clouds, and a Structured Streaming job that actually keeps running on a multi-node cluster instead of only on the driver. Why mTLS instead of IAM MSK offers four authentication modes: plaintext, TLS with client certificates, SASL/SCRAM, and IAM. IAM is the easiest and the best choice when your consumers run in AWS. Cross-cloud, IAM stops being convenient. Azure Databricks executors have no AWS identity. You can bolt one on with OIDC federation and assumed roles, but in most regulated enterprises the decision has already been made elsewhere: there is a corporate PKI, every service-to-service hop uses client certificates issued from it, and the security architecture review is going to ask why this one connection is different. mTLS is the path of least resistance, not the clever choice. One important constraint before you start. MSK will only accept client certificates issued by an AWS Private CA (ACM PCA) that is associated with the cluster. If your corporate PKI is not that CA, you have two options: stand up an ACM PCA subordinate signed by your corporate root, or issue the Databricks client certificate from a dedicated ACM PCA and treat it as a separate trust domain. The subordinate route is usually what security wants, and it takes longer to get approved than everything else in this article combined. Start that conversation first. Network path Three ways to get private connectivity between an Azure VNet and an AWS VPC: Site-to-site VPN. IPsec tunnel between an Azure VPN Gateway and an AWS Virtual Private Gateway. Cheapest, slowest, and fine for moderate throughput. Dedicated interconnect. ExpressRoute on the Azure side and Direct Connect on the AWS side, joined at a colocation provider. Higher throughput, predictable latency, much longer lead time. Corporate backbone. Both clouds already attach to an existing enterprise MPLS or SD-WAN network, and you route between them through it. Common in large organizations, and it usually means your firewall rules go through a change process rather than a Terraform apply. The mechanics below are identical in all three cases. What matters is that the Databricks worker subnets can reach the MSK broker ENIs on TCP 9094. Databricks side The cluster must run in a VNet you control. That means VNet injection, not the managed default VNet, because you need route tables and a path off the Azure network. Enable secure cluster connectivity so workers have no public IPs. Then confirm the worker subnet's effective routes actually send the MSK CIDR toward your gateway or firewall, rather than to Internet. This is the single most common reason a cluster that looks correctly configured cannot connect. DNS, and a trap worth knowing about MSK bootstrap endpoints look like this: b-1.mycluster.a1b2c3.c4.kafka.us-east-1.amazonaws.com:9094 These are public DNS names that resolve to private VPC addresses. So the Databricks cluster needs to be able to resolve public DNS, even though no traffic leaves the private path. If your VNet points at corporate DNS servers with no external resolution, add a conditional forwarder for kafka.<region>.amazonaws.com. Do not shortcut this by creating a private DNS zone with hardcoded A records for each broker. It works on the day you build it. Then MSK replaces a broker during patching, the ENI gets a new address, your pinned record goes stale, and you get a partial outage where two brokers work and one times out. That failure is miserable to diagnose because the consumer keeps running and only some partitions stall. MTU If you are going over IPsec, clamp TCP MSS on the tunnel. A TLS handshake carrying a full certificate chain produces packets large enough to need fragmentation, and some paths drop the fragments silently. The symptom is a handshake that hangs rather than fails, which sends people hunting through certificate configuration for hours. Rule out the network first: if openssl s_client from a VM in the same subnet also hangs, it is not your keystore. Certificates Issue a client certificate from the ACM PCA associated with the cluster. Give it a Distinguished Name you are willing to live with, because MSK uses that DN as the ACL principal and changing it later means rewriting every ACL. Package the certificate, its private key, and the issuing chain into a PKCS12 keystore: openssl pkcs12 -export \ -in client-cert.pem \ -inkey client-key.pem \ -certfile ca-chain.pem \ -name databricks-consumer \ -out kafka.keystore.p12 \ -password pass:"$KEYSTORE_PASSWORD" You probably do not need a truststore. MSK broker certificates are issued by Amazon Trust Services, a public CA already present in the JVM's default cacerts. If you omit the truststore options entirely, the Kafka client uses the JVM default and broker validation just works. Add an explicit truststore only if you have a TLS-inspecting middlebox on the path presenting its own certificates, or if you are running self-managed Kafka rather than MSK. If you do need one: openssl pkcs12 -export -nokeys \ -in ca-chain.pem \ -out kafka.truststore.p12 \ -password pass:"$TRUSTSTORE_PASSWORD" Base64-encode both files and store them in Azure Key Vault, along with the passwords: base64 -w0 kafka.keystore.p12 > keystore.b64 Then create a Databricks secret scope backed by that Key Vault. Nothing sensitive lands in a notebook, a repo, or DBFS. The executor problem Here is the part most tutorials get wrong. The Kafka client on every Spark executor opens its own connection to the brokers. ssl.keystore.location is a local filesystem path read by each JVM independently. If the keystore only exists on the driver, your job works perfectly in a notebook against a single-node cluster and then fails the moment it runs on a real cluster, usually with a confusing FileNotFoundException buried in an executor log that nobody is reading. The fix is a cluster-scoped init script, which runs on the driver and every worker before Spark starts. Init scripts cannot call dbutils.secrets. The way to get secret material into one is through cluster environment variables, which support secret references and are redacted in logs. Set these in Advanced options → Spark → Environment variables: KAFKA_KEYSTORE_B64={{secrets/kafka-mtls/keystore-p12-b64}} KAFKA_TRUSTSTORE_B64={{secrets/kafka-mtls/truststore-p12-b64}} And the init script itself, stored as a workspace file or in a Unity Catalog volume (DBFS-hosted init scripts are no longer supported): #!/bin/bash set -euo pipefail CERT_DIR=/local_disk0/kafka-certs mkdir -p "$CERT_DIR" chmod 700 "$CERT_DIR" echo "$KAFKA_KEYSTORE_B64" | base64 -d > "$CERT_DIR/kafka.keystore.p12" echo "$KAFKA_TRUSTSTORE_B64" | base64 -d > "$CERT_DIR/kafka.truststore.p12" chmod 600 "$CERT_DIR"/*.p12 /local_disk0 is node-local ephemeral storage present on every Databricks node. Writing there keeps the keystore off any shared filesystem and guarantees it disappears when the node does. If you are on Unity Catalog shared access mode, init scripts must be on the metastore allowlist. Dedicated (single-user) access mode avoids that entirely and is the simpler starting point. The Spark reader KEYSTORE = "/local_disk0/kafka-certs/kafka.keystore.p12" TRUSTSTORE = "/local_disk0/kafka-certs/kafka.truststore.p12" keystore_pw = dbutils.secrets.get("kafka-mtls", "keystore-password") truststore_pw = dbutils.secrets.get("kafka-mtls", "truststore-password") BOOTSTRAP = ( "b-1.mycluster.a1b2c3.c4.kafka.us-east-1.amazonaws.com:9094," "b-2.mycluster.a1b2c3.c4.kafka.us-east-1.amazonaws.com:9094," "b-3.mycluster.a1b2c3.c4.kafka.us-east-1.amazonaws.com:9094" ) df = ( spark.readStream.format("kafka") .option("kafka.bootstrap.servers", BOOTSTRAP) .option("subscribe", "orders.events") .option("kafka.security.protocol", "SSL") .option("kafka.ssl.keystore.type", "PKCS12") .option("kafka.ssl.keystore.location", KEYSTORE) .option("kafka.ssl.keystore.password", keystore_pw) .option("kafka.ssl.key.password", keystore_pw) .option("kafka.ssl.truststore.type", "PKCS12") .option("kafka.ssl.truststore.location", TRUSTSTORE) .option("kafka.ssl.truststore.password", truststore_pw) .option("kafka.group.id", "databricks-orders-consumer") .option("startingOffsets", "earliest") .option("failOnDataLoss", "false") .load() ) Every Kafka client property gets the kafka. prefix. security.protocol is SSL, not SSL_PLAINTEXT or SASL_SSL. Note that kafka.ssl.key.password is set to the same value as the keystore password. With a PKCS12 produced by the openssl command above they are the same. If you built the keystore some other way and the key has a separate passphrase, use that instead. A mismatch here produces UnrecoverableKeyException, which at least says exactly what is wrong. Drop the two truststore lines if you are using the JVM default, as discussed above. Then write it somewhere with a checkpoint, so restarts resume rather than replay: ( df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)", "topic", "partition", "offset", "timestamp") .writeStream .format("delta") .outputMode("append") .option("checkpointLocation", "/Volumes/main/streaming/checkpoints/orders_events") .trigger(processingTime="30 seconds") .toTable("main.streaming.orders_events_raw") ) Authorization: the DN is your principal Getting the TLS handshake to succeed is not the same as being allowed to read anything. Under mTLS, MSK derives the ACL principal from the client certificate's Distinguished Name, in RFC 2253 form: User:CN=databricks-consumer,OU=Integration,O=Example Corp,L=Dallas,ST=TX,C=US The string must match exactly, including component order and spacing. Read it off the certificate rather than typing it from the CSR you think you submitted: openssl x509 -in client-cert.pem -noout -subject -nameopt RFC2253 Then grant read on both the topic and the consumer group. Missing the group ACL is a classic: the consumer connects, fetches metadata, and then fails on join with a group authorization error that reads nothing like a permissions problem at first glance. kafka-acls.sh --bootstrap-server "$BOOTSTRAP" \ --command-config admin.properties \ --add \ --allow-principal "User:CN=databricks-consumer,OU=Integration,O=Example Corp,L=Dallas,ST=TX,C=US" \ --operation Read --operation Describe \ --topic orders.events kafka-acls.sh --bootstrap-server "$BOOTSTRAP" \ --command-config admin.properties \ --add \ --allow-principal "User:CN=databricks-consumer,OU=Integration,O=Example Corp,L=Dallas,ST=TX,C=US" \ --operation Read \ --group databricks-orders-consumer Be aware that MSK ships with allow.everyone.if.no.acl.found set to true. Until someone adds the first ACL for a resource, everything is permitted, which means your consumer can appear to work correctly while being completely unauthorized. It stops working the day another team locks down that topic. Set the ACLs deliberately. Debugging Turn on handshake logging on both driver and executors: spark.driver.extraJavaOptions -Djavax.net.debug=ssl:handshake spark.executor.extraJavaOptions -Djavax.net.debug=ssl:handshake This is noisy. Turn it off once you are through. Symptom Almost always means SSLHandshakeException: PKIX path building failed Your truststore does not trust the broker's issuer. On MSK, usually caused by setting a custom truststore that lacks the Amazon roots. Try removing the truststore options. Received fatal alert: bad_certificate The broker rejected your client cert. It was not issued by an ACM PCA associated with this cluster, or the chain in the keystore is incomplete. UnrecoverableKeyException: Cannot recover key ssl.key.password does not match the key's actual passphrase. CertificateException: No subject alternative names matching IP You are connecting by IP, or through something that rewrites the destination. Fix the DNS path rather than disabling hostname verification. TimeoutException: Failed to update metadata Network, not TLS. The handshake never got far enough to fail properly. Check routes and firewall rules. Bootstrap succeeds, then partitions stall Advertised broker endpoints are not all reachable. Someone opened 9094 to one broker's IP instead of all three. Handshake hangs with no error MTU/fragmentation on the tunnel. Clamp MSS. Works on single-node, fails on multi-node The keystore only exists on the driver. Your init script is not attached, not running, or writing somewhere the executors cannot read. For the last one, check the init script logs directly. They land under the cluster's log destination in init_scripts/, and a script that failed with a non-zero exit will have prevented the node from starting at all, which shows up as workers repeatedly terminating during cluster launch. A useful isolation step: before touching Spark, run openssl s_client from a plain VM in the Databricks worker subnet. openssl s_client -connect b-1.mycluster.a1b2c3.c4.kafka.us-east-1.amazonaws.com:9094 \ -cert client-cert.pem -key client-key.pem -CAfile ca-chain.pem If that fails, nothing you do in Spark configuration will help. If it succeeds, the problem is on the Databricks side and you have halved the search space. Certificate rotation Client certificates expire, and a streaming job that has been running for eleven months will stop at an inconvenient moment. The mechanics are simple: issue the new certificate, rebuild the PKCS12, update the Key Vault secret, restart the cluster so the init script materializes the new keystore. The awkward part is that the Kafka client reads the keystore once at startup, so there is no reload without a restart. Plan for a brief gap in a continuously running stream, or run two consumers in the same group and roll them one at a time. Set a calendar reminder at 30 days before expiry, because certificate expiry is one of the few outages that is entirely predictable and still catches everyone. Reproducing this without an AWS bill You do not need MSK to practice any of this. The mTLS mechanics are identical against self-managed Kafka. Stand up a single broker with a self-signed CA, generate a client certificate from it, and point a local Spark job at it. Every failure mode in the table above reproduces, including the executor keystore problem if you run Spark in a small standalone cluster rather than local mode. The only pieces that genuinely need AWS are the ACM PCA association and MSK's specific ACL behavior. Everything else, including the debugging technique that matters most, you can learn for the price of a coffee. Configuration shown here was validated against Databricks Runtime 15.4 LTS and MSK 3.x. Broker DNS names, DNs, and topic names are illustrative.