Kerberized Kafka Ingestion

For an overview of Kafka security features and guidance for setting up Kerberized Kafka, please see:

https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/

In order for Druid to connect with Kerberized Kafka, a JAAS conf file needs to be set up on each service. There are two alternate methods of authentication from within the JAAS conf file:

1. setting useTicketCache=true

2. create keytab based authentication. We provide an example of this below.

1.  Create keytab for Druid

Where <druid-user> is the user that runs the Druid services:

addprinc <druid-user>@EXAMPLE.COM

ktadd -k /keytab/<druid-user>.keytab <druid-user>@{REALM}

2. Create JAAS conf file for Druid as Kafka Client:

/path/to/client_jaas.conf

*Note: The Client section is only necessary if ZK is Kerberized

KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useTicketCache=false
    useKeyTab=true
    storeKey=true
    keyTab="/keytab/<druid-user>.keytab"
    principal="<druid-user>@EXAMPLE.COM";
};
Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useTicketCache=false
    useKeyTab=true
    storeKey=true
    keyTab="/keytab/<druid-user>.keytab"
    principal="<druid-user>@EXAMPLE.COM"; 
};

3. Edit Druid properties

 In the jvm.config for each service, add the path to the JAAS conf file:

-Djava.security.auth.login.config=/client_jaas.conf

In conf/middleManager/runtime.properties(or conf-quickstart if running quickstart), add to druid.indexer.runner.javaOpts:

-Djava.security.auth.login.config=/client_jaas.conf

In conf/druid/_common/common.runtime.properties (or conf-quickstart if running quickstart) - NOTE: this part is only necessary if ZK is Kerberized:

druid.zk.service.host={host}.example.com 

Note: This procedure was tested with Imply 2.6.8 and Kafka kafka_2.11-2.0.0.