MapR Hadoop as deep storage for Druid cluster using mapr client

Following steps need to be followed if MapR Hadoop is deep storage:

Install and configure MapR client

• You need to install the MapR client that matches your MapR Hadoop installation on all middle manager/historical nodes.

.       https://mapr.com/docs/52/AdvancedInstallation/SettingUptheClient-install-mapr-client.html 

         https://mapr.com/docs/61/MapR-DB/Installing-mapr-client.html

• Run the configure.sh command for the MapR client

• Move imply/dist/druid/extensions/druid-hdfs-storage to a backup location
• Copy all mapr jars from /opt/mapr/lib to new directory imply/dist/druid/extensions/druid-hdfs-storage
• Copy druid-hdfs-storage-0.14.0-incubating-iap4.jar from the backup directory into the new druid-hdfs-storage directory. 

• rm dist/druid/extensions/druid-hdfs-storage/jackson-*.jar

• rm dist/druid/extensions/druid-hdfs-storage/guava-14.0.1.jar

• rm dist/druid/extensions/druid-hdfs-storage/joda-time-2.0.jar (if it exists)

Common.runtime.properties (modifications)

• druid-hdfs-storage should remain as an extension in the common.runtime.properties
• common.runtime.properties needs to include 

  druid.storage.type=hdfs
  
  druid.storage.storageDirectory=maprfs:///data/default-rack

• In your common.runtime.properties comment out following parameter ###druid.extensions.hadoopDependenciesDir=dist/druid/hadoop-dependencies
• In your common.runtime.properties set druid.extensions.useExtensionClassloaderFirst=true

         This can be placed after your druid.extension.loadList

Verification

• Verify connectivity from druid middle manager to maprfs hadoop cluster by running the following (you should get a list of directory contents). If this does not work most likely information was entered wrong for MapR client when you ran configure.sh.

           hadoop fs -ls /

Last step

• Restart all Imply processes on your middlemanagers/historical nodes

Reference

MapR Documentation