How to run native batch ingestion using GCP

Objective:

Step by step guide to configure Druid to ingest files from Google Cloud Platform.

Step 1: Login to GCP console:

Go to GCP console page https://console.cloud.google.com/, and log in with your credential.

Step 2: Create a service account that will access the bucket:

Browse to "IAM & Admin" -> "Service Accounts" -> "Create Service Account", Create a service account, in this example "imply-cs"

At permission section of the "imply-cs", make sure your user name is listed as one of the owners

Click on "Create key"

Download the JSON file that created, and save for later use. That's the authentication/authorization credential for your Druid hosts to access GCP.

Step 3: Apply the bucket access permission to the service account just created:

Browse to your GCP bucket through Google navigation menu -> storage -> browser -> YOUR BUCKET ("cs-bucket1" in this example)

Configure user access permission on the bucket

Add the service account as Storage Admin to this bucket.

Step 4: Configure environmental variable on all Druid hosts

Open SSH sessions to all druid hosts, and save the JSON file downloaded in step 2 to Imply Druid's conf (./conf) directory on all hosts. Then in your bash properties file, e.g, '~/.bash_properties' on RHEL/CentOS, create a new environment variable GOOGLE_APPLICATION_CREDENTIALS. Replace [PATH] with the file path of the JSON file that contains your service account key, and [FILE_NAME] with the filename. For example:

[root@ip-172-31-2-115 ec2-user]# cat ~/.bash_profile                                                                                                 
# .bash_profile
export GOOGLE_APPLICATION_CREDENTIALS="/imply-<VERSION>/conf/googleauth.json"

  [root@ip-172-31-2-115 ec2-user]# cat /imply-<VERSION>/conf/googleauth.json                                                                                     
{
  "type": "service_account",
  "project_id": "<ID>",
  "private_key_id": "<KEY_ID>",
  "private_key": "-----BEGIN PRIVATE KEY-----<KEY>-----END PRIVATE KEY-----\n",
  "client_email": "imply-cs@<ID>.iam.gserviceaccount.com",
  "client_id": "<CLIENT_ID>",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "<URL>"
}

Step 5: Add "Druid-google-extensions" to the Druid's "druid.extensions.loadList" in the Druid's "common.running.properties".

Step 6: Add following to replace the current deep storage config in "common.running.properties"

 
druid.storage.type=google                                                                                                                           
druid.google.bucket=cs-bucket1                                                                                                                      
druid.google.prefix=druid/segments

Step 7: Add following to replace the current index logging config in "common.running.properties"

druid.indexer.logs.type=google                                                                                                                      
druid.indexer.logs.bucket=cs-bucket1                                                                                                                
druid.indexer.logs.prefix=druid/indexing-logs

Step 8: restart all Druid services.

Step 9: Upload raw data file "wikipedia-2016-06-27-sampled.json" to GCP bucket "cs-bucket1".

Step 10: POST native batch ingestion to start ingestion:

{
  "type": "index",
  "spec": {
    "dataSchema": {
      "dataSource": "wikipedia",
      "parser": {
        "type": "string",
        "parseSpec": {
          "format": "json",
          "dimensionsSpec": {
            "dimensions": [
              "isRobot",
              "diffUrl",
              {
                "name": "added",
                "type": "long"
              },
              "channel",
              "flags",
              {
                "name": "delta",
                "type": "long"
              },
              "isUnpatrolled",
              "isNew",
              {
                "name": "deltaBucket",
                "type": "long"
              },
              "isMinor",
              "isAnonymous",
              {
                "name": "deleted",
                "type": "long"
              },
              "namespace",
              "comment",
              "page",
              {
                "name": "commentLength",
                "type": "long"
              },
              "user",
              "countryIsoCode",
              "regionName",
              "cityName",
              "countryName",
              "regionIsoCode",
              {
                "name": "metroCode",
                "type": "long"
              }
            ]
          },
          "timestampSpec": {
            "column": "timestamp",
            "format": "iso"
          }
        }
      },
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "DAY",
        "rollup": false,
        "queryGranularity": "none"
      },
      "metricsSpec": []
    },
    "ioConfig": {
      "type": "index",
      "firehose": {
        "type":"static-google-blobstore",
        "blobs”: [
        {
          "bucket":"cs-bucket1",
          "path":"wikipedia-2016-06-27-sampled.json"
        } ]
      },
      "appendToExisting": false
    },
    "tuningConfig": {
      "type": "index",
      "forceExtendableShardSpecs": true,
      "maxRowsInMemory": 1000000,
      "reportParseExceptions": false,
      "maxParseExceptions": 100,
      "maxSavedParseExceptions": 10
    }
  }
}