Elasticsearch Hydrator

This hydrator is in charge of hydrating records to Elasticsearch.

Configuration

Example configuration in a processor:

{
  "name": "Elastic Hydrator",
  "type": "elasticsearch-hydrator",
  "index": "index1",
  "partialUpdate": false,
  "credentialId": "c1f9f0b2-b999-44ea-acd6-0020be2fb007",
  "idFieldPath": "/idFieldPath",
  "servers": [
    {
      "host": "elastic",
      "port": 9200
    }
  ],
  "connection": {
    "connectTimeout": "10s",
    "readTimeout": "10s",
    "pool": {
      "size": 1,
      "keepAlive": "10s"
    }
  },
  "bulk": {
    "actions": 100,
    "dataSize": "5mb",
    "backoffPolicy": {
      "type": "constant",
      "initialDelay": "2m",
      "retries": 5
    }
  }
}

Configuration parameters:

index

(Required, String) index to hydrate data into Elasticsearch.

credentialId

(Optional, String) identifier for credentials to authenticate with when connecting to Elasticsearch.

idFieldPath

(Optional, String) Path to a field within the document to be used as the id of the document when hydrating. If not provided, default auto-generated id will be used.

servers.host

(Required, String) host where Elasticsearch is located.

servers.port

(Required, Int) host port where Elasticsearch is located.

connection.connectTimeout

(Optional, String) The duration for the connection timeout. Default: 60s.

connection.readTimeout

(Optional, String) The duration for the read timeout. Default: 60s.

connection.pool.size

(Optional, Integer) The size of the connection pool. Default: 5.

connection.pool.keepAlive

(Optional, String) The duration of the connection in the pool. Default: 5m.

bulk.actions

(Optional, Integer) Number of actions to run for the bulk operation. Default: 1000.

bulk.dataSize

(Optional, String) Size of the actions to flush the bulk request. Default: 5mb.

backoffPolicy.type

(Optional, String) Type of backoff policy to apply. Available Options: NONE, CONSTANT, EXPONENTIAL Default: NONE.

backoffPolicy.initialDelay

(Optional, String) The initial delay before retrying.

backoffPolicy.retries

(Optional, Integer) The maximum number of retries before exhausting.

partialUpdate

(Optional, boolean) to determine if documents will be indexed with partial update support. Useful when adding extra fields to an already indexed document.