Elasticsearch connector

Connects to Elasticsearch server requested to perform one of the available actions

Pre-Requisites

Elasticsearch server running

Configuration

Common fields

These fields are used for all actions and work as follows

query: any type of query supported by elasticsearch

index: index where the query will be executed

servers: sever running to get the hits

connection: all related to the server connection

Scan Action:

Performs a search and returns the resulting documents based on all the parameters provided

Example seed configuration:

{
  "scan": {
    "query": {
      "match_all": {}
    },
    "idField": "name",
    "index": "seed",
    "scrollTimeout": "30",
    "servers": [
      {
        "host": "test-es",
        "port": "9200"
      }
    ],
    "connection": {
      "timeout": "30s"
    }
  },
  "name": "Test #1",
  "type": "elasticsearch-connector",
  "pipelineId": "ebcc6e40-8810-45d7-a348-7cfd519522df"
}

The resulting output will be the search's result with the provided parameters.

Configuration parameters

idField: a field that represents an id

scrollTimeout: scroll timeout in minutes

pageSize: Size of the page to scroll

Query action

Queries the Elasticsearch collection with an aggregation pipeline that will output a record per distinct field value with a total amount of repetitions.

Example seed configuration:

{
  "type": "elasticsearch-connector",
  "name": "Test #2",
  "config": {
    "scan": {
      "query": {
        "match_all": {}
      },
      "index": "people",
      "fields": [
        "name",
        "age"
      ],
      "servers": [
        {
          "host": "localhost",
          "port": "9200"
        }
      ],
      "connection": {
        "timeout": "30s"
      }
    }
  },
  "pipelineId": "8d0aee9b-0955-4118-ad08-b0b6e02135f6",
  "scanAction": "query"
}

Configuration parameters:

fields

(Required*, List) A list of strings for each field to group by. Fields cannot be blank.

Example

With an Elastic bulk such as the following:

{ "index":{} }
{ "name":"john doe", "age":25, "spoken_lang": ["eng", "es"], "employed": true}
{ "index":{} }
{ "name":"mary smith", "age":25, "spoken_lang": ["es"], "employed": true }

The resulting output will be as follows

{
  "token": "65aedd8a8b94b817604c37d3",
  "content": [
    {
      "_id": "49dd8bfb1db48f40c38aec131ccbfce0",
      "field": "name",
      "value": "john doe",
      "total": 1,
      "__pdp__": {
        "id": "49dd8bfb1db48f40c38aec131ccbfce0",
        "checksum": "9731935eb05616321948d10db9b36852",
        ...
      }
    },
    {
      "_id": "49dd8bfb1db48f40c38aec131ccbfce0",
      "field": "name",
      "value": "mary smith",
      "total": 1,
      "__pdp__": {
        "id": "49dd8bfb1db48f40c38aec131ccbfce0",
        "checksum": "9731935eb05616321948d10db9b36852",
        ...
      }
    },
    {
      "_id": "49dd8bfb1db48f40c38aec131ccbfce0",
      "field": "age",
      "value": "25",
      "total": 2,
      "__pdp__": {
        "id": "49dd8bfb1db48f40c38aec131ccbfce0",
        "checksum": "9731935eb05616321948d10db9b36852",
        ...
      }
    }
}