Connects to AWS S3 service

Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading scalability, data availability, security, and performance. Customers of all sizes and industries can store and protect any amount of data for virtually any use case, such as data lakes, cloud-native applications, and mobile apps. With cost-effective storage classes and easy-to-use management features, you can optimize costs, organize data, and configure fine-tuned access controls to meet specific business, organizational, and compliance requirements. AWS S3

Pre-Requisites

The bucket that will be looked up must exist on the S3 bucket

Configuration

Example configuration:

{
  "type": "s3-connector",
  "name": "{Connector name}",
  "description": null,
  "labels": {},
  "active": true,
  "config": {
    "pull": {
      "metadata": true,
      "onlyContent": false,
      "key": "/filename"
    },
    "client": {
      "connection": {
        "timeout": 60000
      },
      "socket": {
        "timeout": 3600000
      }
    },
    "aws": {
      "bucket": "{bucket name}",
      "region": "{region name with format us-east-1}"
    }
  },
  "credentialId": "{credential id}",
  "processErroredJobs": false,
  "processErroredRecords": false,
  "recordDataStrategy": null
}

Configuration

aws

(Required, JSON) A JSONObject with information about source bucket and region

For example:

{
  "bucket": "{bucket name}",
  "region": "{region name with format US_EAST_1}"
}

Bucket name for S3 Region name for S3 connection, the expected format is us-east-1

client

(Optional, JSON) A JSONObject with information about the connection. For example: timeout.

For example:

{
  "connection": {
    "timeout": 60000
  },
  "socket": {
    "timeout": 3600000
  }
}

pull

(Required, JSON) A JSONObject with information about data that we want to ingest

For example:

{
  "metadata": true,
  "onlyContent": false,
  "key": "/filename"
}

key is a pointer to the data node with the data to retrieve