Skip to main content

2.16 S3 Sink Connector

Create a connector where Type is sink and Database is S3

Pre-condition: CDC service status is Healthy

Steps to create a connector:​

Step 1: In the menu bar, select Data Platform > select Workspace Management > select Workspace name

Step 2: In the My services section, select CDC service

Step 3: On the CDC service detail screen > select the Connectors tab > click Create a connector

image-1

Step 4: Enter the information on the Connector Information screen:

  • Name (required): connector name

Note: The connector name can contain lowercase letters a-z or digits 0-9. In particular, spaces are not allowed — you can replace spaces with "-".

  • Type (required): select sink
  • Database (required): select S3

image-2

Step 5: Click Next in the top-right corner of the screen to proceed to the Properties screen

  • S3 Storage - Enter the S3 storage information
    • Endpoint: S3 storage access address
    • Access key: access key
    • Secret: access secret
    • Bucket name: bucket name
    • Path prefix: path to the folder in storage

image-3

Click Test Connection to verify the connection from the Workspace to the entered S3

  • Converter
    • Converter key: select the key value for the converter
    • Converter key schema enable: select whether or not to use a schema in the Converter key
    • Converter value: select the value for the converter
    • Converter value schema enable: select whether or not to use a schema in the Converter value

image-4

  • Kafka topic
    • Click the '+' to retrieve topic information
    • Note: limited to a maximum of 100 topics

image-5

Step 6. Click Next to proceed to the Additional Properties screen

  • Number of tasks: the maximum number of tasks that can run in parallel

image-6

  • Source:
    • Database type: select the source Database type
    • Timezone: select the timezone

image-7

  • File format: select the file format

image-8

  • Mode (required): The Connector's behavior when a message cannot be processed
    • None: The Connector will skip messages that cannot be sinked into the database
    • All: Failed messages will be sent to the specified topic

image-9

Step 7. Click Next in the top-right corner of the screen to proceed to the Review screen

image-9

Step 8. Review the information, then click Create to finish creating the connector