Skip to main content

Create an S3 source connector

Load the records of files in a bucket into a Kafka topic, so that sink connectors and pipelines can use them like any other stream. The connector reads, scans, and ingests files such as CSV, TSV, Avro, or XML from Amazon S3 or S3-compatible storage like FPT Object Storage or MinIO.

Prerequisites​

  • A CDC Service whose status is Healthy. See Create a CDC Service.
  • The endpoint URL, bucket name, and path of the source files, and an access key and secret key that can read them.

Create the connector​

  1. Select Data platform → Workspace management, then select your workspace.

  2. On the Overview tab, under Services, select the CDC card.

  3. Select the Connectors tab, then select Create Connector.

    Connectors tab of a Healthy CDC Service with the No connector yet message and the Create Connector button

  4. On the Connector Information screen, enter a Name, select source in Type, and select S3 in Database.

    Connector Information screen of the Create Connector form with the Name, Type, and Database fields

  5. Select Next Step. On the Properties screen, enter the S3 details and the Topic, then select Test connection. Next Step becomes available after the test succeeds.

    If a field is empty, Test connection marks it with a message such as "Access Key is required" and does not test the connection.

    S3 Information section with the URL, Access Key, Secret key, Bucket Name, and Path Prefix fields

    Kafka topic section with the Topic field

  6. Select Next Step. On the Additional properties screen, set how the connector reads the files and the error tolerance Mode.

    Reader config section with ROW in Type, a csv pattern in File filter regex pattern, User provider in Header definition, a comma in Delimiter, and the Trim value and Column name fields

    Error Tolerance section with the Mode list

  7. Select Next Step and check the settings on the Review screen.

    Review screen of an S3 source connector with the Basic Information, S3 Information, Reader config, and Error Tolerance sections

  8. Select Create.

    The portal shows "Create Successfully" and opens the Essential Information tab of the CDC Service. On the Connectors tab, the new connector shows Running, with every Tasks counter at 0 until its task starts, for up to about 2.5 minutes.

Connector settings​

Connector information​

FieldRequiredDescription
NameYesThe connector name, in lowercase letters, digits, and hyphens (-). Uppercase letters, _, ., and spaces give the error "Name has invalid characters".
TypeYesSelect source.
DatabaseYesSelect S3.

Properties​

FieldRequiredDescription
URLYesThe endpoint URL of the S3 storage, for example https://YOUR_ENDPOINT. The hint "Only letters, numbers and hyphens are allowed" under the field does not apply to this field.
Access KeyYesThe access key.
Secret keyYesThe secret key that goes with the access key.
Bucket NameYesThe bucket that holds the source files.
Path PrefixYesThe folder that holds the source files. Write it without a leading slash and end it with a slash, for example data/orders/. Otherwise the field shows the error "Path Prefix has invalid characters".
TopicYesThe full name of the Kafka topic that the connector writes all file records to. It is not a prefix.

Test connection checks that the connector can reach the bucket with the details you entered.

Additional properties​

FieldRequiredDescription
TypeYesThe file format to read, for example ROW (CSV, TSV), XML, or Avro.
File filter regex patternYesA regular expression that the whole file name must match. Type a single backslash before the dot: .*\.csv$ accepts only files that end in .csv. A pattern that covers only part of the name, such as \.csv$, matches no file.
Header definitionYesWhere the column names come from. From file takes them from the first line of the file. Autogenerated names them column1, column2, and so on. User provider uses the names you enter in Column name (the label means "user provided").
DelimiterYesThe character between columns. Default ,. You can change it, for example to a tab or a semicolon.
Trim valueYesYes removes spaces at the start and end of each column value. No keeps them.
Column nameWith User providerThe columns of the file. Enter each column as NAME:TYPE, for example id:STRING, then select + to add it as a tag. Every column needs a type. Shown only when Header definition is User provider.
Mode (Error Tolerance)YesWhat the connector does when a record fails. None stops the connector with the status Failed. All skips the records that fail and keeps running. The field has no default. See Error tolerance.

Next steps​