Skip to main content

Create an OpenSearch sink connector

Use an OpenSearch sink connector to make the changes captured from your source database searchable in OpenSearch. The connector reads the Kafka topics you choose and writes each topic to an OpenSearch index.

Prerequisites​

  • A CDC Service whose status is Healthy. See Create a CDC Service.
  • Topics with change data, written by a source connector such as a PostgreSQL source connector.
  • The connection string, username, and password of your OpenSearch database.

Create the connector​

  1. Select Data platform → Workspace management, then select your workspace.

  2. On the Overview tab, under Services, select the CDC card.

  3. Select the Connectors tab, then select Create Connector.

    Connectors tab of a CDC Service with no connector yet and the Create Connector button

  4. On the Connector Information screen, enter a Name, select sink in Type, and select OpenSearch in Database.

    Connector Information screen of the Create Connector form with the Name, Type, and Database fields

  5. Select Next Step. On the Properties screen, enter the connection details, then select Test connection. Next Step becomes available after the test succeeds. If the test fails with "Connection failed due to network failure or invalid information.", check the connection string, the username, and the password, then test again.

    Database Information section with Manual configuration selected and the Connection string, Username, and Password fields

    Database Information section with From FPT Database Engine selected and the Database Name, Connection string, Username, and Password fields

  6. On the same screen, choose a Converter class and set the four converter fields, then select the topics to read in Topics.

    Converter section with the Converter class, Converter key, Converter value, and schema enable fields

    Kafka topic section with the Topics list

  7. Select Next Step. On the Additional properties screen, set the fields in the Task, Map table, and Error Tolerance sections.

    Task section with Tasks max set to 1 and the Data stream enable switch set to Disable

    Map table section with Topic 1, the Create new table switch, and Table 1

    Error Tolerance section with the Mode list

  8. Select Next Step and check the settings on the Review screen.

    Review screen with the Basic Information, Database Information, Kafka topic, Task, Converter, and Error Tolerance of the new connector

  9. Select Create.

    The portal returns to the Essential Information tab of the CDC Service. On the Connectors tab, the new connector shows the status Running, with every Tasks counter at 0 until its task starts, which can take up to about 2.5 minutes.

Connector settings​

Connector information​

FieldRequiredDescription
NameYesThe connector name. Use lowercase letters, digits, and hyphens (-) only. Upper-case letters, _, ., and spaces are rejected with "Name has invalid characters".
TypeYesSelect sink.
DatabaseYesSelect OpenSearch.

Properties​

In the Database Information section, choose From FPT Database Engine or Manual configuration. Manual configuration is selected by default. With From FPT Database Engine, select your instance in Database Name. The portal fills Connection string as http://HOST:9200, which you cannot edit. See Connection mode.

FieldRequiredDescription
Database NameYesShown with From FPT Database Engine only. The OpenSearch instance in FPT Database Engine.
Connection stringYesThe connection URI or host name of your OpenSearch database.
UsernameYesThe username the connector uses to connect to OpenSearch.
PasswordYesThe password for that user.
Converter classYesThe converter class for this connector. The portal offers Opensearch. The four converter fields stay greyed out until you choose a class. See Converter.
Converter key, Converter key schema enable, Converter value, Converter value schema enableYesHow the connector reads the keys and values of the messages, and whether they carry a schema. Converter key offers org.apache.kafka.connect.storage.StringConverter and org.apache.kafka.connect.json.JsonConverter, and Converter value only org.apache.kafka.connect.json.JsonConverter. Both schema enable fields offer true and false.
Topics (under Kafka topic)YesThe topics to read. Select them from the list, which shows the topics in the Kafka cluster of your CDC Service. See Topics.

Additional properties​

FieldRequiredDescription
Tasks maxYesThe number of tasks the connector runs at the same time when the topics have more than one partition. Above 1, records can be consumed out of order, so give related records the same key to keep them in one partition. Default 1. See Tasks max.
Data stream enableNoTurn on to write to a data stream instead of a regular index. Data streams suit time-series and log data: they only append documents, and each record needs a @timestamp field. Default Disable.
Topic 1, Create new table, Table 1 (under Map table)YesThe OpenSearch index that receives the records of each topic. Despite its label, Table 1 takes an index name. Create new table is off by default, and Table 1 lists the existing indices, including system indices whose names start with .. Do not select a system index. Turn on Create new table to type a new index name instead. Table 1 is then empty. See Map table.
Mode (under Error Tolerance)YesWhat the connector does with a message that it cannot read or convert. None stops the connector with status Failed. All skips the message and sends it to the topic you enter for failed records. The field has no default. See Error tolerance.

Next steps​