Skip to main content

Create a ClickHouse (logs) sink connector

Use a ClickHouse (logs) sink connector to keep the full history of the changes a source connector captures in ClickHouse, for analysis or audit. The connector adds every change from the Kafka topics you select as a new row in an existing table, while a ClickHouse (Replication) sink connector keeps only the current state of each row.

Prerequisites​

  • A CDC Service whose status is Healthy. See Create a CDC Service.
  • Topics with change data, written by a source connector such as a MariaDB source connector.
  • A ClickHouse database that the workspace can reach, and a user with the SELECT and INSERT privileges on the target tables.
  • The target tables, created beforehand with columns that match the flattened change events, such as before.id, after.id, op, and ts_ms. The connector does not create tables.

Create the connector​

  1. Select Data platform → Workspace management, then select your workspace.

  2. On the Overview tab, under Services, select the CDC card.

  3. Select the Connectors tab, then select Create Connector.

    Connectors tab of a CDC Service in Healthy status, with no connector yet and the Create Connector button

  4. On the Connector Information screen, enter a Name, select sink in Type, and select ClickHouse (Logs) in Database.

    Connector Information screen of the Create Connector form with the Name, Type, and Database fields

  5. Select Next Step. On the Properties screen, enter the connection details, then select Test connection. Next Step becomes available after the test succeeds. If the test fails with "Connection failed due to invalid information, permission or network failure.", check the connection details and the user's privileges, then test again.

    Database Information section with Manual configuration selected and the Host name, Port, Database, Username, and Password fields

    Database Information section with From FPT Database Engine selected and the Database Name list above the connection fields

  6. On the same screen, choose a Converter class and set the four converter fields, then select the topics to read in Topics.

    Converter section with the Converter class, Converter key, Converter value, and schema enable lists

    Kafka topic section with the Topics list

  7. Select Next Step. On the Additional properties screen, set the fields in the Task, Map table, and Error Tolerance sections.

    Map table section with a topic in Topic 1 and the Table 1 list

  8. Select Next Step and check the settings on the Review screen.

    Review screen with the Basic Information, Database Information, Kafka topic, Task, and Error Tolerance sections and the Create button

  9. Select Create.

Connector settings​

Connector information​

FieldRequiredDescription
NameYesThe connector name. Use lowercase letters, digits, and hyphens (-) only. Upper-case letters, _, ., and spaces are rejected with "Name has invalid characters".
TypeYesSelect sink.
DatabaseYesSelect ClickHouse (Logs).

Properties​

In the Database Information section, choose From FPT Database Engine or Manual configuration. Manual configuration is selected by default. With From FPT Database Engine, select your instance in Database Name to fill in Host name and Port. See Connection mode.

FieldRequiredDescription
Database NameYesShown with From FPT Database Engine only. The ClickHouse instance in FPT Database Engine. Enter the database to write to in Database.
Host nameYesThe host name or IP address of the ClickHouse server.
PortYesThe port of the ClickHouse server, for example 8123. The field has no default value in Manual configuration.
DatabaseYesThe database that the connector writes to.
UsernameYesThe user that the connector connects as.
PasswordYesThe password of that user.
Converter classYesThe converter class for this connector. The portal offers Clickhouse log. The four converter fields stay greyed out until you choose a class. See Converter.
Converter key, Converter key schema enable, Converter value, Converter value schema enableYesHow the connector reads the keys and values of the messages, and whether they carry a schema. With Clickhouse log, Converter key offers only org.apache.kafka.connect.storage.StringConverter and Converter key schema enable only true. Converter value offers only org.apache.kafka.connect.json.JsonConverter, and Converter value schema enable true or false.
Topics (under Kafka topic)YesThe topics to read. Select them from the list, which shows the topics in the Kafka cluster of your CDC Service. See Topics.

Additional properties​

FieldRequiredDescription
Tasks maxYesThe maximum number of tasks for the connector. Default 1. See Tasks max.
Topic 1, Table 1 (under Map table)YesThe ClickHouse table that receives the records of each topic. Select an existing table from the list. If the list shows "No options", Next Step gives "Table is required": create the table in ClickHouse first, then select it. See Map table.
Mode (under Error Tolerance)YesWhat the connector does with a message that it cannot read or convert. None stops the connector with status Failed. All skips the message and sends it to the topic you enter for failed records. The field has no default. See Error tolerance.

Next steps​