Skip to main content

Create connector source Kafka

Connect and ingest data from one or more Kafka topics of another Kafka cluster (which can be an external source or a different system).

Enables synchronization and forwarding of data between clusters.

Use case: Create a connector with Type: source, Database: Kafka

Pre-condition: CDC service status is Healthy

To create a connector, follow these steps:

Step 1: From the menu bar, select Data Platform > Workspace Management > Workspace name

Step 2: Under My services, select CDC service

Step 3: On the CDC service detail screen > Select the Connectors tab > Click Create a connector

create-connector

Step 4: Enter the connector information:

  • Name (required): connector name

Note: The connector name may contain lowercase letters a-z or digits 0-9. Spaces are not allowed; use "-" instead of a space.

  • Type (required): select source
  • Database (required): select Kafka

Step 5: Click Next to proceed to the Properties screen

There are two options: From FPT Database Engine, Manual configuration

  • When selecting Manual configuration

Enter and select the following information:

  • Cluster alias name: Alias name representing the Kafka cluster, containing only letters, digits, and "-"; used as a prefix for topic names when replicating data.
  • Bootstrap server endpoint: enter the Bootstrap server endpoint address
  • Security protocol: select one of the following security protocols:
  • SASL_PLAINTEXT: A simple authentication mechanism using Username and password
    • SASL Mechanism
    • SASL Username
    • SASL Password
  • SASL_SSL: Provides comprehensive security for authentication and data encryption via Username and password
    • SASL Mechanism1
    • SASL Username
    • SASL Password
  • PLAINTEXT: Data transmitted over the network is not encrypted; not recommended
  • SSL: A network security protocol used to protect data when transmitted over the Internet
  • When selecting From FPT Database Engine

Enter and select the following information:

  • Database Name (required): select Database
    • Cluster alias name: Alias name representing the Kafka cluster, containing only letters, digits, and "-"; used as a prefix for topic names when replicating data.
    • Bootstrap server endpoint: enter the Bootstrap server endpoint address
    • Security protocol: select one of the following security protocols:
    • SASL_PLAINTEXT: A simple authentication mechanism using Username and password
      • SASL Mechanism
      • SASL Username
      • SASL Password
    • SASL_SSL: Provides comprehensive security for authentication and data encryption via Username and password
      • SASL Mechanism
      • SASL Username
      • SASL Password
    • PLAINTEXT: Data transmitted over the network is not encrypted; not recommended
    • SSL: A network security protocol used to protect data when transmitted over the Internet
  • Kafka topic

Click the '+' button to retrieve topic information

Note: maximum of 100 topics can be retrieved

Click Test Connection to verify the connection from the Workspace to the Kafka Cluster

Step 6: Click Next to proceed to the Additional properties screen

Enter and select the following information:

  • Task:
    • Number of tasks: Enter the number of tasks for the connector. (This number should be less than or equal to the number of partitions in the Kafka topic.)
  • Replication:
  • Replication policy: Select the policy for replicating topics.
    • DefaultReplicationPolicy: Adds the source cluster alias prefix to the replicated topic name.

Example: topic-name becomes alias-name.topic-name

  • IdentityReplicationPolicy: Keeps the original topic name.

Example: topic-name remains topic-name

  • Replication factor: Enter the number of replicas for each Kafka topic partition.
    • Enter -1 to use the broker's default replication factor.

Step 7: Click Next to proceed to the Review screen

Step 8: Review the information and click Create to complete the connector creation.