Create a MySQL source connector
Stream every row change in your MySQL databases to Kafka topics, where sink connectors can deliver it to other systems. The connector reads the MySQL binary log (binlog), and can start with a consistent snapshot of existing data because MySQL purges old binlog files.
Prerequisites
- A CDC Service whose status is Healthy. See Create a CDC Service.
- A supported MySQL topology:
- Standalone, with the binlog enabled.
- Primary and replica. The connector reads the binlog of one server and sees only the changes on that server.
- High availability.
- The database prepared for CDC as described in Prepare the MySQL database.
Prepare the MySQL database
On a MySQL database from FPT Cloud, the binlog, GTIDs, and binlog_row_value_options are already set up. Run steps 1 to 3 and skip steps 4 to 8.
-
Create the user that the connector connects as.
CREATE USER 'YOUR_USER'@'%' IDENTIFIED BY 'YOUR_PASSWORD'; -
Grant the user
SELECTon the databases to capture, and the global privilegesRELOAD,SHOW DATABASES,REPLICATION SLAVE, andREPLICATION CLIENT.GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'YOUR_USER'@'%';FLUSH PRIVILEGES;To limit
SELECTto one database, run instead:GRANT SHOW DATABASES, RELOAD, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'YOUR_USER'@'%';GRANT SELECT ON YOUR_DATABASE.* TO 'YOUR_USER'@'%';FLUSH PRIVILEGES; -
Grant the user
INSERTandCREATEon the captured database if you plan to select Enable incremental snapshot. Creating the connector then creates the signal tablecdc_signalin that database, and snapshots run through it.GRANT INSERT, CREATE ON YOUR_DATABASE.* TO 'YOUR_USER'@'%';FLUSH PRIVILEGES; -
Check whether the binlog is on.
SHOW GLOBAL VARIABLES LIKE 'log_bin';-- Or, on MySQL 8.xSELECT variable_value FROM performance_schema.global_variables WHERE variable_name = 'log_bin';-- Or, on MySQL 5.xSELECT variable_value FROM information_schema.global_variables WHERE variable_name = 'log_bin'; -
Add these lines to the MySQL configuration file if
log_binisOFF. Setserver-idto the result ofSHOW VARIABLES LIKE 'server_id';.server-id = YOUR_SERVER_IDlog_bin = mysql-binbinlog_format = ROWbinlog_row_image = FULLbinlog_expire_logs_seconds = 864000If the binlog is already on and you only need to change its format, you can run these statements on the running server instead.
SETcannot turn onlog_bin, and the new values last only until the next restart unless you also add them to the configuration file.SET @@global.binlog_format = 'ROW';SET @@global.binlog_row_image = 'FULL';SET @@global.binlog_expire_logs_seconds = 864000; -
Turn on GTIDs. First check both variables.
SHOW GLOBAL VARIABLES LIKE 'gtid_mode';SHOW GLOBAL VARIABLES LIKE 'enforce_gtid_consistency';If both are
OFF, set them in the MySQL configuration file:gtid_mode = ONenforce_gtid_consistency = ONOr set them on a running server. MySQL changes
gtid_modeone step at a time.SET @@global.enforce_gtid_consistency = ON;SET @@global.gtid_mode = OFF_PERMISSIVE;SET @@global.gtid_mode = ON_PERMISSIVE;SET @@global.gtid_mode = ON; -
Restart MySQL so that the changes to the configuration file in steps 5 and 6 take effect. Skip this step if you did not change the configuration file.
warningRestarting MySQL interrupts the applications that use it. Plan the restart for a maintenance window.
-
Set
binlog_row_value_optionsto an empty value, so that the connector can captureUPDATEevents.SHOW GLOBAL VARIABLES LIKE 'binlog_row_value_options';SET @@global.binlog_row_value_options = '';
Create the connector
-
Select Data platform → Workspace management, then select your workspace.
-
On the Overview tab, under Services, select the CDC card.
-
Select the Connectors tab, then select Create Connector.

-
On the Connector Information screen, enter a Name, select source in Type, and select MySQL in Database.

-
Select Next Step. On the Properties screen, enter the connection details, the Database server ID, and the Topic prefix, then select Test connection. Next Step becomes available after the test succeeds with the message "Connect Successfully".



-
Select Next Step. On the Additional properties screen, select the snapshot Mode, the tables and columns to capture, and the Heartbeat interval (ms).



-
Select Next Step and check the settings on the Review screen.

-
Select Create.
The portal shows "Create Successfully" and opens the Essential Information tab of the CDC Service. On the Connectors tab, the new connector shows Running, with every Tasks counter at
0until its task starts, after about 2 minutes. Snapshot Status stays Not Started, because it tracks incremental snapshots only.
Connector settings
Connector information
| Field | Required | Description |
|---|---|---|
| Name | Yes | The connector name, in lowercase letters, digits, and hyphens (-). Uppercase letters, _, ., and spaces give the error "Name has invalid characters". |
| Type | Yes | Select source. |
| Database | Yes | Select MySQL. |
Properties
| Field | Required | Description |
|---|---|---|
| From FPT Database Engine / Manual configuration | Yes | Select a MySQL instance from FPT Database Engine, or enter the connection details yourself. Manual configuration is selected by default. See Connection mode. |
| Database Name | With From FPT Database Engine | The MySQL instance in FPT Database Engine. The list shows instance names only, and two instances can have the same name. Selecting one fills in Host name and Port (3306), which you can still edit. Database, Username, Password, and Database server ID stay empty. |
| Host name | Yes | Host name or IP address of the MySQL server. |
| Port | Yes | The MySQL server port, for example 3306. The field has no default in Manual configuration. |
| Database | Yes | The database to capture changes from. |
| Username | Yes | The user you created in step 1. |
| Password | Yes | The user's password. |
| Database server ID | Yes | A number that the connector uses as its own ID when it reads the binlog as a replication client. It must be unique among the replicas and other replication clients of the MySQL server, and must differ from the server's own server_id. The field has no default. Enter a number from 1000 to 9999. |
| Enable incremental snapshot | No | Lets you run incremental snapshots on this connector. Creating the connector then also creates the signal table cdc_signal in the Database you entered, and Test connection checks the CREATE TABLE and INSERT permissions. Without this option there is no signal table, so Snapshot → Execute cannot run, although the menu still shows it. See Incremental snapshot. |
| Topic prefix | Yes | Prefix of the Kafka topics the connector writes to. See Topic prefix. |
Additional properties
| Field | Required | Description |
|---|---|---|
| Mode | Yes | What the connector does with existing rows. The field has no default, so select one. initial snapshots them, then captures changes. initial_only snapshots them without capturing changes. no_data captures new changes only. See Snapshot mode. |
| Table (Optional) | No | The tables to capture, from the database on the Properties screen. You can select several. |
| Column (Optional) | No | The columns to capture from the selected table. The field appears after you select a table, and its label shows the table name, for example Column (Optional) - (orders). Leave it empty to capture all columns. |
| Heartbeat interval (ms) | Yes | Time between heartbeat messages, from 0 to 30000. Default 0, which sends no heartbeat. See Heartbeat. |