After the Kubernetes cluster is created, you can add the Kafka-Connect component to the Kafka system.
Use the Calabash GUI to edit the Kafka infrastructure object named “lf-kafka.” Check the box “Deploy Kafka-Connect.”
Set the sizing parameters as above. And select the Kubernetes cluster “lf-cluster1.”
We must also expose the Kafka-Connect service to send requests to it to start or stop connectors.
Calabash offers some built-in connectors to support readers. Advanced users can also install custom connectors. What you need to do is to upload the Jar files of your custom connectors to cloud storage. Then you tell Calabash about this location. See below.
If you use JDBC-based readers or writers, you must provide the JDBC driver files that match the databases. You must upload the driver files to cloud storage. You then tell Calabash about the location using the “JDBC drivers URL” parameter.
Finally, some connectors may need to access cloud resources you do not have the access right to. For example, Google Bigquery writer needs to access a dataset. If the dataset is not in the project of the Kubernetes cluster, your access will fail. To gain access, you must provide the credentials from the target project, which is also called a “service account key.” You can download it as a JSON file from the target project. Then cut and paste the key to the “Credentials” text box. See above.
You do not need to worry about the security of the key. It will be encrypted. It is in plaintext only in the UI.
To learn how to download the service account key, please read Creating and Managing Service Account.
Click the Save button to persist the metadata.
To launch the Kafka-Connect in the Kafka system, just deploy the Kafka system again. The following is the transcript of the deployment.
Calabash (tester:lake_finance)> deploy i lf-kafka Deploy to cloud? [y]: Resetting metadata lake_finance__lf-kafka__num-zk to 2 Resetting metadata lake_finance__lf-kafka__num-bk to 2 Continue with Kafka-Connect deployment. Deploying new ms on k8s ... Deploying microservice lf-kafka in namespace lf-kafka ... Namespace created: lf-kafka Creating service ... Creating new service lf-kafka Service created: lf-kafka Checking service status ... retries left: 19 Checking service status ... retries left: 18 Creating metadata lake_finance__lf-kafka__svcip = 104.198.9.204 Creating new secret lf-kafka-secret Secret created: lf-kafka-secret Deployment created: lf-kafka Deployed 2/2 zookeeper nodes, 2/2 broker nodes, Kafka-Connect deployed to 104.198.9.204:8083