Overview of KCE

You can use a Kafka Client Environment (KCE) to manage a Kafka system deployed by Calabash. Depending on whether your Kafka brokers are private or public, you can use an internal KCE or a local KCE:

  • Internal KCE. An internal KCE is an infrastructure object deployed to a VM in the cloud. The KCE in the VM can access both private and public Kafka systems.
  • Local KCE. A local KCE is an environment you install on your personal computer, such as your laptop. It is a directory containing parameter files and shell scripts. In this environment, you can access a public Kafka system only, i.e., the public brokers.

This article describes the two types of KCE’s. The next article, the KCE command reference, explains all KCE commands available in both types.

1. Internal KCE

An internal KCE is an infrastructure object. You will need to use Calabash GUI to design it first. Read Create an Internal Kafka Client Environment (KCE) for how to design an internal KCE.

After designing an internal KCE, you will need to deploy it to the cloud using Calabash CLI. Read the Calabash CLI User Manual for how to deploy an object.

To use the deployed internal KCE, you can log into the VM with SSH in a command window. The command for doing this is similar to the following.

gcloud compute ssh my-kce-vm --zone us-central1-a

where “my-kce-vm” is the VM name for the internal KCE. It is normally the same name as the KCE object in Calabash GUI. In your situation, your zone may be different from the one above.

Once logged in the SSH, find the running containers with this command:

docker ps

You may see a display like this:

CONTAINER ID   IMAGE                                COMMAND         CREATED        STATUS        PORTS      NAMES
bfaf89a9702c   gcr.io/dlb-public-245218/kce:2.0.9   "/app/run.sh"   46 hours ago   Up 46 hours              stoic_sanderson

The container id is important. You use it to get into the container with the following command.

docker exec -it bfaf89a9702c /bin/bash

Once inside the container,

cd /app/bin

You have now arrived at the internal KCE scripts directory. Your environment has been configured to access your Kafka system.

2. Local KCE

A local KCE is a directory created on your local computer. It contains Kafka client property files, SSL keys and certificates, and the KCE scripts.

To set up the local KCE, use the “set-up-kce” command in Calabash CLI. Read Set up Local Kafka Client Environment (KCE) in the Calabash CLI manual.

Suppose you have set up the local KCE in a directory “/user/home/jdoe/calabash” on your laptop, then,

% cd /user/home/jdoe/calabash/bin

You have arrived at the place where you can run scripts to manage your Kafka system.