Track 2 Topic 3: Create a KCE

The KCE is in a container running on the KCE VM. To connect to the container, you first connect to the VM: (In the following the boldface statements are the commands we enter.)

% gcloud compute ssh lf-kce --zone us-central1-f

  ########################[ Welcome ]########################
  #  You have logged in to the guest OS.                    #
  #  To access your containers use 'docker attach' command  #
  ###########################################################
                                                             
user1@lf-kce ~ $ docker ps
CONTAINER ID   IMAGE                                 COMMAND             CREATED              STATUS              PORTS     NAMES
cd7249af46f6   gcr.io/dlb-public-245218/kce:3.0.9    "/app/run.sh"       About a minute ago   Up About a minute             klt-lf-kce-sesn

You can see the container is running with an id “cd7249af46f6.” Now attach to it using this id and run a shell:

user1@lf-kce ~ $ docker exec -it cd7249af46f6 /bin/bash
root@lf-kce:/# 

Now we are in the KCE and we are ready to issue management commands.

All the command scripts are in directory /app/bin:

root@lf-kce:/# cd /app/bin
root@lf-kce:/app/bin# ls
connector_config.sh		create_topic.sh			list_acls.sh
connector_pause.sh		delete_acl_for_consumer.sh	list_connectors.sh
connector_restart.sh		delete_acl_for_groups.sh	list_consumer_groups.sh
connector_resume.sh		delete_acl_for_producer.sh	list_plugins.sh
connector_status.sh		delete_acl_for_topic.sh		list_reader_offsets.sh
consume.sh			delete_acl_for_txid.sh		list_topics.sh
create_acl_for_consumer.sh	delete_connector.sh		produce.sh
create_acl_for_groups.sh	delete_consumer_group.sh	set_connector_config.sh
create_acl_for_producer.sh	delete_topic.sh			set_consumer_offset.sh
create_acl_for_topic.sh		desc_consumer_group.sh		set_reader_offset.sh
create_acl_for_txid.sh		desc_topic.sh
create_connector.sh		env.sh
root@lf-kce:/app/bin#

Next, we create a topic:

root@lf-kce:/app/bin# ./create_topic.sh jdoe log1
Created topic log1.
root@lf-kce:/app/bin# ./list_topics.sh jdoe
log1
root@lf-kce:/app/bin#

Both commands require a superuser name. In our Kafka system “lf-kafka,” the superuser is “jdoe.” The KCE environment has created a Kafka client parameter file and the TLS (SSL) files for user jdoe to function as an admin. So you only need to provide the superuser name to the commands above.

Next, we assign producer permissions to user “jeff.”

root@lf-kce:/app/bin# ./create_acl_for_producer.sh jdoe jeff log1
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=log1, patternType=LITERAL)`: 
 	(principal=User:jeff, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:jeff, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:jeff, host=*, operation=CREATE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=log1, patternType=LITERAL)`: 
 	(principal=User:jeff, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:jeff, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:jeff, host=*, operation=CREATE, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TRANSACTIONAL_ID, name=*, patternType=LITERAL)`: 
 	(principal=User:jeff, host=*, operation=ALL, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TRANSACTIONAL_ID, name=*, patternType=LITERAL)`: 
 	(principal=User:jeff, host=*, operation=ALL, permissionType=ALLOW)

Here “acl” stands for “Access Control List.” The ACL mechanism is how Kafka manages permissions.

The output is a little wordy. Please take a moment to fully understand it. Basically, user “jeff” receives permissions to write to the topic “log1.” He can do so from any host and transactionally, i.e., no loss and no duplicate, and no partial record writes.

Similarly, we assign consumer permissions to the user “april.”

root@lf-kce:/app/bin# ./create_acl_for_producer.sh jdoe april log1
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=log1, patternType=LITERAL)`: 
 	(principal=User:april, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:april, host=*, operation=CREATE, permissionType=ALLOW)
	(principal=User:april, host=*, operation=WRITE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=log1, patternType=LITERAL)`: 
 	(principal=User:april, host=*, operation=CREATE, permissionType=ALLOW)
	(principal=User:jeff, host=*, operation=CREATE, permissionType=ALLOW)
	(principal=User:april, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:jeff, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:jeff, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:april, host=*, operation=WRITE, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TRANSACTIONAL_ID, name=*, patternType=LITERAL)`: 
 	(principal=User:april, host=*, operation=ALL, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TRANSACTIONAL_ID, name=*, patternType=LITERAL)`: 
 	(principal=User:jeff, host=*, operation=ALL, permissionType=ALLOW)
	(principal=User:april, host=*, operation=ALL, permissionType=ALLOW)

At this point, you may have a question: didn’t we declare “jeff” as a producer and “april” as a consumer when we designed the KCE? Why do we still need to assign them permissions explicitly?

The fact is the KCE only sets up the access environment for the users. This includes Kafka parameter files (producer file for “jeff” and consumer file for “april”) and TLS (SSL) files. But when they are actually granted access is still controlled by the superuser “jdoe.”

To check the current ACLs, use the “list_acls.sh”. If your command line arguments are incorrect, the command will show you its correct format. You never really need a manual for these scripts.