1. Design the Microservice on VM
Use the Calabash GUI click on the “Infrastructure” on the top of the window. The Infrastructure page becomes current. Select “lake_finance” as the data system. And click on the green “Create Infrastructure Object” button.
Fill in the form as follows.
The “Microservice (VM) name” is not only the new VM name; it is also
- the consumer group id of the deployed app consuming data from “topic1,”
- the transactional id of the deployed app producing data to “topic2.”
Next, you need to make sure the VM will be in the same network where the PCA is. (We will pick up the PCA later.)
For the tutorial, we can pick a machine type that is the smallest with a small disk.
Next, you need to enter the cloud storage location where the application bundle is.
Calabash will build a docker image in the cloud using the application bundle. In the Calabash GUI, give a label for the docker image. For the tutorial, it is “demo:2.0.0.” The Docker image will reside in the cloud container registry.
Since our container image is based on the datacanals/kafka-client image, we must indicate this fact using the above checkbox. Once this box is checked, we have the chance to define properties for the PCA, KAFKA, etc. See below.
We also define a producer user and a consumer user. These are needed by Kafka. Every action on data must be by a named user with appropriate admin/consumer/producer permission. In this tutorial, the superuser “jdoe” will need to grant producer privilege to “jeff” and consumer privilege to “april.” We will see how to do that later.
The Kafka client application also requires a host volume. Properties about it are automatically filled in by Calabash GUI. And they are read-only, except for the volume name.
The volume will store the “init.properties” file containing initiation properties.
In the tutorial, the microservice will not offer any REST service. It will sit in an infinite loop, copying records between topics. So we leave properties about service empty in the GUI.
2. Deploy the Microservice
Before deploying the microservice “kafka-app-demo,” we must first create the Kafka topics it needs, i.e., “topic1” and “topic2.” We must also assign producer permission on “topic2” to user “jeff” and consumer permission on “topic1” to “april.” All these can be done in a KCE.
If you have not created a KCE, please follow Track 2 Topic 3: Create a KCE to create one.
Connect to the KCE, and issue these commands to create topics and assign permissions.
root@lf-kce:/app/bin# ./create_topic.sh jdoe topic1 Created topic topic1. root@lf-kce:/app/bin# ./create_topic.sh jdoe topic2 Created topic topic2. root@lf-kce:/app/bin# ./create_acl_for_producer.sh jdoe jeff topic2 Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=topic2, patternType=LITERAL)`: (principal=User:jeff, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:jeff, host=*, operation=WRITE, permissionType=ALLOW) (principal=User:jeff, host=*, operation=CREATE, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=topic2, patternType=LITERAL)`: (principal=User:jeff, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:jeff, host=*, operation=WRITE, permissionType=ALLOW) (principal=User:jeff, host=*, operation=CREATE, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=TRANSACTIONAL_ID, name=*, patternType=LITERAL)`: (principal=User:jeff, host=*, operation=ALL, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=TRANSACTIONAL_ID, name=*, patternType=LITERAL)`: (principal=User:jeff, host=*, operation=ALL, permissionType=ALLOW) root@lf-kce:/app/bin# ./create_acl_for_consumer.sh jdoe april topic1 Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=topic1, patternType=LITERAL)`: (principal=User:april, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:april, host=*, operation=READ, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`: (principal=User:april, host=*, operation=READ, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=topic1, patternType=LITERAL)`: (principal=User:april, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:april, host=*, operation=READ, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`: (principal=User:bk, host=*, operation=ALL, permissionType=ALLOW) (principal=User:kc, host=*, operation=ALL, permissionType=ALLOW) (principal=User:kc_consumer, host=*, operation=ALL, permissionType=ALLOW) (principal=User:april, host=*, operation=READ, permissionType=ALLOW)
We are now ready to deploy the microservice. Use the Calabash CLI, issue the “deploy i” command.
Calabash (tester:lake_finance)> deploy i kafka-app-demo Deploy to cloud? [y] Waiting #0 for image build... Latest build: a3d87930-7cce-4ab5-95d1-b4a5c33457e6, Started: 2021-09-16T06:40:31.175829575Z, Current status: SUCCESS Got a successful latest build: a3d87930-7cce-4ab5-95d1-b4a5c33457e6 Creating vm kafka-app-demo ... Deployed to 10.138.31.214 (internal) 34.83.186.95 (external)
3. Using the Application
Now our app is running! Let’s check if it is working.
We will send some data to “topic1” and see if the app can replicate them in “topic2” instantly. We can use the “produce.sh” script in the KCE. But we still have a barrier: there is no user allowed to produce to “topic1.” So for the testing purpose, we will grant “jeff” the producer privilege to “topic1.”
root@lf-kce:/app/bin# ./create_acl_for_producer.sh jdoe jeff topic1
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=topic1, patternType=LITERAL)`:
(principal=User:jeff, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:jeff, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:jeff, host=*, operation=CREATE, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=topic1, patternType=LITERAL)`:
(principal=User:april, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:april, host=*, operation=READ, permissionType=ALLOW)
(principal=User:jeff, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:jeff, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:jeff, host=*, operation=CREATE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TRANSACTIONAL_ID, name=*, patternType=LITERAL)`:
(principal=User:jeff, host=*, operation=ALL, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TRANSACTIONAL_ID, name=*, patternType=LITERAL)`:
(principal=User:jeff, host=*, operation=ALL, permissionType=ALLOW)
To manually produce data to “topic1,” use the “produce.sh” command:
root@lf-kce:/app/bin# ./produce.sh jeff topic1 >key1|this is the value of the first record >key2|the value of the second record >key3|the value of the third >^D root@lf-kce:/app/bin#
In the above, we created three records in “topic1”. The “>” is the prompt before a record.
Each record is a key-value pair. The key and value are separated by a vertical bar. They are both strings. You may type anything as long as the key does not contain a vertical bar.
Type Control-D to exit the “produce.sh” interaction.
Now let’s use another script, “consume.sh,” to see if user “april” can see the three records in “topic2.
root@lf-kce:/app/bin# ./consume.sh april topic2 g1 "key1"|"this is the value of the first record" "key2"|"the value of the second record" "key3"|"the value of the third" ^C root@lf-kce:/app/bin#
In the above, the consumer.sh displays the three records and sits waiting for more. Our app works. Type Control-C to get out of the loop.
In our tutorial so far, we have only one KCE. So we are forced to switch between the producer and consumer scripts. We cannot check if the data appear in “topic2” instantly after “jeff” writes to “topic1.” To check on that, we need two KCEs. This is left as an exercise to the readers.
Finally, make sure you undeloy the microservice before going to the next tutorial topic.