1. Code Generation
During the deployment of a pipeline, Calabash will generate Java code for you. The deployment process will fail if the generated code has a syntax error.
You can optionally generate the code locally and check its syntax before the deployment. It will help ensure the deployment will be a smooth process. Here is how to do it using Calabash CLI.
% bin/calabash.sh Calabash CLI, version 3.0.0 Data Canals, 2022. All rights reserved. Calabash > connect tester Password: Connected. Calabash (tester)> use ds lake_finance Current ds set to lake_finance Calabash (tester:lake_finance)> code-gen payment-p1 Enter a directory for the generated code: /tmp/payment-p1 mkdirs /tmp/payment-p1 mkdirs /tmp/payment-p1/generated/build/libs Codegen is successful!
If you see “Codegen is successful!” then all is fine. Otherwise, you must diagnose where went wrong. The source code is generated in the directory “/tmp/payment-p1/generated.” Let’s go there to take a look.
% cd /tmp/payment-p1/generated % ls -F README.md com/ gradlew.bat src/ build/ gradle/ out/ build.gradle gradlew* settings.gradle % find src src src/main src/main/resources src/main/java src/main/java/com src/main/java/com/dcs src/main/java/com/dcs/kafka src/main/java/com/dcs/kafka/pipeline src/main/java/com/dcs/kafka/pipeline/generated src/main/java/com/dcs/kafka/pipeline/generated/Parser.java src/main/java/com/dcs/kafka/pipeline/generated/CustomFunc.java src/main/java/com/dcs/kafka/pipeline/generated/Lookup.java
The generated code is mainly contained in the three “.java” files. You can compile the code, without having to install anything:
% sh gradlew build BUILD SUCCESSFUL in 4s 3 actionable tasks: 3 executed
If there are any syntax errors, diagnose and modify the Java class file. Then go back to the pipeline editor to correct the errors.
2. Kafka Topics and Permissions
Before deploying the pipeline, we must prepare topics it needs.
First, in KCE, assign consumer permission on topic “log1” to user “pl” so that the pipeline can load data from it.
root@lf-kce:/app/bin# ./create_acl_for_consumer.sh jdoe pl log1 Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=log1, patternType=LITERAL)`: (principal=User:pl, host=*, operation=READ, permissionType=ALLOW) (principal=User:pl, host=*, operation=DESCRIBE, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`: (principal=User:pl, host=*, operation=READ, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`: (principal=User:bk, host=*, operation=ALL, permissionType=ALLOW) (principal=User:kc, host=*, operation=ALL, permissionType=ALLOW) (principal=User:kc_consumer, host=*, operation=ALL, permissionType=ALLOW) (principal=User:april, host=*, operation=READ, permissionType=ALLOW) (principal=User:pl, host=*, operation=READ, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=log1, patternType=LITERAL)`: (principal=User:pl, host=*, operation=READ, permissionType=ALLOW) (principal=User:april, host=*, operation=READ, permissionType=ALLOW) (principal=User:april, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:pl, host=*, operation=DESCRIBE, permissionType=ALLOW)
Second, create all other topics with the following commands.
./create_topic.sh jdoe errors-1-0 ./create_topic.sh jdoe errors-1-1 ./create_topic.sh jdoe errors-1-2 ./create_topic.sh jdoe errors-1-3 ./create_topic.sh jdoe clean-log1
Third, use the following commands to assign producer permissions to user “pl” on these topics so that the pipeline can write to them.
./create_acl_for_producer.sh jdoe pl errors-1-0 ./create_acl_for_producer.sh jdoe pl errors-1-1 ./create_acl_for_producer.sh jdoe pl errors-1-2 ./create_acl_for_producer.sh jdoe pl errors-1-3 ./create_acl_for_producer.sh jdoe pl clean-log1
Finally use the following commands to assign consumer permissions to user “april” so that she will be able to check data in these topics for monitoring purpose.
./create_acl_for_consumer.sh jdoe april errors-1-0 ./create_acl_for_consumer.sh jdoe april errors-1-1 ./create_acl_for_consumer.sh jdoe april errors-1-2 ./create_acl_for_consumer.sh jdoe april errors-1-3 ./create_acl_for_consumer.sh jdoe april clean-log1
3. Deploy payment-p1
We are all set to deploy pipeline “payment-p1.”
Use Calabash CLI, issue the “deploy p” command.
Calabash (tester:lake_finance)> deploy p payment-p1 Deploy to cloud? [y]
Deploying new ms on k8s ... Deploying microservice payment-p1 ... { "kind":"Namespace", "apiVersion":"v1", "metadata":{ "name":"pl" } } Namespace created: pl Creating service. Creating new secret payment-p1-secret Secret created: payment-p1-secret { "kind": "Deployment", "apiVersion": "apps/v1", "metadata": { "name": "payment-p1", "namespace": "pl" }, "spec": { "selector": { "matchLabels": { "exec": "payment-p1" } }, "replicas": 2, "template": { "metadata": { "labels": { "exec": "payment-p1" } }, "spec":{ "hostNetwork": true, "volumes": [{ "name": "secret-volume", "secret": { "secretName":"payment-p1-secret" } }, { "name": "vol1", "hostPath": { "path": "/var/data", "type": "DirectoryOrCreate" } }], "containers": [{ "name": "payment-p1", "image": "gcr.io/dlb-public-245218/pipeline:3.0.4", "ports":[], "volumeMounts": [{ "mountPath": "secret", "name": "secret-volume" }, { "mountPath": "/data", "name": "vol1" }] }] } } } } Deployment created: payment-p1 Deployed
You will see the Kubernetes object specs during deployment. If all is fine, you will see the status “Deployed.”
To check data in the target topic “clean-log1,” issue the “./consume.sh” command.
root@lf-kce:/app/bin# ./consume.sh april clean-log1 g1
"05/06/2021 12:30:30-account0000001"|{"amount":1000,"account_id":"account0000001","ts":"05/06/2021 12:30:30"}
"05/06/2021 16:34:10-account0000002"|{"amount":100,"account_id":"account0000002","ts":"05/06/2021 16:34:10"}
"05/03/2021 02:05:23-account0000001"|{"amount":5000,"account_id":"account0000001","ts":"05/03/2021 02:05:23"}
"05/04/2021 16:35:30-account0000002"|{"amount":100,"account_id":"account0000002","ts":"05/04/2021 16:35:30"}
"05/04/2021 12:30:30-account0000001"|{"amount":1000,"account_id":"account0000001","ts":"05/04/2021 12:30:30"}
"05/07/2021 06:45:30-account0000003"|{"amount":2000,"account_id":"account0000003","ts":"05/07/2021 06:45:30"}
"05/03/2021 12:30:30-account0000001"|{"amount":1000,"account_id":"account0000001","ts":"05/03/2021 12:30:30"}
"05/04/2021 16:34:10-account0000002"|{"amount":100,"account_id":"account0000002","ts":"05/04/2021 16:34:10"}
"05/03/2021 06:45:30-account0000003"|{"amount":2000,"account_id":"account0000003","ts":"05/03/2021 06:45:30"}
"05/07/2021 02:05:23-account0000001"|{"amount":5000,"account_id":"account0000001","ts":"05/07/2021 02:05:23"}
"05/03/2021 12:30:30-account0000001"|{"amount":1000,"account_id":"account0000001","ts":"05/03/2021 12:30:30"}
In the output, the keys and values are separated by vertical bars. You can see the keys contain timestamps and account ids.
You can also use user “april” to consumer the error topics. They should be empty now, but if you add erroneous data in a source data file, you will eventually see the error message in the error topics.