1. Code Generation
During the deployment of a pipeline, Calabash will generate Java code for you. The deployment process will fail if the generated code has a syntax error.
To make the deployment smooth, you can optionally generate the code locally and check its syntax. Here is how to do it using Calabash CLI.
% bin/calabash.sh Calabash CLI, version 3.0.0 Data Canals, 2022. All rights reserved. Calabash > connect tester Password: Connected. Calabash (tester)> use ds lake_finance Current ds set to lake_finance Calabash (tester:lake_finance)> code-gen payment-p3 Enter a directory for the generated code: /tmp/payment-p3 mkdirs /tmp/payment-p3 mkdirs /tmp/payment-p3/generated/build/libs Codegen is successful!
If you see “Codegen is successful!” then all is fine. Otherwise, you must diagnose where went wrong. The source code is generated in the directory “/tmp/payment-p3/generated.” Let’s go there to take a look.
% cd /tmp/payment-p3/generated % ls -F README.md com/ gradlew.bat src/ build/ gradle/ out/ build.gradle gradlew* settings.gradle % find src src src/main src/main/resources src/main/java src/main/java/com src/main/java/com/dcs src/main/java/com/dcs/kafka src/main/java/com/dcs/kafka/pipeline src/main/java/com/dcs/kafka/pipeline/generated src/main/java/com/dcs/kafka/pipeline/generated/Parser.java src/main/java/com/dcs/kafka/pipeline/generated/CustomFunc.java src/main/java/com/dcs/kafka/pipeline/generated/Lookup.java
The generated code is mainly contained in the three “.java” files. You can compile the code, without having to install anything:
% sh gradlew build BUILD SUCCESSFUL in 5s 3 actionable tasks: 3 executed
2. Kafka Topic and Permissions
Before deploying the pipeline, we must prepare topics it needs. First, in KCE, assign consumer permission on topic “converted-log1” to user “pl” so that the pipeline can load data from this topic.
root@lf-kce:/app/bin# ./create_acl_for_consumer.sh jdoe pl converted-log1
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=converted-log1, patternType=LITERAL)`:
(principal=User:pl, host=*, operation=READ, permissionType=ALLOW)
(principal=User:pl, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:
(principal=User:pl, host=*, operation=READ, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:
(principal=User:bk, host=*, operation=ALL, permissionType=ALLOW)
(principal=User:kc, host=*, operation=ALL, permissionType=ALLOW)
(principal=User:kc_consumer, host=*, operation=ALL, permissionType=ALLOW)
(principal=User:april, host=*, operation=READ, permissionType=ALLOW)
(principal=User:pl, host=*, operation=READ, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=converted-log1, patternType=LITERAL)`:
(principal=User:pl, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:april, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:pl, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:pl, host=*, operation=CREATE, permissionType=ALLOW)
(principal=User:april, host=*, operation=READ, permissionType=ALLOW)
(principal=User:pl, host=*, operation=READ, permissionType=ALLOW)
root@lf-kce:/app/bin# ./create_acl_for_consumer.sh jdoe april converted-log1
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=converted-log1, patternType=LITERAL)`:
(principal=User:april, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:april, host=*, operation=READ, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:
(principal=User:april, host=*, operation=READ, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:
(principal=User:bk, host=*, operation=ALL, permissionType=ALLOW)
(principal=User:kc, host=*, operation=ALL, permissionType=ALLOW)
(principal=User:kc_consumer, host=*, operation=ALL, permissionType=ALLOW)
(principal=User:april, host=*, operation=READ, permissionType=ALLOW)
(principal=User:pl, host=*, operation=READ, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=converted-log1, patternType=LITERAL)`:
(principal=User:pl, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:april, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:pl, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:pl, host=*, operation=CREATE, permissionType=ALLOW)
(principal=User:april, host=*, operation=READ, permissionType=ALLOW)
(principal=User:pl, host=*, operation=READ, permissionType=ALLOW)
Second, create all other topics with the following commands.
./create_topic.sh jdoe errors-3-load ./create_topic.sh jdoe errors-3-0 ./create_topic.sh jdoe errors-3-1 ./create_topic.sh jdoe errors-3-2 ./create_topic.sh jdoe aggregated-log1 ./create_topic.sh jdoe payment-p3-kvstore-0-changelog ./create_topic.sh jdoe payment-p3-kvstore-0-changelog-0 ./create_topic.sh jdoe payment-p3-kvstore-0-changelog-1
The last three topics (“payment-p3-kvstore-0-changelog,” “payment-p3-kvstore-0-changelog-0,” and “payment-p3-kvstore-0-changelog-1”) are for buffering data for aggregation.
Third, use the following commands to assign producer permissions to user “pl” on these topics so that the pipeline can write to them.
./create_acl_for_producer.sh jdoe pl errors-3-load ./create_acl_for_producer.sh jdoe pl errors-3-0 ./create_acl_for_producer.sh jdoe pl errors-3-1 ./create_acl_for_producer.sh jdoe pl errors-3-2 ./create_acl_for_producer.sh jdoe pl aggregated-log1 ./create_acl_for_producer.sh jdoe pl payment-p3-kvstore-0-changelog ./create_acl_for_producer.sh jdoe pl payment-p3-kvstore-0-changelog-0 ./create_acl_for_producer.sh jdoe pl payment-p3-kvstore-0-changelog-1
Fourth, use the following commands to assign consumer permissions to user “pl” on the topics used for buffering data. The pipeline must read them out when the waiting time is over.
./create_acl_for_consumer.sh jdoe pl payment-p3-kvstore-0-changelog ./create_acl_for_consumer.sh jdoe pl payment-p3-kvstore-0-changelog-0 ./create_acl_for_consumer.sh jdoe pl payment-p3-kvstore-0-changelog-1
Finally use the following commands to assign consumer permissions to user “april” so that she will be able to check data in these topics for monitoring purposes.
./create_acl_for_consumer.sh jdoe april errors-3-load ./create_acl_for_consumer.sh jdoe april errors-3-0 ./create_acl_for_consumer.sh jdoe april errors-3-1 ./create_acl_for_consumer.sh jdoe april errors-3-2 ./create_acl_for_consumer.sh jdoe april aggregated-log1 ./create_acl_for_consumer.sh jdoe april payment-p3-kvstore-0-changelog ./create_acl_for_consumer.sh jdoe april payment-p3-kvstore-0-changelog-0 ./create_acl_for_consumer.sh jdoe april payment-p3-kvstore-0-changelog-1
3. Deploy payment-p3
We are all set to deploy pipeline “payment-p3.”
Use Calabash CLI, issue the “deploy p” command.
Calabash (tester:lake_finance)> deploy p payment-p3
Deploy to cloud? [y]
Deploying new ms on k8s ...
Deploying microservice payment-p3 ...
Creating service.
Creating new secret payment-p3-secret
Secret created: payment-p3-secret
{
"kind": "Deployment",
"apiVersion": "apps/v1",
"metadata": {
"name": "payment-p3",
"namespace": "pl"
},
"spec": {
"selector": {
"matchLabels": { "exec": "payment-p3" }
},
"replicas": 2,
"template": {
"metadata": {
"labels": { "exec": "payment-p3" }
},
"spec":{
"hostNetwork": true,
"volumes": [{
"name": "secret-volume",
"secret": {
"secretName":"payment-p3-secret"
}
}, {
"name": "vol1",
"hostPath": {
"path": "/var/data",
"type": "DirectoryOrCreate"
}
}],
"containers": [{
"name": "payment-p3",
"image": "gcr.io/dlb-public-245218/pipeline:3.0.4",
"ports":[],
"volumeMounts": [{
"mountPath": "secret",
"name": "secret-volume"
}, {
"mountPath": "/data",
"name": "vol1"
}]
}]
}
}
}
}
Deployment created: payment-p3
Deployed
You will see the Kubernetes object specs during deployment. If all is fine, you will see the status “Deployed.”
To check data in the target topic “aggregated-log1,” issue the “./consume.sh” command.
root@lf-kce:/app/bin# ./consume.sh april aggregated-log1 g1
"account0000001"|{"amount_usd_max":500000,"amount_usd_avg":260000,"amount_usd_min":100000,"count":5}
"account0000002"|{"amount_usd_max":11800,"amount_usd_avg":11800,"amount_usd_min":11800,"count":3}
"account0000003"|{"amount_usd_max":30000,"amount_usd_avg":30000,"amount_usd_min":30000,"count":2}
In the output, the vertical bars separate keys and values.
You can also use user “april” to consume the error topics. They should be empty now, but if you add erroneous data in the source data file, you will eventually see the error message in the error topics.