1. Code Generation
During the deployment of a pipeline, Calabash will generate Java code for you. The deployment process will fail if the generated code has a syntax error.
To make the deployment smooth, you can optionally generate the code locally and check its syntax. Here is how to do it using Calabash CLI.
% bin/calabash.sh Calabash CLI, version 3.0.0 Data Canals, 2022. All rights reserved. Calabash > connect tester Password: Connected. Calabash (tester)> use ds lake_finance Current ds set to lake_finance Calabash (tester:lake_finance)> code-gen payment-p2 Enter a directory for the generated code: /tmp/payment-p2 mkdirs /tmp/payment-p2 mkdirs /tmp/payment-p2/generated/build/libs Codegen is successful!
If you see “Codegen is successful!” then all is fine. Otherwise, you must diagnose where went wrong. The source code is generated in the directory “/tmp/payment-p2/generated.” Let’s go there to take a look.
% cd /tmp/payment-p2/generated % ls -F README.md com/ gradlew.bat src/ build/ gradle/ out/ build.gradle gradlew* settings.gradle % find src src src/main src/main/resources src/main/java src/main/java/com src/main/java/com/dcs src/main/java/com/dcs/kafka src/main/java/com/dcs/kafka/pipeline src/main/java/com/dcs/kafka/pipeline/generated src/main/java/com/dcs/kafka/pipeline/generated/Parser.java src/main/java/com/dcs/kafka/pipeline/generated/CustomFunc.java src/main/java/com/dcs/kafka/pipeline/generated/Lookup.java
The generated code is mainly contained in the three “.java” files. You can compile the code, without having to install anything:
% sh gradlew build BUILD SUCCESSFUL in 5s 3 actionable tasks: 3 executed
2. Kafka Topic and Permissions
Before deploying the pipeline, we must prepare topics it needs.
First, in KCE, assign consumer permission on topic “clean-log1” to user “pl” so that the pipeline can load data from this topic.
root@lf-kce:/app/bin# ./create_acl_for_consumer.sh jdoe pl clean-log1
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=clean-log1, patternType=LITERAL)`:
(principal=User:pl, host=*, operation=READ, permissionType=ALLOW)
(principal=User:pl, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:
(principal=User:pl, host=*, operation=READ, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:
(principal=User:bk, host=*, operation=ALL, permissionType=ALLOW)
(principal=User:kc, host=*, operation=ALL, permissionType=ALLOW)
(principal=User:kc_consumer, host=*, operation=ALL, permissionType=ALLOW)
(principal=User:april, host=*, operation=READ, permissionType=ALLOW)
(principal=User:pl, host=*, operation=READ, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=clean-log1, patternType=LITERAL)`:
(principal=User:pl, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:april, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:pl, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:pl, host=*, operation=CREATE, permissionType=ALLOW)
(principal=User:april, host=*, operation=READ, permissionType=ALLOW)
(principal=User:pl, host=*, operation=READ, permissionType=ALLOW)
Second, create all other topics with the following commands.
./create_topic.sh jdoe errors-2-load ./create_topic.sh jdoe errors-2-1 ./create_topic.sh jdoe errors-2-2 ./create_topic.sh jdoe errors-2-3 ./create_topic.sh jdoe converted-log1 ./create_topic.sh jdoe payment-p2-kvstore-0-changelog ./create_topic.sh jdoe payment-p2-kvstore-0-changelog-0 ./create_topic.sh jdoe payment-p2-kvstore-0-changelog-1
The last three topics (“payment-p2-kvstore-0-changelog,” “payment-p2-kvstore-0-changelog-0,” and “payment-p2-kvstore-0-changelog-1”) are for buffering data for deduplication.
Third, use the following commands to assign producer permissions to user “pl” on these topics so that the pipeline can write to them.
./create_acl_for_producer.sh jdoe pl errors-2-load ./create_acl_for_producer.sh jdoe pl errors-2-1 ./create_acl_for_producer.sh jdoe pl errors-2-2 ./create_acl_for_producer.sh jdoe pl errors-2-3 ./create_acl_for_producer.sh jdoe pl converted-log1 ./create_acl_for_producer.sh jdoe pl payment-p2-kvstore-0-changelog ./create_acl_for_producer.sh jdoe pl payment-p2-kvstore-0-changelog-0 ./create_acl_for_producer.sh jdoe pl payment-p2-kvstore-0-changelog-1
Fourth, use the following commands to assign consumer permissions to user “pl” on the topics used for buffering data. The pipeline must read them out when the waiting time is over.
./create_acl_for_consumer.sh jdoe pl payment-p2-kvstore-0-changelog ./create_acl_for_consumer.sh jdoe pl payment-p2-kvstore-0-changelog-0 ./create_acl_for_consumer.sh jdoe pl payment-p2-kvstore-0-changelog-1
Finally use the following commands to assign consumer permissions to user “april” so that she will be able to check data in these topics for monitoring purposes.
./create_acl_for_consumer.sh jdoe april errors-2-load ./create_acl_for_consumer.sh jdoe april errors-2-1 ./create_acl_for_consumer.sh jdoe april errors-2-2 ./create_acl_for_consumer.sh jdoe april errors-2-3 ./create_acl_for_consumer.sh jdoe april converted-log1 ./create_acl_for_consumer.sh jdoe april payment-p2-kvstore-0-changelog ./create_acl_for_consumer.sh jdoe april payment-p2-kvstore-0-changelog-0 ./create_acl_for_consumer.sh jdoe april payment-p2-kvstore-0-changelog-1
3. Deploy payment-p2
We are all set to deploy pipeline “payment-p1.”
Use Calabash CLI, issue the “deploy p” command.
Calabash (tester:lake_finance)> deploy p payment-p2
Deploy to cloud? [y]
Deploying new ms on k8s ...
Deploying microservice payment-p2 ...
Creating service.
Creating new secret payment-p2-secret
Secret created: payment-p2-secret
{
"kind": "Deployment",
"apiVersion": "apps/v1",
"metadata": {
"name": "payment-p2",
"namespace": "pl"
},
"spec": {
"selector": {
"matchLabels": { "exec": "payment-p2" }
},
"replicas": 2,
"template": {
"metadata": {
"labels": { "exec": "payment-p2" }
},
"spec":{
"hostNetwork": true,
"volumes": [{
"name": "secret-volume",
"secret": {
"secretName":"payment-p2-secret"
}
}, {
"name": "vol1",
"hostPath": {
"path": "/var/data",
"type": "DirectoryOrCreate"
}
}],
"containers": [{
"name": "payment-p2",
"image": "gcr.io/dlb-public-245218/pipeline:3.0.4",
"ports":[],
"volumeMounts": [{
"mountPath": "secret",
"name": "secret-volume"
}, {
"mountPath": "/data",
"name": "vol1"
}]
}]
}
}
}
}
Deployment created: payment-p2
Deployed
You will see the Kubernetes object specs during deployment. If all is fine, you will see the status “Deployed.”
To check data in the target topic “converted-log1,” issue the “./consume.sh” command.
root@lf-kce:/app/bin# ./consume.sh april converted-log1 g1
"account0000001"|{"amount":5000,"account_id":"account0000001","amount_usd":500000,"currency":"USD","conversion_rate":1.0,"ts":"05/03/2021 02:05:23"}
"account0000001"|{"amount":1000,"account_id":"account0000001","amount_usd":100000,"currency":"USD","conversion_rate":1.0,"ts":"05/04/2021 12:30:30"}
"account0000001"|{"amount":1000,"account_id":"account0000001","amount_usd":100000,"currency":"USD","conversion_rate":1.0,"ts":"05/06/2021 12:30:30"}
"account0000001"|{"amount":1000,"account_id":"account0000001","amount_usd":100000,"currency":"USD","conversion_rate":1.0,"ts":"05/03/2021 12:30:30"}
"account0000001"|{"amount":5000,"account_id":"account0000001","amount_usd":500000,"currency":"USD","conversion_rate":1.0,"ts":"05/07/2021 02:05:23"}
"account0000002"|{"amount":100,"account_id":"account0000002","amount_usd":11800,"currency":"EUR","conversion_rate":1.18,"ts":"05/04/2021 16:35:30"}
"account0000002"|{"amount":100,"account_id":"account0000002","amount_usd":11800,"currency":"EUR","conversion_rate":1.18,"ts":"05/06/2021 16:34:10"}
"account0000003"|{"amount":2000,"account_id":"account0000003","amount_usd":30000,"currency":"CNY","conversion_rate":0.15,"ts":"05/03/2021 06:45:30"}
"account0000002"|{"amount":100,"account_id":"account0000002","amount_usd":11800,"currency":"EUR","conversion_rate":1.18,"ts":"05/04/2021 16:34:10"}
"account0000003"|{"amount":2000,"account_id":"account0000003","amount_usd":30000,"currency":"CNY","conversion_rate":0.15,"ts":"05/07/2021 06:45:30"}
In the output, keys and values are separated by vertical bars. You can see the keys contain only account ids. We isolate them into the key to preparing for group-by-key in the next tutorial track.
You can also use user “april” to consumer the error topics. They should be empty now, but if you add erroneous data in the source data file, you will eventually see the error message in the error topics.