In the last blog post I showed you how to write rules, so you can check your data. Today I show you that the plugin step for Pentaho PDI not only returns the statistical numbers of how many groups and rules failed, but you can also retrieve the full details of what happened when the ruleengine ran.
The plugin step has two output streams: one gives you only the statistics of how many groups and rules failed. You can use this to split your data into failed rows and passed rows and further process them. For each input row there will be one output row.
The second output stream (which is optional) will give you one output row for each input row and rule. So if you have 10000 rows and 10 rules, you will get 100000 output rows. Each row will show the name of the group, subgroup and rule and if the group failed, if the subgroup failed and if the rule failed. And it will give you a message (the message you defined when writing the rule) showing why the rule failed or passed. With these details you can debug your rule engine results easily. And of course you can also further process the detail rows in PDI.
I have added two more steps to the transformation we used the last time: one select step to reduce the number of fields (so it is easier to demonstrate to you) and a dummy step.
When you open the rule engine step, you can select the step where the rule results go to (Rule Result Step). And in the dropdown below, you can select which data to output to the step. You can pass all rows to the step or all rows where the group failed. Or also all rows where the group passed.
The screenshot above shows a preview of the detailed results from the rule engine as described above. On the very right there is a column showing the message that was produced by the rule engine, indicating why the rule passed or failed.
This is of course just a quick example for the purpose of demonstration. A typical use case would be to have a set of data, where you would expect to get no errors. But when errors occur, the rule engine details as described above give you a very good way to determine what happend.