The ruleengine will allow you to maintain complex business logic outside your Nifi flow. This is done through a web application that is freely available. You can check the content of flow files and the ruleengine will return some indicators, if the content is according to the defined business rules/logic.
These checks could be realized inside the Nifi flow as well, but as the logic gets more complex, if you implement it in the flow, the flow gets more cluttered. A cluttered flow is more difficult to understand and work on and contradicts the agile way for working with changed requirements. Embedding you business logic in the flow is also a quality issue: if there is no central place for keeping the business logic, it will be spread all over the flow. Over time the logic grows and it gets harder and harder to overview and understand it.
The other argument in this discussion is, that the business experts should define the business logic and not the IT expert. This is a clear devision of responsabilities and each expert is doing what he or she is best at in his/her domain.
Also, when you use the ruleengine, you have a central place for your business logic. That means if somebody wants to understand what logic is in place, then he does not need to search for it in various tools and applications and IT code, but it is there in the Business Rules Maintenance Tool (web application) to be reviewed. Again, this enhances quality and avoids errors.
Here is now an example of a Nifi flow using the ruleengine. The flow conists of 6 processors:
- GetFile: get a CSV file from the filesystem
- SplitText: split the CSV file into individual rows
- RuleEngine: runs the business rules against the content of each flowfile
- UpdateAttribute: Assign a new filename for each flowfile
- RouteOnAttribute: Route the flow file depending on the results of the ruleengine
- PutFile: store the flowfile in the filesystem
Here is a screenprint of the application after I logged in and clicked on "Projects" in the menu on the left side:
Clicking on the project name "Test Nifi 2" will reveal the groups that are inside the project. Groups bundle the logic for a certain purpose together. As there is only one group here in this example, I skip this and show you what is inside the group:
Here is a sample from the CSV file I use. It contains free information about geographical placenames. You can download it from geonames.org. The individual fields are separated by a tab character. Besides other information each line has a feature code (in this case: "SLP") and also timezone information ("Europe/Andorra").
3038814 Costa de Xurius Costa de Xurius 42.50692 1.47569 T SLP AD 07 0 1839 Europe/Andorra 2015-03-08
And there is an action. Actions are related to the group as a whole. An action is fired either if the group of rules passes the logic or if it fails the logic. In this case if the data passes the logic, the timezone field is updated to contain "XXXX".
The project in the web tool can be exported as a Zip file as mentioned above. It contains all the logic as discussed above. In the Nifi RuleEngine processor, reference the Zip file as shown below. Specify the names of the individual fields of the CSV row. If you do this, then in the rules you can reference these field names. Otherwise you will need a use the index of the fields which is more cumbersome. Also specify how the individual fields of the rows are separated from each other (here: a tab character).
If you have e.g. 5 groups, then you can decide that only if zero groups failed, the flow file is routed onwards. The other way around, you can decide that in case all groups failed the flow file is routed somewhere else. You have many possibilities here.
In the flow I use the RouteOnAttribute processor after the RuleEngine processor. Here I evaluate if the number of failed groups is equal to zero. And then I pass there flow files onwards to the PutFile processor.
3038838 Costa Verda Costa Verda 42.48297 1.66086 T SLP AD 08 0 2572 XXXX 2015-05-06
But now comes the really interesting part: If you change the business logic, then you DO NOT have to touch your flow at all! Update the business logic in the Business Rules web application and export the project again. The RuleEngine processor in the flow will now use the new business logic definition and produce the new result.
E.g. you could add a rule checking if the name (second column) equals "Costa Verda". So only rows where the feature code is "SLP" and the timezone starts with "Europe" AND where the name is "Costa Verda" would pass the business logic.
In the screenprint below, you can see that I added this rule to check the name column. I exported the project again and used it with the Nifi flow. This time, out of the 100 input lines, the ruleengine found only 2 rows that passed the business logic.
I hope you enjoyed this blog post. Next time I will blog about how to see the detailed results of what happened when the ruleengine ran. You can output these details also to flow files and it will show you exactly which rule passed or failed and why.
Carpe Diem