Now in regards to the ruleengine, the business logic for a certain topic or project is maintained in a web application. So the ETL logic and the business logic are separated and decoupled. From the web application the project can be exported into a single compressed file and then simply used by the ruleengine step in Tweakstreet. This process can be scripted and automated if required. The business can learn how to define the logic in the web application and at that point they are responsible for it and not IT anymore. And I have seen this happening.
But here is more information on using a ruleengine:
Advantages:
- arbitary complex logic possible
- avoiding duplication of logic in various, unconnected places
- logic is NOT in the code: if the rules change (changed requirements) the code does not change
- proper separation of responsibilities between IT and Business
- central store for business logic
- business logic can come into effect or expire over time
- advanced features: testing logic, creating detailed documentation for the business logic, search logic, set validity dates on logic
- manage users and roles and histroy of changes of the logic
- leaner IT code: more agile processes and better quality
- detailed output of the results of the rules: good for debugging, error analysis or as an audit trail
The ruleengine can be used to:
- quality check and validate data based on rule execution results
- filter data based on rule execution results
- update/calculate data based on rule execution results
The first point under advantages is interesting. Of course one could put the logic in a Tweakstreet modul to make it reusable. But the logic then still is a part of the IT process and cannot be maintained by the business; meaning IT will have to do it. Especially when you have complex business logic with a lot of "if" conditions which are connected with varying "and" and "or" conditions, or when the business frequently changes the requirements, the ruleengine approach delivers a higher quality, more user satisfaction and quicker implementation times.
Let's have a look at a sample in Tweakstreet. It is a flow to calculate travel discount for persons flying to certain destination airports. It uses randomly generated data.
- the "Generate Rows" step generates a defined number of rows
- For each input row from the "Generate Rows" step, a set of random data is generated
- the "JaRE Rules" step executes the business logic against the data it receives from the "Random Data" step
- the results of the ruleengine are filtert into persons that received a discount and those who didn't
The random data generated consists of:
- the destination airport
- the destination region (Europe, America, Asia, etc.)
- the age of the traveling person
- an indicator if the person is a frequent traveller
The ruleengine step uses the defined business logic to evaluate if certain conditions in the data are met. Based on these conditions the data "passes" or "fails" and some fields are updated to indicate what was the result of the execution of the business rules and which discount applies to the concrete data set.
The rules - the business logic - is as follows:
- a person at the age of 50 or above travelling to Europe gets a 25% discount
- a person at the age 4 or below gets a discount of 100% for any destination region
The first rule needs to check the age and the destination region of the person. The other one needs to only check the age. Both rules - in case the logic applies - then update fields in the relevant data row with the following: an id of the discount, the discount percentage and a yes/no indicator if the row passed or failed the business logic.
Here is a screenshot of the business logic from the Business Rules Maintenance tool - the web application to orchestrate the business logic:
Let's have a look at the first group. It checks if the person is 50 years of age or older and if the destination region is Europe. The rulegroup consists of a subgroup and two rules: one for the age check and one for the destination region check. The rule connector is set to "and", so both conditions must be met for the rulegroup to pass. Note that the rulegroup has a validity from/to date to indicate in which timeframe this logic is valid.
The other rulegroup is similar but contains only one rule where the age of the person is checked and if it is 4 years or less, a 100% discount is applied.
The ruleengine step in Tweakflow looks like this:
The "Output fields" section shows which fields are selected to be output: the discount fields discussed further above, plus a summary of the results for the data row and the detailed results for that row.
Running the flow before the business logic is applied (before the "JaRE Rules" step), shows following situation:
The next screenshot shows the same data after it has passed the ruleengine step:
The last two fields contain the summary of the execution and the details of the execution. Here is the summary field content:
The field with the detailed information looks like this. I just show a part of it because the details for the two groups and 3 business rules are much more but it is not appropriate to show all of it here. So these are the details of the first rulegroup checking the 50+ age and the destination region of Europe. This now is for a record which passed both checks and subsequently was updated by the actions.
These details can be used for debugging and testing or for making decisions further down the flow based on these details. But it can also be used as a full audit trail of why and which data passed or failed together with the corresponding business logic.
The ruleengine gives you a lot of functionality:
- rulegroups have a validity start and end date so business logic can automatically expire or become effective
- rulegroups can depend on the results of other rulegroups
- the web interface lets you import/export projects, copy, search and test logic
- the web application manages users and roles, history of changes
- the web tool allows to generate complete documentation for the business logic of a project or parts of it
The web application comes free of charge and is available for download here: https://github.com/uwegeercken/rule_maintenance_war. It requires a MySql or MariaDb instance and a web server which can use a war file. Here is also a link to various documentation.
If you are interested to use the ruleengine step in Tweakstreet, then send me an email. I will provide it free of charge and in the near future there will be a central place to download it. You can also download the source code from Github and use maven to create the plugin yourself - it is not complicated.
I hope this was helpful for you. Remember the points from the beginning: mixing your IT logic with business logic is not a good idea. In the end it will be you as an IT person to do both: Maintaining the data integration process AND maintaining the business logic. This means you will be held responsible for both parts, although the business should really be responsible for their part in defining the business logic. You will benefit from leaner processes and thus increased agility. The ruleengine helps to keep things simple and that has a direct impact on the quality of your data integration process.
Carpe Diem