And as times passes by, more and more logic get's inside our processes and over time it gets difficult to administer. Also, we are mixing our code with the business logic: if we change one or the other, we risk that the other part is affected as well. It is also in-transparent, because business users have usually no access to the ETL processes and thus do not have a place to look up the logic. So what they do is to call you as the IT guy, asking you to verify why data is wrong or to implement additional logic. Or they ask to change existing logic. It's always you to react.
And really we as developers of ETL processes or other code are not the first place to implement or verify business logic. It's the business users who should define and maintain the logic that is responsible for filtering data, for defining output for interfaces or to make quality checks on our data.
Here a rule engine comes into play: it allows to maintain the rules - defining the business logic - in a central place and outside of the ETL process or code. Once the business logic is located outside - e.g. in files - there is no need to change the ETL process or your code, if the business logic changes. Furthermore, a central repository of rules is more transparent to the user; it's a collections of business rules in a central place and through a user centric interface it allows the user to understand how the logic is defined and to define additional logic.
A rule engine helps the IT to concentrate on their work and avoids implementing business logic where it does not belong: in between your code or transformations. Users gain transparency and IT will not need to touch the code or transformation if the logic changes. A clear devision of responsabilities. And the resposability for the business logic is with the business user - where is belongs.
I have written a rule engine in Java, which allows the IT to "outsource" the business logic. It can be used in any project or application that is based on Java: web applications, Java script languages such as Groovy or other Java based programs.
The other part is a web application to maintain rules - the business logic - in a central place and to do maintenance: defining rules, grouping rules together using complex log, adding, deleting and updating.
Additionally, for the Pentaho ETL tool, there is a plugin step available, which connects the ETL process to the business logic in the form of rules. So a simple step is replacing all the logic that previously was spread all over the place (transformation). The transformation get's simpler and - as already indicated - does not need to be touched, when the business comes up with new rules or changes the existing ones. This is of course also a quality issue: it enhances the quality of your IT services.
If you have a lot of transformations, respectively ETL processes, using a rule engine can make a big difference. Quality of service, division of responsibilities, clearness of code and processes and also more transparency.
All code and tools mentioned above are open source and can be freely used - at no cost.