I have a large collection of data in an excel file (and csv files). The data needs to be placed into a database (mysql). However, before it goes into the database it needs to be processed..for example if columns 1 is less than column 3 add 4 to column 2. There are quite a few rules that must be followed before the information is persisted.
What would be a good design to follow to accomplish this task? (using java)
The process needs to be automated. In the sense that I don't have to manually go in and alter the data. We're talking about thousands of lines of data with 15 columns of information per line.
Currently, I have a sort of chain of responsibility design set up. One class(Java) for each rule. When one rule is done, it calls the following rule.
Typically there are about 5000 rows per data sheet. Speed isn't a huge concern because this large input doesn't happen often.
I've considered drools, however I wasn't sure the task was complicated enough for drols.
All currency (data in specific columns) must not contain currency symbols.
Category names must be uniform (e.g. book case = bookcase)
Entry dates can not be future dates
Text input can only contain [A-Z 0-9 \s]
Additionally if any column of information is invalid it needs to be reported when processing is complete (or maybe stop processing).
My current solution works. However I think there is room for improvement so I'm looking for ideals as to how it can be improved and or how other people have handled similar situations.
I've considered (very briefly) using drools but i wasn't sure the work was complicated enough to take advantage of drools.