As processing power has increased, data caching has jumped to the forefront and user-demand for immediate response has grown, developers seem to have moved away from batch processing. But in a lot of situations, this can be a better approach than real-time processing.

A few years ago I was tasked with building a piece of middleware to facilitate data exchange between 2 SQL Server-based systems. The main system was a Navision system that maintained the company's financials and inventory. The other was an Indaba installation which managed inventory for sale on 3rd party marketplaces, such as Amazon. Additionally, Indaba managed the actual orders placed on the marketplaces. So when an order was placed it came first to Indaba and Indaba double checked its inventory to make sure the order could be fulfilled. However, Navision was the system from which all order fulfillment took place; all the warehouse paperwork was generated in it and all shipping information was pulled from it.

The data flow went something like this: as new inventory was added into Navision, it was pushed into Indaba. As orders from the marketplaces came into Indaba they were checked for availability against Indaba's copy of the inventory and pushed into Navision. Both were seen as needing to be done in real-time.

Ignoring the complexity of the data in Navision (quantities are NEVER stored, always calculated) this sounds pretty simple, right? I thought so too. I jumped in and built a SQL trigger that pushed new inventory to Indaba each time a new item was added. On the flip side, I wanted to add a trigger in Indaba's database to push order information to Navision. This is when I ran into a problem.You see, Navision is looked at as a black box. It has its own programming language called C/SIDE that allows for development within its environment. Our Navision consultant explained that if I just pushed data into the sales order tables from the database side, it would by-pass all the checks and balances built into the code and in general be VERY bad.

Since I respected this man's opinion, we set about finding a way around this. What we ended up doing was creating a method to process the orders in a batch. Once the orders were placed in Indaba, we weren't going to lose them. So we built staging tables in Navision that Indaba pushed the orders into. Then, on a schedule, Navision's internal code could process the data from those tables and generate the sales orders it needed.

While it was imperative that Indaba maintain a real-time inventory copy, it was not necessary that the orders be brought into Navision in real-time. Batch processing was the answer. In this age where developers believe a customer cannot even be bothered with a page refresh (hello AJAX) the idea that all processing should be done in real-time can cause problems. Keep batch processing in mind.

This entry was posted on January 18, 2012 at 1:21 PM and has received 401 views. There are currently 0 comments. Print this entry.