Business Intelligence Data Load Monitoring
This article will cover data load monitoring for a business intelligence global solution. I completed a recent business intelligence survey of 20 clients where 45% of these clients had a dedicated data load monitoring (DLM) team and 35% had a 24 * 7 data load monitoring coverage.
Driver for a dedicated data load monitoring (DLM) team
My experience of a global business intelligence solution where no DLM team was in place for a client was that both the data load failures where not picked up in sufficient time which impacted data availability for the BI reports and the productivity of the business intelligence support team (level 2 / 3) was impacted due to them monitoring the data loads – basically trying to monitor data load activities in parallel to handling incidents and break-fix changes.
Determine if DLM team is required.
Not all business intelligence solutions will require a dedicated data load monitoring (DLM) team and if a DLM team is necessary a 24 * 7 coverage is not always required.
The following are key factors to determine if a DLM team is necessary along with the coverage required:
Overall number of extractions / jobs executed on daily basis.
Number of critical extractions / jobs executed on daily basis.
Number of extractions / jobs executed over weekend.
Schedules per time-zone (i.e. global business intelligence solution).
SLA defined for each schedule.
If the volume of data extractions is above, one thousand extractions on a daily basis and have critical extractions (for example month-end extractions) then I’d recommend that a DLM team is in place.
The coverage of the DLM team will depend if the schedules are executed over all time-zones and the weekend.
Take the scenario that a client has two thousand daily extractions which include month end extractions and have three main schedules with a SLA of six hours to cover each time zone along with weekend extractions then a DLM team with a coverage of 24 * 7 would be required.
DLM Team Size
Based on my experience, if a DLM team is required for 24 * 7 then the ideal coverage is a 3-shift rota with a DLM team size of seven DLM team members with one of these being the DLM team lead. If DLM team is required for 24 * 5 then the ideal coverage would be a 3-shift rota with a DLM team size of five DLM team members with one of these being the DLM team lead.
Career Path
The majority of DLM team members (except for DLM team lead) should be graduates or graduates with 1 years’ experience. After 2 years with the DLM team, the graduates should transition into level 2 / 3 support team and be backfilled by new graduates.
As the turnover of Level 2 / 3 support resources is high, the DLM team would be a pipeline of resources into Level 2 / 3 support team.
Other Activities
If the overall data schedules are relatively stable, the following are activities that could be done by DLM team in parallel to their data monitoring activities:
Data load Schedule performance improvements.
Manual activities (flat file uploads, data refreshes etc).
Data Reconciliation
How to monitor and status updates
The best strategy to monitor data load extractions is to put an email alert on all schedules if they fail along with an email alert on all schedules if an SLA threshold is reached. The distribution list for all DLM members should be created and this distribution list should be linked to all failure and SLA alerts.
A fifteen-minute daily call should be in place by the DLM team with the support lead to review all the main schedules along with any failures. If there is a support lead in each time zone then three fifteen minute meetings can be scheduled, one for each time zone to review the schedules along with any failures in the past eight hours.
Knowledge Bank
One main success factor of the DLM team is to ensure that all fixes / workarounds for each data load failure is document and stored in a Knowledge bank for future reference. All new fixes/workarounds logged should be communicated to all DLM team members.
Conclusion
From a supports lead perspective, having an offshore team with a clear separation of data monitoring from incident management activities allows for clear lines of responsibility along with higher productivity for both teams.
Having a DLM team has the following benefits
Clear responsibilities for DLM and L2/L3 team members.
L2/L3 can focus on incident management.
Data load failures are picked up more efficiently.
Pipeline of resources for L2/L3 team.
Manual items for example data reconciliation and data refreshes can be picked up by the DLM team.
Performance improvements to the data load schedules can be suggested and implemented by DLM team members.
Business Intelligence Solutions Support Survey
If you’re open to contributing, we’d love to hear your views in a survey that we’re conducting.
The survey has 17 questions but we’ve carefully designed and tested it to ensure it can be completed quickly, it should take no longer than 10 minutes.
In return for doing so, we’ll send you a report on the study’s full findings and we think you’ll find the results to be of value to your business.
Survey
