From time to time environments may suffer from slow ingestion into Enterprise Vault (EV). This article gives some guidelines on how to review these environments to find where the bottleneck may lie.
When looking at ingest performance the two things to look at are:
A/ Is Archive Shuttle providing enough work to EV?
B/ Is EV keeping up?
Is Archive Shuttle providing enough work to EV?
To see if Archive Shuttle is producing enough work for EV look in the EV Import Module log file, and every minute the following will be logged:
2014-09-10 10:07:34Z|3164| 9| INFO| OnQueueStatisticsTimerElapsed|Working copy threads:  | Waiting Copy Items:  | Working Ingest Threads:  | Waiting Ingest Items:  2014-09-10 10:07:48Z|3164|239| INFO| OnPerformanceTimerElapsedImpl|Last minute statistics: Count:  | Items/s: [8.20] | kb/s:  | ACPU:  | MaxCPU:  | MinCPU:  | TSize: [59124 kb] | ASize: [120 kb] | MaxSize: [2011 kb] | | MinSize (kb):  | ATime: [3615 ms] | MaxTime: [16288 ms] | MinTime: [86 ms] 2014-09-10 10:07:48Z|3164| 9| INFO| OnPerformanceTimerElapsedImpl|Last minute statistics: Free memory: [429 MB] | Avg Free Memory: [409 MB] | Process Memory: [214 MB] | Avg Process Memory: [220 MB] | APCPU: [17 %] | PCPU: [19 %]
Working Copy Threads is the number of Archive Shuttle threads getting data ready to send to EV.
Working Ingest Threads is the number of threads actively pushing data into Enterprise Vault.
Waiting Ingest Items is the queue of the items waiting to be processed by EV.
If this last number is 100, 200, 300 .. always … then we are pushing enough work to EV … EV is just ‘taking time’ to process the items.
If this last number is always low then the problem may lie within the Archive Shuttle environment, and QUADROtech Support can assist with investigating that.
Is EV keeping up?
Archive Shuttle’s EV Import module uses the Symantec approved the method of ingesting data into a target. That is the EV API. Sometimes called the ECMAPI. This is part of the StorageOnlineOPNS process.
To check whether there is a bottleneck inside that process it is necessary to do a DTRACE of the StorageOnlineOPNS process on the target server (or servers). It’s best to try to run that for 10-15 minutes whilst the ingest is happening. The suggestion is 10-15 minutes, but it can be less, it can be more. It depends how busy the system actually is because the DTRACE log files grow to 100 Mb and roll over quickly, which then makes investigating and reviewing those files more difficult.
Quadrotech Support can then take a quick look at that DTrace using a tool called Jiggle. Jiggle looks for ‘large deltas’ of time between steps of the trace. For example, if looking up user information in Active Directory takes 5000 ms, then that would be discoverable in Jiggle. If it was always above 5000ms then the troubleshooting would head in the Active Directory route. If it’s taking time to store items to disk … then troubleshooting would go that route.
Quadrotech Support can only take a quick look at this and offer advice based on our cumulative years of experience. It could be that nothing stands out as a possible cause, or that something that we’re not familiar with stands out.
In either of these cases and for verification/confirmation, a ticket would need to be raised with Symantec.