", what's the bottleneck on performance?
Major impacts are these three areas
- Disk fragmentation
- Amount of dedupe data
- Disk performance (seek times/read speeds)
Rehydrating the data relies heavily on disk I/O and can certainly impact on aux copy times if your read times are slow.
With respect to aux copy and performance tunables I can share the following to provide some additional direction (Please forgive formatting)
AUXILIARY COPY DATA MOVEMENT OPTIMIZATION OPTIONS
An auxiliary copy operation allows you to create secondary copies of data associated with data protection operations, independent of the original copy.
Configure the auxiliary copy fallen behind alert for notification when the data to be copied for the associated storage policy exceeds the threshold and/or the number of days the jobs for the associated storage policy have not been copied exceeds the set threshold. This is a GUI-accessible parameter and is set in the Storage Policy Properties (Advanced) window.
Optimizing and Troubleshooting Auxiliary Copy Operations
- If the source to be copied resides on magnetic (disk) storage, increasing the Chunk Size will generally increase the performance.
- If the source to be copied resides on magnetic (disk) storage, the disk may be I/O constrained. Using the DiskRead utility on the Resource Pack CD, configured with the Non Backup API selection, check the disk performance for bottlenecking. If the resulting reads are the same as the Auxiliary Copy performance, the disk source is the performance bottleneck.
- Prior to v8, if the disk I/O is constrained, check for disk fragmentation. Fragmented disk is characterized by slow read operations. If the disk is fragmented, defragment the disk. To prevent disk fragmentation on a go forward basis add the DWORD registry value on each Media Agent attached to the Magnetic Library:
HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Platform Information\ControlSet001(machinename)\MediaAgent\nMagneticChunkFileIncrSize
Set the value of this key to 128
For v8 and above, this option is set in the GUI. This setting applies to all Windows-based Media Agents. See the screen capture below:
|Unix Media Agents do not support this configuration setting.
- If the source to be copied resides on tape, complete a Drive Validation operation and note the performance statistics for read and write throughputs from the tape device. When the drive validation is performed from the CommCell Console, the system performs all the operations that are necessary for operations. This includes operations such as mounting the media, writing on the media, re-winding and seeking data and then reading back from the media. If reads from the tape drives are the same as the Auxiliary Copy performance, check the following:
- Software and hardware compression are enabled for the jobs to be copied. Hardware compression, as completed, by the tape drive, will take longer to read software compressed operations.
- Review AuxCopy logs to determine if the same media is being mounted multiple times in order to complete the required read operation(s).
- Check if the source data is based on multiplexed writes. If the Auxiliary Copy type is selective, it is possible that the delay is due to read and discard operation of data from other archive files that are part of the ‘Plex that are not required to complete the Copy operation Also, check if the source chunk(s) is being read from multiple times. If this is the case, the Auxiliary Copy operation is trying to copy the data in a chunk, before all the backups which wrote to that chunk are finished.
CONFIGURATION / INFRASTRUCTURE
- Compare the number of MediaAgents being used as source and number of MediaAgents used as Destination for the Auxiliary Copy operations. If the ratio of readers to writers is not aligned, re-order jobs to load balance operations more evenly.
- Determine if the copy operation is LAN-Free or LAN-based. If LAN-based copy operations are constrained check for bottlenecks associated with the network configuration (NIC Duplexing) and consider upgrading the bandwidth of the LAN connection between readers and writers (i.e.; 100Base to 1000Base; 1000Base to 10000Base).
- Verify the number of streams used for the Auxiliary Copy operation and confirm that data movement operations are completing on all configured streams. If some of them are not transferring data copy operations will be compromised. Check if there are any errors reported on those streams. If the copy operation is not utilizing all destination streams even when there is data to be copied from the source streams, complete the following troubleshooting tasks:
a. Confirm that all resources necessary to complete the copy operation are available for the secondary target (tape drives, spare media, etc).
b. Enable Resource Manager logging and verify if the AuxcopyMgr.log displays any resource allocation issues for the streams which are not copied.
c. Use the getAuxCopySourceMediaSeq tool from the Resource Pack CD on the CommServe in order to determine what data is to be copied in the streams specified. This tool will output the list of source media from which specified copies are to be made.
d. Note that if there is a Combined Stream option is selected, a number of source streams will be mapped and copied to the given number of destination streams.
e. On the CommServe, create the following DWORD registry key under the CommServe Section of the registry:
HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Platform Information\ControlSet001(MachineName)\Commserve\AUXCOPY_SKIP_RESERVING_SEC
Set the value to 5. Suspend and resume the Auxiliary Copy operation.
- Verify if the Auxiliary Copy Source or Destination is waiting on resources during the copy operation. Make any necessary adjustments to eliminate resource contention on the affected host(s).
- Enable the following DWORD registry key on the CommServe:
HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Platform Information\ControlSet001(tangerine)\CommServe\ AUXCOPY_REPORT_PROGRESS_MB
Set the value to 4096. Changing this parameter reduces the number of updates transferred to the CommServe from the Media Agent completing the copy operation, reducing the overhead of those operations.
- Increase the Auxiliary Copy logging verbosity to include performance information by enabling the following DWord registry keys on the reader and writer Media Agents:
HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Platform Information\ControlSet001(tangerine)\MediaAgent\nAuxCopyCountersLogInterval
Set the value to 900
HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Platform Information\ControlSet001(tangerine)\MediaAgent\nDSBackupCountersLogInterval
Set the value to 900
With these registry keys defined, the log file will have more information on the average time taken for read and write operations to complete, allowing for more efficient troubleshooting.
7. For Auxiliary Copy operations based on non-deduplicated configurations (v8 and above) it is possible to increase performance through the use of the following DWord registry key on the reader and writer Media Agents. This key functions when the "Unbuffer I/O" option is set on the MountPaths and deduplication is not in use. This key is used to determine the read ahead in unbuffered I/O operations.
Set this key to a higher value to increase the read ahead.
Setting this key to a high value will cause increased memory usage by the process.
Higher values are recommended only for Auxiliary copies with a slow read performance using Unbuffered I/O mode.
<Instance Root>\MediaAgent dwMaxAsyncIoRequests
Auxiliary Copy Stream Randomization
When a storage policy is configured to use more than one data stream, stream randomization may be enabled through the GUI from the Storage Policy Properties (General) window. When this parameter is enabled, streams are randomly chosen to complete data copy operations, which evenly distributes the data across all the streams, thereby increasing the rate at which data is copied during auxiliary copy operations.