I saw in a poll taken for a recent SearchServerVirtualization.com article that input/output, or I/O bottlenecks are the number one challenge for VMware users. This didn’t really surprise me because I/O bottlenecks are sometimes not that obvious and can be difficult to find and troubleshoot. Why is that? Because most users don’t understand how to look for them or how to interpret the data that is presented to them from monitoring utilities that would indicate a bottleneck. Some of the most common causes of I/O bottlenecks are improperly architected/configured hosts and network/storage devices, too many high disk I/O VM’s on a single host or LUN and excessive use of vSMP.
So how do you find I/O bottlenecks? You start with the built-in performance monitoring tools in the vSphere Client on the Performance tab. The esxtop command line utility can also be very helpful for looking at real-time metrics.
Using the Windows built-in Performance Monitor (perfmon) utility is usually not recommended, because some statistics may not be accurate because they are not aware of the underlying virtualization layer. VMware, however, has recently added some VM specific performance counters to perfmon that are aware of the virtualization layer and provide accurate metrics. There is also a utility introduced in ESX 3.5 called vscsiStats that can be used for storage performance analysis.
Understanding how to properly use these built-in tools to collect data and interpret it can take some practice. There are a number of free and paid third-party tools that can help you with this such as Reflex Systems VMC, Veeam Monitor, Vizioncore vFoglight, eG Monitor, Hyper9 and Netuitive Si.
If you’d like to try and learn more about performance troubleshooting that will help you identify I/O bottlenecks, check out some of the below resources:
- Understanding Performance
- Using esxtop to Troubleshoot Performance Problems
- Interpreting esxtop Statistics
- Using the esxtop tool to identify VMware ESX memory use
- Using Perfmon For Accurate, ESX Performance Counters
- Performance Troubleshooting for VMware vSphere 4 and ESX 4.0
- Understanding VirtualCenter Performance Statistics
- Using vscsiStats for Storage Performance Analysis
- Performance Analysis Methods
- CPU Performance Analysis and Monitoring
- Memory Performance Analysis and Monitoring
- Network Performance Analysis and Monitoring
- Storage System Performance Analysis with Iometer
- VMware Infrastructure – Performance Optimization & Troubleshooting (VMware ESX / VMware ESXi) (VMworld 2008 – free registration required)
- Intepreting Performance Statistics in VI3 (VMworld 2008 – free registration required)
- Troubleshooting virtual machine performance issues (KB Article)
- Using esxtop to identify storage performance issues (KB Article)