[SASAG] ESX + Debian 6 = high I/O

Mon Jul 18 12:40:08 PDT 2011

Unfortunately, I don't have access to the host, and all the ISP has to  
say is that VMWare is not the issue.

I'm not entirely surprised at their response. When we asked for a  
Linux install, they appear to have downloaded and installed a full  
desktop version of Debian.

They also originally provided a single cpu instance, with VMWare bound  
to 2 cpus, when we asked for 4 cpus!

As someone pointed out (and I missed) they also installed a 32-bit  
version.

For Apache, I don't think 32-bit would be much of an issue.
For mySQL, I assume it would just be a capacity/speed issue... but  
that wouldn't cause the problems we are seeing.

I'm requesting that we create separate servers for apache and mysql so  
we can isolate the services and maybe find out which load  
characteristics are  more problematic.

We found this link:
http://forums.cpanel.net/f5/extreme-load-peaks-vmware-server-158233.html

Which suggested rebooting daily and setting the I/O scheduler to noop.

Bill

Bill Levering
idbill at planx.com
KFP: 0C38 4D7E 5B50 94FE 992D  406D 6C81 DE33 5459 A1AC

On Jul 18, 2011, at 9:36 AM, Ski Kacoroski wrote:

> Bill,
>
> Just got back from vacation and was wondering if you figured this  
> out. If not, what you need to do is run esxtop on the esx host and  
> it will tell you where the problem lies.
>
> cheers,
>
> ski
>
On Jul 14, 2011, at 1:11 PM, Hazen, Jeff wrote:

> Have you tried running esxtop on the ESX host when the high system  
> load occurs?  Not knowing what else is going on in your environment,  
> it could be that it's having trouble allocating 4 CPUs  
> simultaneously to run each 'tick' of your VM.  In particular, look  
> for %CSTP - Percent coprocessor stop time - the amount of time that  
> a VM cannot run ONLY because the VM has multiple cores that cannot  
> be scheduled simultaneously because the ESX host is too busy. Over  
> 3% = potentially big problems.
>
> -Jeff
> On 07/14/2011 12:08 PM, Bill Levering wrote:
>>
>> I'm working on an Debian 6 server that is acting really strange.
>>
>> The VM is as follows:
>> 3 gig RAM (is now dedicated to this VM)
>> 4 cpu (Intel(R) Xeon(R) CPU X5680 @ 3.33GHz )
>> 165 Gig drive space on a Network Appliance connected via Fiber  
>> Channel
>>
>> OS: Linux 2.6.32-5-686
>> Debian: 6.0.1
>> Hardware: i686
>>
>> Services running:
>> mysql 5.1.49
>> apache 2.2
>> w/ libphp and mod_passenger
>>
>>
>> Load most of the time is between 5 and 8.
>> Overall Cpu usage is between 2-12%
>> Free RAM is between 80meg and 125meg
>> Only 90m of swap is active
>>
>> Memory usage:
>> - mysql ~ 1.2gig
>> - apache:
>> ### Processes: 27
>> ### Total private dirty RSS: 164.73 MB
>> ---- Passenger processes -----
>> ### Processes: 6
>> ### Total private dirty RSS: 66.25 MB
>> - php imports ~40meg
>>
>>
>> The box will spike to a load of around 20 when things start to stop
>> responding, but even then the overall CPU's usage never exceeds 35%.
>>
>> Under load iostat shows stats like the following:
>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await
>> svctm %util
>> sda 1.00 1216.00 11.00 135.00 984.00 10808.00 80.77 20.06 137.37  
>> 2.93 42.80
>> sda 0.00 137.00 65.00 973.00 1168.00 8880.00 9.68 0.95 0.92 0.76  
>> 79.20
>> sda 0.00 148.00 81.00 1414.00 704.00 12496.00 8.83 0.93 0.62 0.55  
>> 82.00
>> sda 1.00 165.00 42.00 1468.00 432.00 13064.00 8.94 1.04 0.69 0.55  
>> 83.20
>> sda 0.00 160.00 24.00 1125.00 192.00 13136.00 11.60 0.84 0.73 0.69  
>> 78.80
>> sda 0.00 148.00 56.00 1571.00 448.00 13752.00 8.73 0.71 0.44 0.44  
>> 70.80
>> sda 0.00 172.00 43.00 1327.00 376.00 11992.00 9.03 0.79 0.58 0.56  
>> 76.80
>> sda 0.00 142.00 55.00 1306.00 464.00 11584.00 8.85 0.82 0.61 0.53  
>> 72.00
>> sda 0.00 167.00 47.00 1490.00 376.00 13256.00 8.87 0.71 0.46 0.45  
>> 69.60
>>
>> mpstat shows:
>> # mpstat 1 20
>> Linux 2.6.32-5-686 (pmm) 07/13/2011 _i686_ (4 CPU)
>>
>> 01:21:45 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest  
>> %idle
>> 01:21:46 PM all 0.27 0.00 0.53 72.27 0.00 0.00 0.00 0.00 26.93
>> 01:21:47 PM all 0.73 0.00 0.49 72.37 0.00 0.00 0.00 0.00 26.41
>> 01:21:48 PM all 4.49 0.00 0.47 31.68 0.00 0.00 0.00 0.00 63.36
>> 01:21:49 PM all 8.15 0.00 0.99 42.22 0.00 0.25 0.00 0.00 48.40
>> 01:21:50 PM all 11.86 0.00 0.77 41.49 0.00 0.26 0.00 0.00 45.62
>> 01:21:51 PM all 7.07 0.00 0.25 38.38 0.00 0.00 0.00 0.00 54.29
>> 01:21:52 PM all 6.93 0.00 0.80 51.20 0.00 0.00 0.00 0.00 41.07
>> 01:21:53 PM all 6.83 0.00 0.68 35.54 0.00 0.00 0.00 0.00 56.95
>> 01:21:54 PM all 10.39 0.00 0.00 32.21 0.00 0.00 0.00 0.00 57.40
>> 01:21:55 PM all 6.13 0.00 0.24 37.03 0.00 0.00 0.00 0.00 56.60
>> 01:21:56 PM all 16.14 0.00 0.49 35.21 0.00 0.00 0.00 0.00 48.17
>> 01:21:57 PM all 11.68 0.00 0.25 39.09 0.25 0.00 0.00 0.00 48.73
>> 01:21:58 PM all 12.29 0.00 1.18 41.37 0.00 0.00 0.00 0.00 45.15
>> 01:21:59 PM all 9.47 0.00 0.79 51.32 0.00 0.26 0.00 0.00 38.16
>> 01:22:00 PM all 9.52 0.00 0.71 36.43 0.00 0.00 0.00 0.00 53.33
>> 01:22:01 PM all 14.65 0.00 0.76 35.10 0.00 0.00 0.00 0.00 49.49
>> 01:22:02 PM all 12.05 0.00 0.48 40.96 0.00 0.00 0.00 0.00 46.51
>> 01:22:03 PM all 7.57 0.00 0.52 33.94 0.00 0.00 0.00 0.00 57.96
>> 01:22:04 PM all 6.45 0.00 0.46 37.10 0.00 0.00 0.00 0.00 55.99
>> 01:22:05 PM all 16.45 0.00 0.51 35.99 0.00 0.00 0.00 0.00 47.04
>> Average: all 8.94 0.00 0.57 41.86 0.01 0.04 0.00 0.00 48.57
>> # w
>> 13:22:05 up 48 days, 12:43, 2 users, load average: 12.00, 11.50,  
>> 10.10
>>
>> More recently, I've tweaked mysql quite bit and switched the mysql
>> engine from myisam to innodb, but the developer is convinced that is
>> making things worse, so we are switching it back to myisam tonight.
>>
>> The imports are not threaded, so they create quite a bit of load, but
>> the developer doesn't have the time or the impetus to fix this. (The
>> import does a select, then based on response, does a delete, then  
>> add...
>> based on a parsed xml file, for about 35-40 minutes as fast as the
>> single thread can go.)
>>
>> BUT, this same setup was on Slicehost (XEN) and in 2 years, we've  
>> never
>> had these problems.
>>
>> I keep coming back to an i/o issue, but don't know how to  
>> troubleshoot
>> from there.
>>
>> Am I missing something?
>> Do you have any other suggestions?
>>
>> Bill
>>
>> Bill Levering
>> idbill at planx.com
>> KFP: 0C38 4D7E 5B50 94FE 992D 406D 6C81 DE33 5459 A1AC
>>
>> PS. I must confess, this is my second run-in with ESX(i) and my last
>> experience was pretty much identical. VMWare says the server has  
>> room to
>> run, but Guest runs like crap and is unusable. I now dread working on
>> any project that even mentions VMWare.
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Members mailing list
>> Members at lists.sasag.org
>> http://lists.sasag.org/mailman/listinfo/members
>
> -- 
> "When we try to pick out anything by itself, we find it
> connected to the entire universe"            John Muir
>
> Chris "Ski" Kacoroski, kacoroski at gmail.com, 206-501-9803
> or ski98033 on most IM services

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 203 bytes
Desc: This is a digitally signed message part
URL: <http://lists.sasag.org/pipermail/members/attachments/20110718/3722be5d/attachment.bin>