[SASAG] ESX + Debian 6 = high I/O... conclusion
Bill Levering
idbill at planx.com
Sun Jul 24 15:27:45 PDT 2011
Yeah, VMWare tools were installed. One of the things it appears the
ASP had to do was custom compile the VMWare v8.3.7 tools (vmstoolsd
and vmmemctl), because the source is in /usr/local/src.
This is odd, because there is a package that provides this: (which is
even a later version 8.4.2)
While running top, I noticed that vmmemctl was in the top 5 daemons
listed when ordered by cpu usage.
At first I thought the server was acting like it was in swap hell, but
that didn't make any sense, since no swap was being used. Then I
wondered if ESX was swapping memory underneath the guest...
When I brought this up with the ASP, they implied that the resource
pool of memory could not accommodate all of their vm requirements at
one time. (AKA, they were overselling their resources.) But they
agreed to dedicate 3gig of RAM to our VM directly.
Bill
Bill Levering
idbill at planx.com
KFP: 0C38 4D7E 5B50 94FE 992D 406D 6C81 DE33 5459 A1AC
On Jul 24, 2011, at 11:18 AM, Nick Webb wrote:
> Thanks for the post, interesting read. Was vmware tools installed on
> the guest? I'd think that should take care of this?
>
> Nick
>
> Sent from my mobile device.
>
> On Jul 22, 2011, at 11:24 AM, Bill Levering <idbill at planx.com> wrote:
>
>> For those of you following this... here is a little write-up about
>> the experience.
>>
>> http://pugetworks.com/2011/07/vmware-and-ubuntu-high-load-issues/
>>
>> Comments are disabled, but I'm open to receiving comments directly.
>>
>> Thanks,
>> Bill
>>
>> Bill Levering
>> idbill at planx.com
>> KFP: 0C38 4D7E 5B50 94FE 992D 406D 6C81 DE33 5459 A1AC
>>
>>
>>
>>
>> On Jul 18, 2011, at 9:39 PM, Justin Huff wrote:
>>
>>> Looks to me like IO *is* your problem:)
>>>
>>> IOWait is pretty high as is IOUtil. I'm not sure what the workload
>>> looks like, but IOops don't look bad. queue size makes sense for a
>>> single threaded process. Service times don't look too bad either.
>>>
>>> If I were to guess, I think you might be expecting more IO out of
>>> this
>>> VM than it can do. Perhaps the provider can allocate more spindles
>>> or
>>> something.
>>>
>>> --Justin
>>>
>>> On Thu, Jul 14, 2011 at 12:08 PM, Bill Levering <idbill at planx.com>
>>> wrote:
>>>>
>>>> I'm working on an Debian 6 server that is acting really strange.
>>>>
>>>> The VM is as follows:
>>>> 3 gig RAM (is now dedicated to this VM)
>>>> 4 cpu (Intel(R) Xeon(R) CPU X5680 @ 3.33GHz )
>>>> 165 Gig drive space on a Network Appliance connected via Fiber
>>>> Channel
>>>>
>>>> OS: Linux 2.6.32-5-686
>>>> Debian: 6.0.1
>>>> Hardware: i686
>>>>
>>>> Services running:
>>>> mysql 5.1.49
>>>> apache 2.2
>>>> w/ libphp and mod_passenger
>>>>
>>>>
>>>> Load most of the time is between 5 and 8.
>>>> Overall Cpu usage is between 2-12%
>>>> Free RAM is between 80meg and 125meg
>>>> Only 90m of swap is active
>>>>
>>>> Memory usage:
>>>> - mysql ~ 1.2gig
>>>> - apache:
>>>> ### Processes: 27
>>>> ### Total private dirty RSS: 164.73 MB
>>>> ---- Passenger processes -----
>>>> ### Processes: 6
>>>> ### Total private dirty RSS: 66.25 MB
>>>> - php imports ~40meg
>>>>
>>>>
>>>> The box will spike to a load of around 20 when things start to stop
>>>> responding, but even then the overall CPU's usage never exceeds
>>>> 35%.
>>>>
>>>> Under load iostat shows stats like the following:
>>>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
>>>> avgqu-sz await svctm %util
>>>> sda 1.00 1216.00 11.00 135.00 984.00 10808.00
>>>> 80.77
>>>> 20.06 137.37 2.93 42.80
>>>> sda 0.00 137.00 65.00 973.00 1168.00 8880.00 9.68
>>>> 0.95 0.92
>>>> 0.76 79.20
>>>> sda 0.00 148.00 81.00 1414.00 704.00 12496.00
>>>> 8.83 0.93
>>>> 0.62 0.55 82.00
>>>> sda 1.00 165.00 42.00 1468.00 432.00 13064.00
>>>> 8.94 1.04
>>>> 0.69 0.55 83.20
>>>> sda 0.00 160.00 24.00 1125.00 192.00 13136.00
>>>> 11.60 0.84
>>>> 0.73 0.69 78.80
>>>> sda 0.00 148.00 56.00 1571.00 448.00 13752.00
>>>> 8.73 0.71
>>>> 0.44 0.44 70.80
>>>> sda 0.00 172.00 43.00 1327.00 376.00 11992.00
>>>> 9.03 0.79
>>>> 0.58 0.56 76.80
>>>> sda 0.00 142.00 55.00 1306.00 464.00 11584.00
>>>> 8.85 0.82
>>>> 0.61 0.53 72.00
>>>> sda 0.00 167.00 47.00 1490.00 376.00 13256.00
>>>> 8.87 0.71
>>>> 0.46 0.45 69.60
>>>>
>>>> mpstat shows:
>>>> # mpstat 1 20
>>>> Linux 2.6.32-5-686 (pmm) 07/13/2011 _i686_ (4 CPU)
>>>>
>>>> 01:21:45 PM CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal
>>>> %guest %idle
>>>> 01:21:46 PM all 0.27 0.00 0.53 72.27 0.00
>>>> 0.00 0.00
>>>> 0.00 26.93
>>>> 01:21:47 PM all 0.73 0.00 0.49 72.37 0.00
>>>> 0.00 0.00
>>>> 0.00 26.41
>>>> 01:21:48 PM all 4.49 0.00 0.47 31.68 0.00
>>>> 0.00 0.00
>>>> 0.00 63.36
>>>> 01:21:49 PM all 8.15 0.00 0.99 42.22 0.00
>>>> 0.25 0.00
>>>> 0.00 48.40
>>>> 01:21:50 PM all 11.86 0.00 0.77 41.49 0.00
>>>> 0.26 0.00
>>>> 0.00 45.62
>>>> 01:21:51 PM all 7.07 0.00 0.25 38.38 0.00
>>>> 0.00 0.00
>>>> 0.00 54.29
>>>> 01:21:52 PM all 6.93 0.00 0.80 51.20 0.00
>>>> 0.00 0.00
>>>> 0.00 41.07
>>>> 01:21:53 PM all 6.83 0.00 0.68 35.54 0.00
>>>> 0.00 0.00
>>>> 0.00 56.95
>>>> 01:21:54 PM all 10.39 0.00 0.00 32.21 0.00
>>>> 0.00 0.00
>>>> 0.00 57.40
>>>> 01:21:55 PM all 6.13 0.00 0.24 37.03 0.00
>>>> 0.00 0.00
>>>> 0.00 56.60
>>>> 01:21:56 PM all 16.14 0.00 0.49 35.21 0.00
>>>> 0.00 0.00
>>>> 0.00 48.17
>>>> 01:21:57 PM all 11.68 0.00 0.25 39.09 0.25
>>>> 0.00 0.00
>>>> 0.00 48.73
>>>> 01:21:58 PM all 12.29 0.00 1.18 41.37 0.00
>>>> 0.00 0.00
>>>> 0.00 45.15
>>>> 01:21:59 PM all 9.47 0.00 0.79 51.32 0.00
>>>> 0.26 0.00
>>>> 0.00 38.16
>>>> 01:22:00 PM all 9.52 0.00 0.71 36.43 0.00
>>>> 0.00 0.00
>>>> 0.00 53.33
>>>> 01:22:01 PM all 14.65 0.00 0.76 35.10 0.00
>>>> 0.00 0.00
>>>> 0.00 49.49
>>>> 01:22:02 PM all 12.05 0.00 0.48 40.96 0.00
>>>> 0.00 0.00
>>>> 0.00 46.51
>>>> 01:22:03 PM all 7.57 0.00 0.52 33.94 0.00
>>>> 0.00 0.00
>>>> 0.00 57.96
>>>> 01:22:04 PM all 6.45 0.00 0.46 37.10 0.00
>>>> 0.00 0.00
>>>> 0.00 55.99
>>>> 01:22:05 PM all 16.45 0.00 0.51 35.99 0.00
>>>> 0.00 0.00
>>>> 0.00 47.04
>>>> Average: all 8.94 0.00 0.57 41.86 0.01
>>>> 0.04 0.00
>>>> 0.00 48.57
>>>> # w
>>>> 13:22:05 up 48 days, 12:43, 2 users, load average: 12.00,
>>>> 11.50, 10.10
>>>>
>>>> More recently, I've tweaked mysql quite bit and switched the
>>>> mysql engine
>>>> from myisam to innodb, but the developer is convinced that is
>>>> making things
>>>> worse, so we are switching it back to myisam tonight.
>>>>
>>>> The imports are not threaded, so they create quite a bit of load,
>>>> but the
>>>> developer doesn't have the time or the impetus to fix this. (The
>>>> import does
>>>> a select, then based on response, does a delete, then add...
>>>> based on a
>>>> parsed xml file, for about 35-40 minutes as fast as the single
>>>> thread can
>>>> go.)
>>>>
>>>> BUT, this same setup was on Slicehost (XEN) and in 2 years, we've
>>>> never had
>>>> these problems.
>>>>
>>>> I keep coming back to an i/o issue, but don't know how to
>>>> troubleshoot from
>>>> there.
>>>>
>>>> Am I missing something?
>>>> Do you have any other suggestions?
>>>>
>>>> Bill
>>>>
>>>> Bill Levering
>>>> idbill at planx.com
>>>> KFP: 0C38 4D7E 5B50 94FE 992D 406D 6C81 DE33 5459 A1AC
>>>>
>>>> PS. I must confess, this is my second run-in with ESX(i) and my
>>>> last
>>>> experience was pretty much identical. VMWare says the server has
>>>> room to
>>>> run, but Guest runs like crap and is unusable. I now dread
>>>> working on any
>>>> project that even mentions VMWare.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Members mailing list
>>>> Members at lists.sasag.org
>>>> http://lists.sasag.org/mailman/listinfo/members
>>>>
>>
>> _______________________________________________
>> Members mailing list
>> Members at lists.sasag.org
>> http://lists.sasag.org/mailman/listinfo/members
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 203 bytes
Desc: This is a digitally signed message part
URL: <http://lists.sasag.org/pipermail/members/attachments/20110724/b806acbb/attachment.bin>
More information about the Members
mailing list