High Load after server update.. CentOS 5.2 to 5.3

[dognose]

Hi all,

I'm seeing very high load on my server after server update (Centos 5 x86_64) today.. I haven't updated IM (ImageMagick 6.4.8-7 2009-01-22 Q16 OpenMP)
(I'm running on a multi core machine)

From the initial tests I've done, it seems that it has something to do with multiple convert operations running at the same time, although, I haven't figured out specific examples..

if I run
convert ... &
convert ... &
wait

it takes MUCH longer than if I run:

convert ...
convert ...

Does anyone know what might be going on? I'm thinking it's something with OpenMP or weird memory locking? I really don't know the internals of what's going on, but I don't know that this is a SIGNIFICANT degradation in performance.

an example of the operation I'm performing, but it really seems to be happening with loads of different processes.

convert imagein.png -bordercolor white -border 10 -bordercolor grey60 -border 1 -background none -rotate 5 -background black \( +clone -shadow 60x4+4+4 \) +swap -background transparent -flatten imageout.png

[magick]

Try disabling OpenMP and see if that makes a difference (e.g. ./configure --disable-openmp). Recent releases of ImageMagick have a number of performance improvements so one would expect your commands to run faster, not slower.

[dognose]

Well, I'd love to, but, this was my stable IM version. I always have trouble compiling the new versions.

Just wondering if anyone else is having problems with concurrency issues or whatnot with os updates.

I do appreciate the OpenMP.. so, I hope that's not fallen by the way side.

[dognose]

Found a log of what was updated... Does anything here look like it could cause trouble like this?

Jun 22 11:14:26 Updated: libgcc - 4.1.2-44.el5.x86_64
Jun 22 11:14:26 Updated: libgcc - 4.1.2-44.el5.i386
Jun 22 11:14:30 Updated: tzdata - 2009i-2.el5.noarch
Jun 22 11:14:46 Updated: glibc-common - 2.5-34.x86_64
Jun 22 11:14:48 Updated: glibc - 2.5-34.i686
Jun 22 11:14:53 Updated: glibc - 2.5-34.x86_64
Jun 22 11:14:54 Updated: bash - 3.2-24.el5.x86_64
Jun 22 11:14:54 Updated: audit-libs - 1.7.7-6.el5_3.3.x86_64
Jun 22 11:14:57 Updated: perl - 4:5.8.8-18.el5_3.1.x86_64
Jun 22 11:14:58 Updated: glib2 - 2.12.3-4.el5_3.1.i386
Jun 22 11:14:58 Updated: glib2 - 2.12.3-4.el5_3.1.x86_64
Jun 22 11:14:58 Updated: popt - 1.10.2.3-9.el5.x86_64
Jun 22 11:14:58 Updated: tcp_wrappers - 7.6-40.6.el5.x86_64
Jun 22 11:14:59 Updated: freetype - 2.2.1-21.el5_3.x86_64
Jun 22 11:14:59 Updated: libpng - 2:1.2.10-7.1.el5_3.2.x86_64
Jun 22 11:14:59 Updated: elfutils-libelf - 0.137-3.el5.x86_64
Jun 22 11:14:59 Updated: file - 4.17-15.el5_3.1.x86_64
Jun 22 11:14:59 Updated: libpng - 2:1.2.10-7.1.el5_3.2.i386
Jun 22 11:15:00 Updated: libstdc++ - 4.1.2-44.el5.x86_64
Jun 22 11:15:00 Updated: libgcrypt - 1.2.4-1.el5.x86_64
Jun 22 11:15:00 Updated: php-common - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:15:00 Updated: procps - 3.2.7-11.1.el5.x86_64
Jun 22 11:15:00 Updated: libvolume_id - 095-14.20.el5_3.x86_64
Jun 22 11:15:01 Updated: freetype - 2.2.1-21.el5_3.i386
Jun 22 11:15:01 Updated: libgcrypt - 1.2.4-1.el5.i386
Jun 22 11:15:01 Updated: audit-libs - 1.7.7-6.el5_3.3.i386
Jun 22 11:15:02 Updated: iproute - 2.6.18-9.el5.x86_64
Jun 22 11:15:02 Updated: popt - 1.10.2.3-9.el5.i386
Jun 22 11:15:02 Updated: perl-DBI - 1.52-2.el5.x86_64
Jun 22 11:15:03 Updated: lm_sensors - 2.10.7-4.el5.x86_64
Jun 22 11:15:03 Updated: libstdc++ - 4.1.2-44.el5.i386
Jun 22 11:15:03 Updated: libvolume_id - 095-14.20.el5_3.i386
Jun 22 11:15:03 Updated: php-pdo - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:15:04 Updated: iputils - 20020927-45.el5.x86_64
Jun 22 11:15:05 Updated: binutils - 2.17.50.0.6-9.el5.x86_64
Jun 22 11:15:05 Updated: ethtool - 6-2.el5.x86_64
Jun 22 11:15:05 Updated: tcp_wrappers - 7.6-40.6.el5.i386
Jun 22 11:15:05 Updated: nash - 5.1.19.6-44.1.x86_64
Jun 22 11:15:05 Updated: centos-release-notes - 5.3-3.x86_64
Jun 22 11:15:05 Updated: centos-release - 10:5-3.el5.centos.1.x86_64
Jun 22 11:15:05 Updated: setup - 2.5.58-4.el5.noarch
Jun 22 11:15:06 Updated: libwmf - 0.2.8.4-10.2.x86_64
Jun 22 11:15:06 Updated: pango - 1.14.9-5.el5.centos.x86_64
Jun 22 11:15:07 Updated: ORBit2 - 2.14.3-5.el5.x86_64
Jun 22 11:15:07 Updated: numactl - 0.9.8-7.el5.x86_64
Jun 22 11:15:07 Updated: libgomp - 4.3.2-7.el5.x86_64
Jun 22 11:15:07 Updated: lcms - 1.18-0.1.beta1.el5_3.2.x86_64
Jun 22 11:15:07 Installed: avahi-compat-libdns_sd - 0.6.16-1.el5_2.1.x86_64
Jun 22 11:15:08 Updated: crash - 4.0-7.2.3.el5.centos.1.x86_64
Jun 22 11:15:08 Updated: php-gd - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:15:08 Updated: php-xml - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:15:08 Updated: ypbind - 3:1.19-11.el5.x86_64
Jun 22 11:15:08 Updated: unzip - 5.52-3.el5.x86_64
Jun 22 11:15:09 Updated: cpuspeed - 1:1.2.1-5.el5.x86_64
Jun 22 11:15:09 Updated: tmpwatch - 2.9.7-1.1.el5.2.x86_64
Jun 22 11:15:09 Updated: traceroute - 3:2.0.1-5.el5.x86_64
Jun 22 11:15:09 Updated: libhugetlbfs - 1.3-3.el5.x86_64
Jun 22 11:15:09 Installed: giflib - 4.1.3-7.1.el5_3.1.x86_64
Jun 22 11:15:09 Updated: dosfstools - 2.11-7.el5.x86_64
Jun 22 11:15:09 Updated: rdate - 1.4-8.el5.x86_64
Jun 22 11:15:09 Updated: zip - 2.31-2.el5.x86_64
Jun 22 11:15:09 Updated: patch - 2.5.4-29.2.3.el5.x86_64
Jun 22 11:15:10 Updated: ftp - 0.17-35.el5.x86_64
Jun 22 11:15:10 Updated: libwmf - 0.2.8.4-10.2.i386
Jun 22 11:15:12 Updated: pango - 1.14.9-5.el5.centos.i386
Jun 22 11:15:19 Updated: ORBit2 - 2.14.3-5.el5.i386
Jun 22 11:15:20 Updated: numactl - 0.9.8-7.el5.i386
Jun 22 11:15:24 Updated: libgomp - 4.3.2-7.el5.i386
Jun 22 11:15:24 Updated: lcms - 1.18-0.1.beta1.el5_3.2.i386
Jun 22 11:15:25 Updated: libhugetlbfs - 1.3-3.el5.i386
Jun 22 11:15:26 Updated: filesystem - 2.4.0-2.el5.centos.x86_64
Jun 22 11:15:27 Updated: redhat-menus - 6.7.8-3.el5.noarch
Jun 22 11:15:28 Updated: redhat-logos - 4.9.99-11.el5.centos.noarch
Jun 22 11:15:31 Updated: man-pages - 2.39-12.el5.noarch
Jun 22 11:15:31 Updated: words - 3.0-9.1.noarch
Jun 22 11:15:32 Updated: libselinux - 1.33.4-5.1.el5.i386
Jun 22 11:15:34 Updated: device-mapper - 1.02.28-2.el5.i386
Jun 22 11:15:36 Updated: e2fsprogs-libs - 1.39-20.el5.i386
Jun 22 11:15:37 Updated: cryptsetup-luks - 1.0.3-4.el5.i386
Jun 22 11:15:39 Updated: parted - 1.8.1-23.el5.i386
Jun 22 11:15:44 Updated: mesa-libGL - 6.5.1-7.7.el5.i386
Jun 22 11:15:51 Updated: libselinux - 1.33.4-5.1.el5.x86_64
Jun 22 11:15:51 Updated: shadow-utils - 2:4.0.17-14.el5.x86_64
Jun 22 11:15:52 Updated: device-mapper - 1.02.28-2.el5.x86_64
Jun 22 11:15:52 Updated: e2fsprogs-libs - 1.39-20.el5.x86_64
Jun 22 11:15:52 Updated: findutils - 1:4.2.27-5.el5.x86_64
Jun 22 11:15:55 Updated: coreutils - 5.97-19.el5.x86_64
Jun 22 11:15:56 Updated: krb5-libs - 1.6.1-31.el5_3.3.x86_64
Jun 22 11:15:56 Updated: openssl - 0.9.8e-7.el5.x86_64
Jun 22 11:15:59 Updated: python - 2.4.3-24.el5.x86_64
Jun 22 11:16:13 Updated: pam - 0.99.6.2-4.el5.x86_64
Jun 22 11:16:13 Updated: openldap - 2.3.43-3.el5.x86_64
Jun 22 11:16:13 Updated: module-init-tools - 3.3-0.pre3.1.42.el5.x86_64
Jun 22 11:16:17 Updated: curl - 7.15.5-2.1.el5_3.4.x86_64
Jun 22 11:16:17 Updated: kpartx - 0.4.7-23.el5_3.4.x86_64
Jun 22 11:16:17 Updated: e2fsprogs - 1.39-20.el5.x86_64
Jun 22 11:16:21 Updated: newt - 0.52.2-12.el5.x86_64
Jun 22 11:16:21 Updated: cups-libs - 1:1.3.7-8.el5_3.6.x86_64
Jun 22 11:16:24 Updated: cryptsetup-luks - 1.0.3-4.el5.x86_64
Jun 22 11:16:24 Updated: dmraid - 1.0.0.rc13-33.el5.x86_64
Jun 22 11:16:25 Updated: device-mapper-multipath - 0.4.7-23.el5_3.4.x86_64
Jun 22 11:16:27 Updated: nscd - 2.5-34.x86_64
Jun 22 11:16:27 Updated: apr-util - 1.2.7-7.el5_3.1.x86_64
Jun 22 11:16:30 Updated: bind-libs - 30:9.3.4-10.P1.el5.x86_64
Jun 22 11:16:34 Updated: libpcap - 14:0.9.4-14.el5.x86_64
Jun 22 11:16:42 Updated: net-snmp-libs - 1:5.3.2.2-5.el5_3.1.x86_64
Jun 22 11:16:42 Updated: logrotate - 3.7.4-9.x86_64
Jun 22 11:16:42 Updated: audit-libs-python - 1.7.7-6.el5_3.3.x86_64
Jun 22 11:16:42 Updated: libselinux-python - 1.33.4-5.1.el5.x86_64
Jun 22 11:16:43 Updated: m2crypto - 0.16-6.el5.3.x86_64
Jun 22 11:16:43 Updated: device-mapper-event - 1.02.28-2.el5.x86_64
Jun 22 11:17:35 Updated: lvm2 - 2.02.40-6.el5.x86_64
Jun 22 11:17:37 Updated: ghostscript - 8.15.2-9.4.el5_3.7.x86_64
Jun 22 11:17:37 Updated: php-cli - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:17:38 Updated: pam_krb5 - 2.2.14-10.x86_64
Jun 22 11:17:39 Updated: ecryptfs-utils - 56-8.el5.x86_64
Jun 22 11:17:39 Updated: perl-DBD-MySQL - 3.0007-2.el5.x86_64
Jun 22 11:17:45 Installed: fipscheck - 1.0.3-1.el5.x86_64
Jun 22 11:17:47 Updated: nss_ldap - 253-17.el5.x86_64
Jun 22 11:17:47 Updated: neon - 0.25.5-10.el5.x86_64
Jun 22 11:17:48 Updated: alsa-lib - 1.0.17-1.el5.x86_64
Jun 22 11:17:48 Updated: parted - 1.8.1-23.el5.x86_64
Jun 22 11:17:48 Updated: libutempter - 1.1.4-4.el5.x86_64
Jun 22 11:17:50 Updated: mesa-libGL - 6.5.1-7.7.el5.x86_64
Jun 22 11:17:50 Installed: libselinux-utils - 1.33.4-5.1.el5.x86_64
Jun 22 11:17:50 Updated: bind-utils - 30:9.3.4-10.P1.el5.x86_64
Jun 22 11:17:51 Updated: gnupg - 1.4.5-14.x86_64
Jun 22 11:17:52 Updated: autofs - 1:5.0.1-0.rc2.102.x86_64
Jun 22 11:17:52 Updated: stunnel - 4.15-2.el5.1.x86_64
Jun 22 11:17:52 Updated: php-mysql - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:17:53 Updated: tcpdump - 14:3.9.4-14.el5.x86_64
Jun 22 11:17:53 Updated: pkinit-nss - 0.7.6-1.el5.x86_64
Jun 22 11:17:53 Updated: ksh - 20080202-2.el5.x86_64
Jun 22 11:17:54 Updated: audit - 1.7.7-6.el5_3.3.x86_64
Jun 22 11:17:54 Updated: psacct - 6.3.2-44.el5.x86_64
Jun 22 11:17:54 Updated: tcsh - 6.14-14.el5.x86_64
Jun 22 11:17:54 Updated: mlocate - 0.15-1.el5.1.x86_64
Jun 22 11:17:56 Updated: pam - 0.99.6.2-4.el5.i386
Jun 22 11:17:57 Updated: udev - 095-14.20.el5_3.x86_64
Jun 22 11:17:58 Updated: util-linux - 2.13-0.50.el5.x86_64
Jun 22 11:17:58 Updated: usermode - 1.88-3.el5.2.x86_64
Jun 22 11:17:59 Updated: krb5-libs - 1.6.1-31.el5_3.3.i386
Jun 22 11:17:59 Updated: authconfig - 5.3.21-5.el5.x86_64
Jun 22 11:18:00 Updated: SysVinit - 2.86-15.el5.x86_64
Jun 22 11:18:01 Updated: initscripts - 8.45.25-1.el5.centos.x86_64
Jun 22 11:18:01 Updated: mcstrans - 0.2.11-3.el5.x86_64
Jun 22 11:18:02 Updated: dhclient - 12:3.0.5-18.el5.x86_64
Jun 22 11:18:02 Updated: openssh - 4.3p2-29.el5.x86_64
Jun 22 11:18:03 Updated: httpd - 2.2.3-22.el5.centos.1.x86_64
Jun 22 11:18:04 Updated: kbd - 1.12-21.el5.x86_64
Jun 22 11:18:04 Updated: mkinitrd - 5.1.19.6-44.1.x86_64
Jun 22 11:18:04 Installed: dnsmasq - 2.45-1.el5_2.1.x86_64
Jun 22 11:18:05 Updated: ppp - 2.4.4-2.el5.x86_64
Jun 22 11:18:05 Updated: openssl - 0.9.8e-7.el5.i686
Jun 22 11:18:05 Updated: cups-libs - 1:1.3.7-8.el5_3.6.i386
Jun 22 11:18:06 Updated: openldap - 2.3.43-3.el5.i386
Jun 22 11:18:07 Updated: ghostscript - 8.15.2-9.4.el5_3.7.i386
Jun 22 11:18:07 Installed: fipscheck - 1.0.3-1.el5.i386
Jun 22 11:18:08 Updated: nss_ldap - 253-17.el5.i386
Jun 22 11:18:08 Updated: ecryptfs-utils - 56-8.el5.i386
Jun 22 11:18:08 Updated: pam_krb5 - 2.2.14-10.i386
Jun 22 11:18:09 Updated: libutempter - 1.1.4-4.el5.i386
Jun 22 11:18:09 Updated: python-urlgrabber - 3.1.0-5.el5.noarch
Jun 22 11:18:09 Updated: hwdata - 0.213.11-1.el5.noarch
Jun 22 11:18:09 Updated: php - 5.1.6-23.2.el5_3.x86_64
Jun 22 11:18:09 Updated: openssh-clients - 4.3p2-29.el5.x86_64
Jun 22 11:18:10 Updated: openssh-server - 4.3p2-29.el5.x86_64
Jun 22 11:18:10 Updated: ipsec-tools - 0.6.5-13.el5_3.1.x86_64
Jun 22 11:18:10 Updated: ccid - 1.3.8-1.el5.x86_64
Jun 22 11:18:11 Updated: vixie-cron - 4:4.1-76.el5.x86_64
Jun 22 11:18:11 Updated: acpid - 1.0.4-7.el5_3.1.x86_64
Jun 22 11:18:11 Updated: smartmontools - 1:5.38-2.el5.x86_64
Jun 22 11:18:12 Updated: dhcpv6-client - 1.0.10-16.el5.x86_64
Jun 22 11:18:12 Updated: quota - 1:3.13-1.2.5.el5.x86_64
Jun 22 11:18:13 Updated: krb5-workstation - 1.6.1-31.el5_3.3.x86_64
Jun 22 11:18:15 Updated: nfs-utils - 1:1.0.9-40.el5.x86_64
Jun 22 11:18:17 Updated: prelink - 0.4.0-2.el5.x86_64
Jun 22 11:18:17 Updated: sudo - 1.6.9p17-3.el5_3.1.x86_64
Jun 22 11:18:18 Updated: mkinitrd - 5.1.19.6-44.1.i386
Jun 22 11:20:30 Installed: kernel - 2.6.18-128.1.14.el5.x86_64
Jun 22 11:20:31 Updated: sos - 1.7-9.16.el5_3.5.noarch
Jun 22 11:20:31 Updated: rpm-libs - 4.4.2.3-9.el5.x86_64
Jun 22 11:20:32 Updated: rpm - 4.4.2.3-9.el5.x86_64
Jun 22 11:20:32 Updated: rpm-python - 4.4.2.3-9.el5.x86_64
Jun 22 11:20:34 Updated: policycoreutils - 1.33.12-14.2.el5.x86_64
Jun 22 11:20:35 Updated: pm-utils - 0.99.3-10.el5.centos.x86_64
Jun 22 11:20:35 Updated: dbus - 1.1.2-12.el5.i386
Jun 22 11:20:35 Installed: dbus-libs - 1.1.2-12.el5.x86_64
Jun 22 11:20:36 Updated: dbus-glib - 0.73-8.el5.x86_64
Jun 22 11:20:37 Updated: hal - 0.5.8.1-38.el5.x86_64
Jun 22 11:20:37 Installed: dbus-libs - 1.1.2-12.el5.i386
Jun 22 11:20:37 Updated: kudzu - 1.2.57.1.21-1.el5.centos.x86_64
Jun 22 11:20:37 Updated: wpa_supplicant - 1:0.5.10-8.el5.x86_64
Jun 22 11:20:38 Updated: dhcdbd - 2.2-2.el5.x86_64
Jun 22 11:20:38 Updated: dbus - 1.1.2-12.el5.x86_64
Jun 22 11:20:40 Updated: cups - 1:1.3.7-8.el5_3.6.x86_64
Jun 22 11:20:41 Updated: net-snmp - 1:5.3.2.2-5.el5_3.1.x86_64
Jun 22 11:20:41 Updated: dbus-glib - 0.73-8.el5.i386
Jun 22 11:20:42 Updated: system-config-network-tui - 1.3.99.12-1.el5.noarch
Jun 22 11:20:43 Updated: hal - 0.5.8.1-38.el5.i386
Jun 22 11:20:44 Updated: selinux-policy - 2.4.6-203.el5.noarch
Jun 22 11:22:34 Updated: firstboot-tui - 1.4.27.7-1.el5.centos.x86_64
Jun 22 11:22:35 Updated: NetworkManager - 1:0.7.0-4.el5_3.x86_64
Jun 22 11:22:35 Installed: NetworkManager-glib - 1:0.7.0-4.el5_3.x86_64
Jun 22 11:22:35 Updated: yum - 3.2.19-18.el5.centos.noarch
Jun 22 11:22:36 Updated: yum-fastestmirror - 1.1.16-13.el5.centos.noarch

[dognose]

Yes, it was openMP causing the problem.. how can that be fixed?

[magick]

We've had really good success with OpenMP except on the one CentOS 5.3 server where it slows the process down considerably. We have several other CentOS 5.3 servers where ImageMagick built with OpenMP support works just fine (i.e. processes speed-up rather than slow-down). Right now we do not know under what conditions the slow-down occurs. The one CentOS 5.3 slow-down is on a particular virtual server. Are you on a virtual server as well?

[dognose]

I've now seen the problem on 3 different servers. Two were core i7's, which generally run very fast, with 4 cores and 8 'threads' and the other was as q9400

I've also tried downgrading to centos5.2 and just updating the IM dependencies, and I still have the problem. My only theory is that it's something in the update list above.
Did one of them go from openmp safe to not?

[dognose]

Hmm, this thread indicated problems with gcc-4.1.2 and openmp

http://www.nabble.com/GCC-and-OpenMP-td17220268.html

it's not easy to update the compiler though.

[balamm]

So is there a solution for this yet?

Sure, I can recompile IM and disable openMP, which I just did on my own VPS and it solved the deadlock problems, but what about end users of the applications I develop, who are on shared hosting?

Telling them to tell their hosts to recompile IM isn't going to fly.

And you know how often most shared hosts update their imagemagick....
Even if you disable openMP by defaut, It could be 2 years before any new version fixes filter down to these shared hosts.

Is there no command line switch that might be used to address this for any IM version at the application script level?

[SEO]

Hi!

I may be experiencing the same problem, the load of my server sometimes jumps to over 200 while normally it is at 0,5/1,0 (low traffic).

I watched TOP and noticed mass convert and montage commands.

How could I test if OpenMP is increasing the server load per command?

Best Regards,
Jan Jaap