Convert big PSD (1.7GB) to JPG hangs

[Swoops]

Hello,

Whenever I execute one of the bad commands it looks like it gets stuck in some form of infinite loop. When running 'top', this is the end result:

PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
159 135 root D 20044m 59% 2 0% convert ./huge-document.psd -flatten ./huge-document.jpg
143 135 root T 20044m 59% 2 0% convert ./huge-document.psd ./huge-document.jpg

Without the -flatten option and using the preview layer (the good command) all is fine.
Any ideas would be helpful .

Version: 7.0.4-4
Environment: Alpine 3.5 in Docker 1.12
Machine: AWS EC2 c4.4xlarge
Memory: 30GB
Storage: AWS EFS
Cores: 16
Settings:
----------> MAGICK_AREA_LIMIT: 8GiB
----------> MAGICK_HEIGHT_LIMIT: 300MP
----------> MAGICK_MAP_LIMIT: 16GiB
----------> MAGICK_MEMORY_LIMIT: 4GiB
----------> MAGICK_TEMPORARY_PATH: /tmp
----------> MAGICK_THREADS_LIMIT: 6
----------> MAGICK_TMPDIR: /tmp
----------> MAGICK_WIDTH_LIMIT: 300MP
----------> OMP_NUM_THREADS: 6

Download link: http://s3-eu-west-1.amazonaws.com/image ... cument.psd

Bad command 1: convert ./huge-document.psd ./huge-document.jpg
Bad command 2: convert ./huge-document.psd -flatten ./huge-document.jpg
Good command: convert ./huge-document.psd[0] ./huge-document.jpg

Configuration:
Path: /usr/local/lib/ImageMagick-7.0.4//config-Q16HDRI/configure.xml

Name Value
-------------------------------------------------------------------------------
CC gcc
CFLAGS -I/usr/include/libxml2 -I/usr/include/cairo -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/librsvg-2.0 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/gdk-pixbuf-2.0 -pthread -I/usr/include/libpng16 -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/pango-1.0 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/harfbuzz -I/usr/include/OpenEXR -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/freetype2 -I/usr/include/libpng16 -fopenmp -Wno-deprecated-declarations -Wno-error=unused-variable -Wall -mtune=core2 -fexceptions -DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16
CODER_PATH /usr/local/lib/ImageMagick-7.0.4/modules-Q16HDRI/coders
CONFIGURE ./configure '--with-gslib=yes' '--with-x=yes' '--with-rsvg=yes' '--without-magick-plus-plus' 'CC=gcc' 'CFLAGS=-Wno-deprecated-declarations -Wno-error=unused-variable'
CONFIGURE_PATH /usr/local/etc/ImageMagick-7/
COPYRIGHT Copyright (C) 1999-2016 ImageMagick Studio LLC
CPPFLAGS -I/usr/local/include/ImageMagick-7
CXX g++
CXXFLAGS -g -O2
DEFS -DHAVE_CONFIG_H
DELEGATES bzlib mpeg fftw fontconfig freetype gslib jng jpeg openexr pango png ps rsvg tiff webp wmf x xml zlib
DISTCHECK_CONFIG_FLAGS 'CC=gcc' 'CFLAGS=-Wno-deprecated-declarations -Wno-error=unused-variable' --disable-deprecated --with-quantum-depth=16 --with-jemalloc=no --with-umem=no --with-autotrace=no --with-fontpath= --with-perl=no
DOCUMENTATION_PATH /usr/local/share/doc/ImageMagick-7
EXEC-PREFIX /usr/local
EXECUTABLE_PATH /usr/local/bin
FEATURES DPC HDRI Cipher OpenMP
FILTER_PATH /usr/local/lib/ImageMagick-7.0.4/modules-Q16HDRI/filters
GIT_REVISION 11509
HOST x86_64-unknown-linux-gnu
INCLUDE_PATH /usr/local/include/ImageMagick-7
LDFLAGS -L/usr/local/lib
LIB_VERSION 0x704
LIB_VERSION_NUMBER 7,0,4,4
LIBRARY_PATH /usr/local/lib/ImageMagick-7.0.4
LIBS -ltiff -lfreetype -ljpeg -lgs -lpng16 -lfftw3 -lfontconfig -lfreetype -lwebp -lwmflite -lXext -lX11 -lbz2 -lIlmImf -lImath -lHalf -lIex -lIexMath -lIlmThread -lpangocairo-1.0 -lpango-1.0 -lgobject-2.0 -lglib-2.0 -lintl -lcairo -lrsvg-2 -lm -lgio-2.0 -lgdk_pixbuf-2.0 -lgobject-2.0 -lglib-2.0 -lintl -lcairo -lxml2 -L/lib -lz -lm -lgomp
NAME ImageMagick
PCFLAGS -fopenmp -DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16
PREFIX /usr/local
QuantumDepth 16
RELEASE_DATE 2017-01-16
SHARE_PATH /usr/local/share/ImageMagick-7
SHAREARCH_PATH /usr/local/lib/ImageMagick-7.0.4/config-Q16HDRI
TARGET_CPU x86_64
TARGET_OS linux-gnu
TARGET_VENDOR unknown
VERSION 7.0.4
WEBSITE http://www.imagemagick.org

Path: [built-in]

Name Value
-------------------------------------------------------------------------------
FEATURES OpenMP
NAME ImageMagick
QuantumDepth 16

[snibgo]

The PSD is too large for me to download. How many pixels does it have? How much real memory do you have? Perhaps the commands that read all layers need more memory than you have, so IM will store the pixels on disk, which takes forever, giving the appearance of an infinite loop.

[Swoops]

The resolution is 10115 × 20000. The moment everything is started and we are about to run the convert it has 28GB of memory available. I am pretty sure it is not just taking a long time, as we let it run for days actually. Also its not writing anything to disk the moment it hangs and uses no CPU at all.

[snibgo]

You have 28 GB, but various limits in place. You might try relaxing those limits.

Q16 v6 takes 8 bytes/pixel. (I'm not sure about v7.) So that is 1.6 GB per layer. How many layers?

[Swoops]

There are 7 layers. The reason we put these limits in place as we don't want the server to crash, as the server is used as a rendition generator and most likely have to deal with multiple conversions at the same time. Will try to do the same task with 1952 GB of memory and 128 cores (AWS EC2 x1.32xlarge), without any limits in magick config .

[Swoops]

Ok, it seems like you are right. Now it did finish the conversion in 10 minutes. Sorry for reporting it as a bug. Will try to tweak to the optimal setting.

[magick]

Large images force the pixel cache on disk rather than memory. Since I/O is a thousand times slower than memory, conversion of the image is expected to be slow but still complete. You can extract the composite image with this command (bypassing the layers):

Code: Select all

convert -limit area 1GB -debug cache './huge-document.psd[0]' huge-document.jpg

We converted the image on a SSD and the process completed in one minute. On a SATA disk, it took 5 minutes to complete.

[Swoops]

Hi magick, thanks for giving it a try. For that command it also works perfect for us. However does it also finish so well for you when you run either of these?:

Bad command 1: convert ./huge-document.psd ./huge-document.jpg
Bad command 2: convert ./huge-document.psd -flatten ./huge-document.jpg

With these it never completes. Not even after days with the system as described in first post.

[magick]

On an SSD, the flatten command ran in 5 and 1/2 minutes:

Code: Select all

convert -limit area 1GB ./huge-document.psd -flatten huge-document.jpg

We're using 7.0.4-4 with HDRI-enabled on a Linux desktop.

You can track progress of your command by adding '-debug cache' to your command line.

[Swoops]

Ok, I got the bad command completed in 50 seconds, by disabling all disk cache and pure heap memory cache. The heap memory cache is 4GiB, as before. Writing to disk, especially AWS EFS, was the root cause of this. Thanks Magick for pointing me to the debug option, that really helped.