gray area fills missing pixels

[pyramation]

I've been running into rare occurrences where an image will have missing pixels and the area is simply gray: https://b0c8025ff93679be66fe-34fbaafa6f ... 17fd2a.jpg

The commands that I've been running that have caused such results are (in sprintf notation):

Code: Select all

convert %(inFile)s -resize %(width)sx%(height)s\! -strip -interlace Plane -quality %(quality)s %(outFile)s

It is really hard to reproduce, and there is no error message. If there is any insight at all in to what might cause this, any help is greatly appreciated.

[glennrp]

"identify -verbose" and "identify -debug coder" don't tell me anything useful. Firefox and Chrome both display the image the same way as ImageMagick does. Evidently the image is corrupt, and I can't explain why none of the applications reject the image or even warn about it.

[fmw42]

Possibly your tmp directory is too full.

Post a link to an example input image and identify your version of IM, platform and libjpeg

[magick]

It looks like the image was converted from a corrupt image. JPEG returns gray scanlines when the image is corrupt. The gray pixels were saved to a new JPEG which visually displays gray pixels but does not report a corrupt image because its not corrupt!

[pyramation]

Thanks for the replies! I've posted another corrupt image plus the input, as well as the system information requested (except libjpeg version, how can I find this?).

Here is another example with input and result:

INPUT:
http://8b7ba2e22fc2d1253129-be2542a8acd ... 73b01e.jpg

OUTPUT:
http://8b7ba2e22fc2d1253129-be2542a8acd ... 20x480.jpg

NOTES:

I've listed information about the machine, it's happened on a few similar machines as well.

IMAGEMAGIC INFO:

Code: Select all

Version: ImageMagick 6.7.7-10 2013-09-10 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2012 ImageMagick Studio LLC
Features: OpenMP

LIBJPEG version:

How can I check this? (thanks)

SYSTEM INFO:

Disk Usage in the folder where uploads were going was high, although not enough to fill the disk.

Code: Select all

$ du -sh .
5.7G	.

all block devices

Code: Select all

$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  40G  0 disk
└─xvda1 202:1    0  40G  0 part /
xvde    202:64   0  80G  0 disk
└─xvde1 202:65   0  80G  0 part

system info

Code: Select all

$ lscpu
Architecture:          x86_64
CPU op-mode(s):        64-bit
CPU(s):                4
Thread(s) per core:    1
Core(s) per socket:    1
CPU socket(s):         4
NUMA node(s):          1
Vendor ID:             AuthenticAMD
CPU family:            16
Model:                 4
Stepping:              2
CPU MHz:               2200.088
Hypervisor vendor:     Xen
Virtualization type:   para
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K
L3 cache:              6144K

free space

Code: Select all

$ free
             total       used       free     shared    buffers     cached
Mem:       8162612    4750876    3411736          0     426272    3180512
-/+ buffers/cache:    1144092    7018520
Swap:     16771852          0   16771852

Code: Select all

$ uname -mrs
Linux 2.6.38-8-virtual x86_64

Linux distribution and version

Code: Select all

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 13.10
Release:	13.10
Codename:	saucy

[glennrp]

Also check the space available on your /tmp ($TMPDIR or whatever temporary) disk.
To find out the libjpeg version, you can locate the "convert" executable ("which convert")
then run "ldd" on that.

[pyramation]

Again, thanks for the help everyone! This has been very helpful.

CORRUPT IMAGE

Regarding the possibility of the corrupt image. This could be the issue, and if so, I'd like to let everyone know so it's not a mysterious issue for imagemagick and I'll do my best to verify this case.

It's possible that the transmission between the CDN and the server is where the data is lost, and hence why its generating JPEGs with grey areas. There are a few steps that give a possible chance for data loss, and I have not put any checksums there, and should. Also, the network is probably much more unreliable than imagemagick. Hence, I am leaning towards the issue being a network issue and hope I haven't caused any false alarms. I will change my application code and post my research after I conclude it's a network issue here (or whatever my results are).

TMP DIR

Every upload is removed from our tmp dir (/tmp) and I verified that's where they are stored.

LIBJPEG

Thanks again for the help here. Not sure if the libjpeg showed up in this list:

Code: Select all

$ ldd /usr/bin/convert
	linux-vdso.so.1 =>  (0x00007fff7f731000)
	libMagickCore.so.5 => /usr/lib/x86_64-linux-gnu/libMagickCore.so.5 (0x00007f719840f000)
	libMagickWand.so.5 => /usr/lib/x86_64-linux-gnu/libMagickWand.so.5 (0x00007f71980fb000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7197edd000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7197b15000)
	liblcms2.so.2 => /usr/lib/x86_64-linux-gnu/liblcms2.so.2 (0x00007f71978c0000)
	libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007f719761d000)
	liblqr-1.so.0 => /usr/lib/x86_64-linux-gnu/liblqr-1.so.0 (0x00007f7197405000)
	libfftw3.so.3 => /usr/lib/x86_64-linux-gnu/libfftw3.so.3 (0x00007f719700c000)
	libfontconfig.so.1 => /usr/lib/x86_64-linux-gnu/libfontconfig.so.1 (0x00007f7196dcf000)
	libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f7196bbd000)
	libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f7196888000)
	libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f7196677000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f719645e000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f719615a000)
	libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f7195f4a000)
	libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007f7195d40000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f71988b7000)
	libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007f7195a3e000)
	libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f7195814000)
	libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f71955f6000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f71953f1000)
	libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f71951b2000)
	libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f7194fad000)
	libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f7194da7000)

[fmw42]

try

convert -list format


JPEG* JPEG rw- Joint Photographic Experts Group JFIF format (90)
JPG* JPEG rw- Joint Photographic Experts Group JFIF format (90)

I think that the 90 means version 9.0

Do you see jpg listed with

convert -version

or

convert -list configure (at the line starting with Delegates)

[pyramation]

After running "convert -list format", I got:

Code: Select all

     JPEG* JPEG      rw-   Joint Photographic Experts Group JFIF format (80)
      JPG* JPEG      rw-   Joint Photographic Experts Group JFIF format (80)

Which I'm guessing means 8.0

For your other question:

Code: Select all

$ convert -version
Version: ImageMagick 6.7.7-10 2013-09-10 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2012 ImageMagick Studio LLC
Features: OpenMP

[fmw42]

convert -version
Version: ImageMagick 6.7.7-10 2013-09-10 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2012 ImageMagick Studio LLC
Features: OpenMP

Your system is too old for convert -version to show the list of delegates.


What do you get from

convert -list configure (on the line starting with Delegates)

[pyramation]

I get this

Code: Select all

DELEGATES     bzlib djvu fftw fontconfig freetype jbig jpeg jng jp2 lcms2 lqr lzma openexr pango png rsvg tiff x11 xml wmf zlib

The entire output is below if this helps:

Code: Select all

 convert -list configure

Path: /usr/lib/x86_64-linux-gnu/ImageMagick-6.7.7/config/configure.xml

Name          Value
-------------------------------------------------------------------------------
CC            gcc -std=gnu99 -std=gnu99
CFLAGS        -I/usr/include/lqr-1 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -fopenmp -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall -pthread
CODER_PATH    /usr/lib/x86_64-linux-gnu/ImageMagick-6.7.7/modules-Q16/coders
CONFIGURE     ./configure  '--prefix=/usr' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--sysconfdir=/etc' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' 'CFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' 'FFLAGS=-g -O2' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' '--with-modules' '--with-gs-font-dir=/usr/share/fonts/type1/gsfonts' '--with-magick-plus-plus' '--with-djvu' '--with-wmf' '--without-gvc' '--enable-shared' '--without-dps' '--without-fpx' '--with-perl-options=INSTALLDIRS=vendor' '--x-includes=/usr/include/X11' '--x-libraries=/usr/lib/X11'
CONFIGURE_PATH /etc/ImageMagick/
COPYRIGHT     Copyright (C) 1999-2012 ImageMagick Studio LLC
CPPFLAGS      -I/usr/include/ImageMagick
CXX           g++
CXXFLAGS      -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -pthread
DEFS          -DHAVE_CONFIG_H
DELEGATES     bzlib djvu fftw fontconfig freetype jbig jpeg jng jp2 lcms2 lqr lzma openexr pango png rsvg tiff x11 xml wmf zlib
DISTCHECK_CONFIG_FLAGS 'CFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' --disable-deprecated --with-quantum-depth=16 --with-umem=no --with-autotrace=no --with-dps=no --with-fpx=no --with-gslib=no --with-fontpath= --with-gs-font-dir=/usr/share/fonts/type1/gsfonts --with-gvc=no --with-perl=no
DOCUMENTATION_PATH /usr/share/doc/ImageMagick-6.7.7
EXEC-PREFIX   /usr
EXECUTABLE_PATH /usr/bin
FEATURES      OpenMP
FILTER_PATH   /usr/lib/x86_64-linux-gnu/ImageMagick-6.7.7/modules-Q16/filters
HOST          x86_64-unknown-linux-gnu
INCLUDE_PATH  /usr/include/ImageMagick
LDFLAGS       -L/usr/lib/x86_64-linux-gnu -Wl,-Bsymbolic-functions -Wl,-z,relro -L/usr/lib/X11 -L/usr/lib
LIB_VERSION   0x677
LIB_VERSION_NUMBER 6,7,7,10
LIBRARY_PATH  /usr/lib/x86_64-linux-gnu/ImageMagick-6.7.7
LIBS          -lMagickCore -llcms2 -ltiff -lfreetype -ljpeg -llqr-1 -lglib-2.0 -lfftw3 -lfontconfig -lXext -lSM -lICE -lX11 -lXt -llzma -lbz2 -lz -lm -lgomp -lpthread -lltdl
NAME          ImageMagick
PCFLAGS       -fopenmp
PREFIX        /usr
QuantumDepth  16
RELEASE_DATE  2013-09-10
SHARE_PATH    /usr/share/ImageMagick-6.7.7
SVN_REVISION  8535
TARGET_CPU    x86_64
TARGET_OS     linux-gnu
TARGET_VENDOR unknown
VERSION       6.7.7
WEBSITE       http://www.imagemagick.org

Path: [built-in]

Name          Value
-------------------------------------------------------------------------------
NAME          ImageMagick

[pyramation]

Also, I used

Code: Select all

apt-get install imagemagick

when I was installing, perhaps I should be building from source to get the latest, 6.7.7 vs. the latest 6.8.8

[fmw42]

I suspect that your problem is what magick said -- a corrupt input jpg. I don't think it is your configuration or version of IM or libjpeg. You have all the delegates, so that is not likely the issue.