Page 1 of 1
gray area fills missing pixels
Posted: 2014-01-29T12:03:16-07:00
by pyramation
I've been running into rare occurrences where an image will have missing pixels and the area is simply gray:
https://b0c8025ff93679be66fe-34fbaafa6f ... 17fd2a.jpg
The commands that I've been running that have caused such results are (in sprintf notation):
Code: Select all
convert %(inFile)s -resize %(width)sx%(height)s\! -strip -interlace Plane -quality %(quality)s %(outFile)s
It is really hard to reproduce, and there is no error message. If there is any insight at all in to what might cause this, any help is greatly appreciated.
Re: gray area fills missing pixels
Posted: 2014-01-29T12:25:59-07:00
by glennrp
"identify -verbose" and "identify -debug coder" don't tell me anything useful. Firefox and Chrome both display the image the same way as ImageMagick does. Evidently the image is corrupt, and I can't explain why none of the applications reject the image or even warn about it.
Re: gray area fills missing pixels
Posted: 2014-01-29T12:35:23-07:00
by fmw42
Possibly your tmp directory is too full.
Post a link to an example input image and identify your version of IM, platform and libjpeg
Re: gray area fills missing pixels
Posted: 2014-01-29T12:39:08-07:00
by magick
It looks like the image was converted from a corrupt image. JPEG returns gray scanlines when the image is corrupt. The gray pixels were saved to a new JPEG which visually displays gray pixels but does not report a corrupt image because its not corrupt!
Re: gray area fills missing pixels
Posted: 2014-01-29T13:15:59-07:00
by pyramation
Thanks for the replies! I've posted another corrupt image plus the input, as well as the system information requested (except libjpeg version, how can I find this?).
Here is another example with input and result:
INPUT:
http://8b7ba2e22fc2d1253129-be2542a8acd ... 73b01e.jpg
OUTPUT:
http://8b7ba2e22fc2d1253129-be2542a8acd ... 20x480.jpg
NOTES:
I've listed information about the machine, it's happened on a few similar machines as well.
IMAGEMAGIC INFO:
Code: Select all
Version: ImageMagick 6.7.7-10 2013-09-10 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2012 ImageMagick Studio LLC
Features: OpenMP
LIBJPEG version:
How can I check this? (thanks)
SYSTEM INFO:
Disk Usage in the folder where uploads were going was high, although not enough to fill the disk.
all block devices
Code: Select all
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 40G 0 disk
└─xvda1 202:1 0 40G 0 part /
xvde 202:64 0 80G 0 disk
└─xvde1 202:65 0 80G 0 part
system info
Code: Select all
$ lscpu
Architecture: x86_64
CPU op-mode(s): 64-bit
CPU(s): 4
Thread(s) per core: 1
Core(s) per socket: 1
CPU socket(s): 4
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 16
Model: 4
Stepping: 2
CPU MHz: 2200.088
Hypervisor vendor: Xen
Virtualization type: para
L1d cache: 64K
L1i cache: 64K
L2 cache: 512K
L3 cache: 6144K
free space
Code: Select all
$ free
total used free shared buffers cached
Mem: 8162612 4750876 3411736 0 426272 3180512
-/+ buffers/cache: 1144092 7018520
Swap: 16771852 0 16771852
Code: Select all
$ uname -mrs
Linux 2.6.38-8-virtual x86_64
Linux distribution and version
Code: Select all
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 13.10
Release: 13.10
Codename: saucy
Re: gray area fills missing pixels
Posted: 2014-01-29T13:34:28-07:00
by glennrp
Also check the space available on your /tmp ($TMPDIR or whatever temporary) disk.
To find out the libjpeg version, you can locate the "convert" executable ("which convert")
then run "ldd" on that.
Re: gray area fills missing pixels
Posted: 2014-01-29T14:53:20-07:00
by pyramation
Again, thanks for the help everyone! This has been very helpful.
CORRUPT IMAGE
Regarding the possibility of the corrupt image. This could be the issue, and if so, I'd like to let everyone know so it's not a mysterious issue for imagemagick and I'll do my best to verify this case.
It's possible that the transmission between the CDN and the server is where the data is lost, and hence why its generating JPEGs with grey areas. There are a few steps that give a possible chance for data loss, and I have not put any checksums there, and should. Also, the network is probably much more unreliable than imagemagick. Hence, I am leaning towards the issue being a network issue and hope I haven't caused any false alarms. I will change my application code and post my research after I conclude it's a network issue here (or whatever my results are).
TMP DIR
Every upload is removed from our tmp dir (/tmp) and I verified that's where they are stored.
LIBJPEG
Thanks again for the help here. Not sure if the libjpeg showed up in this list:
Code: Select all
$ ldd /usr/bin/convert
linux-vdso.so.1 => (0x00007fff7f731000)
libMagickCore.so.5 => /usr/lib/x86_64-linux-gnu/libMagickCore.so.5 (0x00007f719840f000)
libMagickWand.so.5 => /usr/lib/x86_64-linux-gnu/libMagickWand.so.5 (0x00007f71980fb000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7197edd000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7197b15000)
liblcms2.so.2 => /usr/lib/x86_64-linux-gnu/liblcms2.so.2 (0x00007f71978c0000)
libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007f719761d000)
liblqr-1.so.0 => /usr/lib/x86_64-linux-gnu/liblqr-1.so.0 (0x00007f7197405000)
libfftw3.so.3 => /usr/lib/x86_64-linux-gnu/libfftw3.so.3 (0x00007f719700c000)
libfontconfig.so.1 => /usr/lib/x86_64-linux-gnu/libfontconfig.so.1 (0x00007f7196dcf000)
libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f7196bbd000)
libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f7196888000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f7196677000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f719645e000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f719615a000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f7195f4a000)
libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007f7195d40000)
/lib64/ld-linux-x86-64.so.2 (0x00007f71988b7000)
libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007f7195a3e000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f7195814000)
libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f71955f6000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f71953f1000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f71951b2000)
libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f7194fad000)
libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f7194da7000)
Re: gray area fills missing pixels
Posted: 2014-01-29T15:31:31-07:00
by fmw42
try
convert -list format
JPEG* JPEG rw- Joint Photographic Experts Group JFIF format (90)
JPG* JPEG rw- Joint Photographic Experts Group JFIF format (90)
I think that the 90 means version 9.0
Do you see jpg listed with
convert -version
or
convert -list configure (at the line starting with Delegates)
Re: gray area fills missing pixels
Posted: 2014-01-29T16:48:51-07:00
by pyramation
After running "convert -list format", I got:
Code: Select all
JPEG* JPEG rw- Joint Photographic Experts Group JFIF format (80)
JPG* JPEG rw- Joint Photographic Experts Group JFIF format (80)
Which I'm guessing means 8.0
For your other question:
Code: Select all
$ convert -version
Version: ImageMagick 6.7.7-10 2013-09-10 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2012 ImageMagick Studio LLC
Features: OpenMP
Re: gray area fills missing pixels
Posted: 2014-01-29T17:26:59-07:00
by fmw42
convert -version
Version: ImageMagick 6.7.7-10 2013-09-10 Q16
http://www.imagemagick.org
Copyright: Copyright (C) 1999-2012 ImageMagick Studio LLC
Features: OpenMP
Your system is too old for convert -version to show the list of delegates.
What do you get from
convert -list configure (on the line starting with Delegates)
Re: gray area fills missing pixels
Posted: 2014-01-29T18:21:59-07:00
by pyramation
I get this
Code: Select all
DELEGATES bzlib djvu fftw fontconfig freetype jbig jpeg jng jp2 lcms2 lqr lzma openexr pango png rsvg tiff x11 xml wmf zlib
The entire output is below if this helps:
Code: Select all
convert -list configure
Path: /usr/lib/x86_64-linux-gnu/ImageMagick-6.7.7/config/configure.xml
Name Value
-------------------------------------------------------------------------------
CC gcc -std=gnu99 -std=gnu99
CFLAGS -I/usr/include/lqr-1 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -fopenmp -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall -pthread
CODER_PATH /usr/lib/x86_64-linux-gnu/ImageMagick-6.7.7/modules-Q16/coders
CONFIGURE ./configure '--prefix=/usr' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--sysconfdir=/etc' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' 'CFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' 'FFLAGS=-g -O2' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' '--with-modules' '--with-gs-font-dir=/usr/share/fonts/type1/gsfonts' '--with-magick-plus-plus' '--with-djvu' '--with-wmf' '--without-gvc' '--enable-shared' '--without-dps' '--without-fpx' '--with-perl-options=INSTALLDIRS=vendor' '--x-includes=/usr/include/X11' '--x-libraries=/usr/lib/X11'
CONFIGURE_PATH /etc/ImageMagick/
COPYRIGHT Copyright (C) 1999-2012 ImageMagick Studio LLC
CPPFLAGS -I/usr/include/ImageMagick
CXX g++
CXXFLAGS -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -pthread
DEFS -DHAVE_CONFIG_H
DELEGATES bzlib djvu fftw fontconfig freetype jbig jpeg jng jp2 lcms2 lqr lzma openexr pango png rsvg tiff x11 xml wmf zlib
DISTCHECK_CONFIG_FLAGS 'CFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' --disable-deprecated --with-quantum-depth=16 --with-umem=no --with-autotrace=no --with-dps=no --with-fpx=no --with-gslib=no --with-fontpath= --with-gs-font-dir=/usr/share/fonts/type1/gsfonts --with-gvc=no --with-perl=no
DOCUMENTATION_PATH /usr/share/doc/ImageMagick-6.7.7
EXEC-PREFIX /usr
EXECUTABLE_PATH /usr/bin
FEATURES OpenMP
FILTER_PATH /usr/lib/x86_64-linux-gnu/ImageMagick-6.7.7/modules-Q16/filters
HOST x86_64-unknown-linux-gnu
INCLUDE_PATH /usr/include/ImageMagick
LDFLAGS -L/usr/lib/x86_64-linux-gnu -Wl,-Bsymbolic-functions -Wl,-z,relro -L/usr/lib/X11 -L/usr/lib
LIB_VERSION 0x677
LIB_VERSION_NUMBER 6,7,7,10
LIBRARY_PATH /usr/lib/x86_64-linux-gnu/ImageMagick-6.7.7
LIBS -lMagickCore -llcms2 -ltiff -lfreetype -ljpeg -llqr-1 -lglib-2.0 -lfftw3 -lfontconfig -lXext -lSM -lICE -lX11 -lXt -llzma -lbz2 -lz -lm -lgomp -lpthread -lltdl
NAME ImageMagick
PCFLAGS -fopenmp
PREFIX /usr
QuantumDepth 16
RELEASE_DATE 2013-09-10
SHARE_PATH /usr/share/ImageMagick-6.7.7
SVN_REVISION 8535
TARGET_CPU x86_64
TARGET_OS linux-gnu
TARGET_VENDOR unknown
VERSION 6.7.7
WEBSITE http://www.imagemagick.org
Path: [built-in]
Name Value
-------------------------------------------------------------------------------
NAME ImageMagick
Re: gray area fills missing pixels
Posted: 2014-01-29T18:24:57-07:00
by pyramation
Also, I used
when I was installing, perhaps I should be building from source to get the latest, 6.7.7 vs. the latest 6.8.8
Re: gray area fills missing pixels
Posted: 2014-01-29T19:38:39-07:00
by fmw42
I suspect that your problem is what magick said -- a corrupt input jpg. I don't think it is your configuration or version of IM or libjpeg. You have all the delegates, so that is not likely the issue.
