Page 1 of 2

IM-produced "standard" JPEGs don't work with LaTeX/pdflatex

Posted: 2015-05-07T12:37:48-07:00
by pipitas
I noticed yesterday, that I cannot use JPEGs which are produced by my version of ImageMagick (6.9.0-0 Q16 x86_64 2014-12-06 on OS X from MacPorts).

It appears to me that the reason for this is that ImageMagick changed how it writes some metadata about resolution into the JPEG files. I did this to verify:

Code: Select all

$ convert wizard: -density 300 wiz-300densi.jpg
$ convert wizard:              wiz-standard.jpg

$ identify wiz-300densi.jpg wiz-standard.jpg
  wiz-300densi.jpg    JPEG 480x640 480x640+0+0 8-bit sRGB 62.2KB 0.000u 0:00.000
  wiz-standard.jpg[1] JPEG 480x640 480x640+0+0 8-bit sRGB 62.2KB 0.000u 0:00.000
I can successfully insert wiz-300densi.jpg into a LaTeX document and it gets converted to PDF successfully by the pdflatex command.

However, LaTeX files using wiz-standard.jpg produce this error message:

Code: Select all

pdfTeX warning: pdflatex: arithmetic: number too big

pdfTeX warning: pdflatex: arithmetic: number too big
<./wiz-standard.jpg, id=4, --32768.0pt x 0.0pt>

! Package graphics Error: Division by 0.
See the graphics package documentation for explanation.

Type  H <return>  for immediate help.
 ...                                              
l.63 \includegraphics{./wiz-standard.jpg}
?
Somehow LaTeX/pdflatex seem to evaluate the metadata from the JPEG file, base their processing of the raster data on this and arrive at a point where it attempts a "Division by 0.".

Files which use wiz-300densi.jpg work flawlessly.

It even works flawlessly if I create the JPEGs with `-density 10` or `-density 1`.

Code: Select all

$ identify -verbose wiz-300densi.jpg > wiz-300densi.log
$ identify -verbose wiz-standard.jpg > wiz-standard.log

Code: Select all

$ sdiff -sbB wiz-densi300.log wiz-standard.log
    Image: wiz-300densi.jpg                        |    Image: wiz-standard.jpg
      Resolution: 300x300                          <
      Print size: 1.6x2.13333                      <
        date:create: 2015-05-07T12:09:45+02:00     |        date:create: 2015-05-07T12:09:20+02:00
        date:modify: 2015-05-07T12:09:45+02:00     |        date:modify: 2015-05-07T12:09:20+02:00
        filename: wiz-300densi.jpg                 |        filename: wiz-standard.jpg

$ exiftool wiz-standard.jpg | grep Resolution
    Resolution Unit                 : inches
    X Resolution                    : 1
    Y Resolution                    : 1

$ exiftool wiz-300densi.jpg | grep Resolution
    Resolution Unit                 : inches
    X Resolution                    : 300
    Y Resolution                    : 300

$ ls -l wiz-300densi.jpg wiz-standard.jpg 
 -rw-r--r--  1 kp  staff  62177  7 Mai 12:09 wiz-300densi.jpg
 -rw-r--r--  1 kp  staff  62177  7 Mai 12:09 wiz-standard.jpg
So the only difference between the two JPEG files seems to be in the metadata/EXIFdata. A `compare wiz-300densi.jpg wiz-standard.jpg -compose src delta.png` does not reveal any differences in the pixel colors.

Therefore, currently I think that this is a flaw or bug in LaTeX/pdflatex. (After all, I think it is stupid for pdflatex/lualatex/xelatex to rely on this metadata info in the JPEG and to kill itself if that is found empty. Instead it should gracefully work with the pixels which are there, or assume some other resolution value: Because the actual value of that metadata doesn’t matter anyway: I can use `-density 300` or `-density 10` when creating the JPEG — on the page image there is no difference.)

Why do I bring it up in this forum then?

Well, one thing after the other... :-) I'll be sure to report this to some appropriate LaTeX forum too and see what they think of it.

I only would like to know from you guys:
  1. Has this IM behavior when producing JPEGs been changed in recent versions of ImageMagick? (I'm asking because I seem to remember that the same procedure -- JPEGs generated by IM from `wizard:` and `logo:` and using them in LaTeX's PDF output -- did work a year or so ago...)
  2. If so, was this change on purpose? Or did it happen by accident, and could possibly be reverted?
  3. Is there any (official or inofficial) "standard" for JPEGs which are expecting a valid entry about the desired rendering resolution inside their metadata? (In this case it could be argued that LaTeX is "right" to assume proper values to work with -- otherwise it's the contrary.)

Re: IM-produced "standard" JPEGs don't work with LaTeX/pdflatex

Posted: 2015-05-07T14:39:38-07:00
by fmw42
Did you change your version of libjpeg? Is it the most current. Did you check the changelog for any changes to jpeg?

http://www.imagemagick.org/script/changelog.php (right now it is only showing IM7 changes and I have reported this to switch it it IM6)

Re: IM-produced "standard" JPEGs don't work with LaTeX/pdflatex

Posted: 2015-05-07T16:24:07-07:00
by pipitas
fmw42 wrote:Did you change your version of libjpeg? Is it the most current.
I'm not sure I understand your question whether I changed my version of libjpeg...

This system is OS X Mavericks with MacPorts extra packages. Here are all the MacPort packages with 'JPEG' in their names:

Code: Select all

$ port installed *jpeg*

 The following ports are currently installed:
  jpeg @9a_1+universal (active)
  jpeg2ps @1.9_0 (active)
  jpegoptim @1.2.4_1 (active)
  openjpeg @2.1.0_0+universal (active)
  openjpeg15 @1.5.1_0+universal (active)
The `otool -L` on OSX command works similar to `ldd` on Linux and shows the dynamically loaded libraries. Here is what it produces:

Code: Select all

$ otool -L /opt/local/bin/convert

 /opt/local/bin/convert:
	/opt/local/lib/libMagickCore-6.Q16.2.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/local/lib/libMagickWand-6.Q16.2.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/local/lib/liblcms2.2.dylib (compatibility version 3.0.0, current version 3.6.0)
	/opt/local/lib/liblqr-1.0.dylib (compatibility version 4.0.0, current version 4.2.0)
	/opt/local/lib/libglib-2.0.0.dylib (compatibility version 4201.0.0, current version 4201.1.0)
	/opt/local/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.2.0)
	/opt/local/lib/libfftw3.3.dylib (compatibility version 8.0.0, current version 8.4.0)
	/opt/local/lib/libfontconfig.1.dylib (compatibility version 10.0.0, current version 10.0.0)
	/opt/local/lib/libfreetype.6.dylib (compatibility version 18.0.0, current version 18.2.0)
	/opt/local/lib/libXext.6.dylib (compatibility version 11.0.0, current version 11.0.0)
	/opt/local/lib/libXt.6.dylib (compatibility version 7.0.0, current version 7.0.0)
	/opt/local/lib/liblzma.5.dylib (compatibility version 6.0.0, current version 6.7.0)
	/opt/local/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.6)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
	/opt/local/lib/libltdl.7.dylib (compatibility version 11.0.0, current version 11.1.0)
	/opt/local/lib/libSM.6.dylib (compatibility version 7.0.0, current version 7.1.0)
	/opt/local/lib/libICE.6.dylib (compatibility version 10.0.0, current version 10.0.0)
	/opt/local/lib/libX11.6.dylib (compatibility version 10.0.0, current version 10.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
There is no "jpeg" in sight here...

Does this seem to suggest that the `convert` command isn't relying on any dynamically loaded libjpeg libraries, but uses statically compiled in ones?

Not sure if I answered your question. (I'm not a developer...)

Also, I don't know which is the 'most current' version of libjpeg...

Re: IM-produced "standard" JPEGs don't work with LaTeX/pdflatex

Posted: 2015-05-07T18:16:50-07:00
by fmw42
I'm not sure I understand your question whether I changed my version of libjpeg...
From the time it worked successfully until now when it does not.

I am not a developer either.

I use MacPorts also and have

jpeg @9a_1 (active)


Might the difference be that you have a universal version?

This MacPorts version seems to match with what is available at http://www.ijg.org/


I am not sure if IM uses this delegate library to write images as well as read them.

One of the developers will have to respond for a proper reply.

Re: IM-produced "standard" JPEGs don't work with LaTeX/pdflatex

Posted: 2015-05-07T19:03:48-07:00
by pipitas
fmw42 wrote:I use MacPorts also and have

jpeg @9a_1 (active)

Might the difference be that you have a universal version?
No idea about that at all.

If you create a wizard JPEG with your IM and the non-universal JPEG version, what is the respective output for it when you run the `identify -verbose ... | grep Resolution` and the `exiftool ... | grep -E '(Resolution|Print size:)'` commands?

And, should you have LaTeX installed, would you test it ? There is a minimally working example TeX file -- only 8 short lines of code! -- and some more description here:

Re: IM-produced "standard" JPEGs don't work with LaTeX/pdflatex

Posted: 2015-05-07T21:43:21-07:00
by fmw42
identify -verbose says

Image: wizard.jpg
Format: JPEG (Joint Photographic Experts Group JFIF format)
Mime type: image/jpeg
Class: DirectClass
Geometry: 265x352+0+0
Resolution: 118x118
Print size: 2.24576x2.98305
Units: PixelsPerCentimeter


Image: wiz-300densi.jpg
Format: JPEG (Joint Photographic Experts Group JFIF format)
Mime type: image/jpeg
Class: DirectClass
Geometry: 480x640+0+0
Resolution: 300x300
Print size: 1.6x2.13333
Units: Undefined


Image: wiz-standard.jpg
Format: JPEG (Joint Photographic Experts Group JFIF format)
Mime type: image/jpeg
Class: DirectClass
Geometry: 480x640+0+0
Units: Undefined


Note that for both output images, units get undefine from pixelspercentimeter, which is odd. Also the image size (Geometry) gets changed. That should not happen, in my opinion. So something is strange here or a bug.



exiftool -s -ee -g1 -u -n -D wizard.jpg
shows

- ImageWidth : 265
- ImageHeight : 352
2 ResolutionUnit : 2
3 XResolution : 118
5 YResolution : 118

This matches IM


exiftool -s -ee -g1 -u -n -D wiz-300densi.jpg

- ImageWidth : 480
- ImageHeight : 640
2 ResolutionUnit : 0
3 XResolution : 300
5 YResolution : 300

This matches IM



exiftool -s -ee -g1 -u -n -D wiz-standard.jpg
shows

- ImageWidth : 480
- ImageHeight : 640
2 ResolutionUnit : 1
3 XResolution : 1
5 YResolution : 1


This does not match IM. IM identify -verbose is missing the Resolution and undefined units, which should be 0 in exiftool. So this may be where the issue comes from in addition to the size/geometry change.

Perhaps wizard.jpg has erroneous values in it, especially with regard to the image size.

I believe IM convert or wizard.jpg has a bug/problem!

Sorry I do not have LaTex.

Re: IM-produced "standard" JPEGs don't work with LaTeX/pdflatex

Posted: 2015-05-08T01:58:55-07:00
by dlemstra
Fred, why are you suddenly talking about wizard.jpg? The image 'wizard:' is 480x640. But it seems that we set the units to 'pixels per inch' when the density is not specified. I will look into this. What happens when you use the following file: https://www.dropbox.com/s/97w3q16wdjeyp ... t.jpg?dl=0?

Re: IM-produced "standard" JPEGs don't work with LaTeX/pdflatex

Posted: 2015-05-08T09:47:35-07:00
by fmw42
I downloaded the image from http://www.imagemagick.org/script/forma ... tin-images and it is not 480x640, but when processed produces images that are 480x640. So something seem wrong with that file.

Re: IM-produced "standard" JPEGs don't work with LaTeX/pdflatex

Posted: 2015-05-09T19:06:09-07:00
by pipitas
fmw42 wrote:identify -verbose says

Image: wizard.jpg
Format: JPEG (Joint Photographic Experts Group JFIF format)
Mime type: image/jpeg
Class: DirectClass
Geometry: 265x352+0+0
Resolution: 118x118
Print size: 2.24576x2.98305
Units: PixelsPerCentimeter
Ummm... I'm not sure how this 'wizard.jpg' file now entered the discussion? The filenames I used where only these two (plus the built-in 'wizard:'):
fmw42 wrote:Image: wiz-300densi.jpg
Format: JPEG (Joint Photographic Experts Group JFIF format)
Mime type: image/jpeg
Class: DirectClass
Geometry: 480x640+0+0
Resolution: 300x300
Print size: 1.6x2.13333
Units: Undefined

Image: wiz-standard.jpg
Format: JPEG (Joint Photographic Experts Group JFIF format)
Mime type: image/jpeg
Class: DirectClass
Geometry: 480x640+0+0
Units: Undefined

Note that for both output images, units get undefine from pixelspercentimeter, which is odd.


As I said: a 'wizard.jpg' with defined pixelspercentimeter is not mine and cannot be used as the basis of an argument... However, there are still some weird facts in this story. See my more detailed results further down.
fmw42 wrote:Also the image size (Geometry) gets changed.
No, it doesn't! Because I never used 'wizard.jpg' in this context.
fmw42 wrote:That should not happen, in my opinion. So something is strange here or a bug.
It must be a misunderstanding, somehow. I always used (during these tests) the built-in 'wizard:' image as original.
fmw42 wrote:exiftool -s -ee -g1 -u -n -D wizard.jpg
shows

- ImageWidth : 265
- ImageHeight : 352
2 ResolutionUnit : 2
3 XResolution : 118
5 YResolution : 118

This matches IM
See above about 'wizard.jpg'...
fmw42 wrote:exiftool -s -ee -g1 -u -n -D wiz-300densi.jpg

- ImageWidth : 480
- ImageHeight : 640
2 ResolutionUnit : 0
3 XResolution : 300
5 YResolution : 300

This matches IM
Yes.
fmw42 wrote:exiftool -s -ee -g1 -u -n -D wiz-standard.jpg
shows

- ImageWidth : 480
- ImageHeight : 640
2 ResolutionUnit : 1
3 XResolution : 1
5 YResolution : 1


This does not match IM. IM identify -verbose is missing the Resolution and undefined units, which should be 0 in exiftool. So this may be where the issue comes from in addition to the size/geometry change.
It matches IM, yes. But please don't compare to a 'wizard.jpg' and conclude a size/geometry change from it -- see above about 'wizard.jpg'...
fmw42 wrote:Perhaps wizard.jpg has erroneous values in it, especially with regard to the image size.

I believe IM convert or wizard.jpg has a bug/problem!

Sorry I do not have LaTex.
I created a table about the results of a series of

Code: Select all

 'convert wizard: -density $Xx$Y wiz-$X-$Y.jpg'
with variations of values for X and Y.

But for preparation, I first looked up the JFIF documentation ( http://www.w3.org/Graphics/JPEG/jfif3.pdf ) and found on page 5 the description of these three fields in the header:
  • At byte offset `0x0d` the "Units" field. This field has a 1 byte length. It can hold 3 values. The values describe *"Units of the X and Y densities"*.
    • `0`, means: "no units, X and Y describe the pixel aspect ratios".
    • `1`, means: "X and Y are dots per inch";
    • `2`, means: "X and Y are dots per cm".
  • At byte offset `0x0e` the "XDensity" field. This field has 2 bytes length. It describes the "Horizontal pixel density".
  • At byte offset `0x10` the "YDensity" field. Its length is 2 bytes. It describes the "Vertical pixel density".

Code: Select all

    +---------------+---------+-----------------+---------------------+
    |   Byte-Offset |   0x0d  |        0x0e     |        0x10         |
    +---------------+---------+-----------------+---------------------+
    |    Field-Name | "Units" |   "XDensity"    |   "YDensity"        |
    +---------------+---------+-----------------+---------------------+
    | Value/Meaning |      0  | "X and Y describe pixel aspect ratio" |
    +---------------+---------+-----------------+---------------------+
    | Value/Meaning |      1  |     "X and Y are dots per inch"       |
    +---------------+---------+-----------------+---------------------+
    | Value/Meaning |      1  |      "X and Y are dots per cm"        |
    +---------------+---------+-----------------+---------------------+


My tests involved looking at the byte values with a hex editor too, not just relying on `identify -verbose` or `exiftool`. Additionally, I also employed `file` to see what this little tool reported.

So finally, here are the results of my 'convert wizard: -density $Xx$Y wiz-$X-$Y.jpg' commands with variations in X and Y values. Note, I also used "odd" and maybe "illegal" values like `000`, `0.2` and `1.5` instead of only "sane" integer numbers for the density (because even for basic testing, it can never be wrong to introduce some fuzzing as well):

Image

These results seem to suggest first of all a potential improvement for the `identify` tool:
  • Change the output reported for the "Units:" field.
  • Instead of reporting "Undefined" if the JFIF field in fact contains a `0`, it should report something like "0 (means: X and Y Resolution values describe pixel aspect ratio)". See also what the `file` utility reports.
  • However, even if developers are not willing to improve identify's output here and change it: at the same time there is also a bug involved. Because *sometimes* this "Undefined" is wrong: whenever `file` reports "aspect ratio" (that is, the value of JFIF header field "Units" is either `1` {dots per inch} or `2` {dots per cm}).
Some of the other results are also remarkable:
  • In every single case my hex editor double checking confirmed that Exiftool's and `file`'s reporting and of the values are correct.
  • `identify` reported nothing for the "Resolution:" of the results from those `convert` commands where I had used `-density 0.2|0.5|0.8|1.2|000|001` to create the JPEG.
  • `convert -density 0.2` created a JPEG where Exiftool and hex editor see a `0x0` resolution with the real meaning of "dots per inch"), while `convert -density 0|000` created a JPEG where the same value becomes `1x1`.
  • `convert -density 000` created a JPEG where Exiftool see resolution unit "inches", but for `convert -density 000x000` Exiftool sees resolution unit "None"


You can download the script here:
It is only ~60 lines of Bash code. In the tarball included are also the log files of the script (from which I extracted above table). So if you cannot to run it yourself (you may be on Windows), then you can still look at its results.

Re: IM-produced "standard" JPEGs don't work with LaTeX/pdflatex

Posted: 2015-05-09T19:54:27-07:00
by fmw42
If you looked at my previous post, you will see where I got wizard.jpg. Sorry I did not read your original post carefully enough to use wizard:

"I downloaded the image from http://www.imagemagick.org/script/forma ... tin-images and it is not 480x640, but when processed produces images that are 480x640. So something seems wrong with that file."

That was the file I found after you put up your post. I have not tested with the wizard: file.

I do not know what to tell you. I think you need to wait for the IM developers to answer.

Re: IM-produced "standard" JPEGs don't work with LaTeX/pdflatex

Posted: 2015-05-10T03:32:15-07:00
by dlemstra
Did you try the test file that I gave you?

Re: IM-produced "standard" JPEGs don't work with LaTeX/pdflatex

Posted: 2015-05-10T06:40:42-07:00
by pipitas
fmw42 wrote:If you looked at my previous post, you will see where I got wizard.jpg.
Ah, sorry, hadn't read it when I posted my answer.

I started to write up my answer about 1 day before I posted it. But then I got distracted by other business and couldn't complete it. So for more than a day I kept coming back to the open window with my draft, adding bits and pieces, getting distracted again, coming back... So during that time I never saw an update to the thread.

Sorry again.

Re: IM-produced "standard" JPEGs don't work with LaTeX/pdflatex

Posted: 2015-05-10T06:46:37-07:00
by pipitas
dlemstra wrote:Did you try the test file that I gave you?
I don't know which testfile you mean? Is it this:

Code: Select all

kp@mbp:>  convert   http://www.imagemagick.org/image/wizard.png  wizard-from-website-converted.jpg
kp@mbp:>  identify  http://www.imagemagick.org/image/wizard.png  wizard-from-website-converted.jpg 

 http://www.imagemagick.org/image/wizard.png=>wizard.png PNG 1104x1468 1104x1468+0+0 8-bit sRGB 1.363MB 0.000u 0:00.000
 wizard-from-website-converted.jpg[1] JPEG 1104x1468 1104x1468+0+0 8-bit sRGB 294KB 0.000u 0:00.000

Re: IM-produced "standard" JPEGs don't work with LaTeX/pdflatex

Posted: 2015-05-10T08:08:05-07:00
by dlemstra
The link I posted earlier: https://www.dropbox.com/s/97w3q16wdjeyp ... t.jpg?dl=0. Does this image work for you?

Re: IM-produced "standard" JPEGs don't work with LaTeX/pdflatex

Posted: 2015-05-10T11:41:59-07:00
by pipitas
dlemstra wrote:The link I posted earlier: https://www.dropbox.com/s/97w3q16wdjeyp ... t.jpg?dl=0. Does this image work for you?
You mean "Does it work with LaTeX for you"? Yes, it does!

So for any visitor from the future reading this thread, here are the differences between the two files in question:

Code: Select all

$>  convert wizard: wiz-stddensity.jpg
$>  wget https://www.dropbox.com/s/97w3q16wdjeypmi/wizard-test.jpg

$>  identify wizard-test.jpg wiz-stddensity.jpg 
  wizard-test.jpg JPEG 480x640 480x640+0+0 8-bit sRGB 80.3KB 0.000u 0:00.000
  wiz-stddensi.jpg[1] JPEG 480x640 480x640+0+0 8-bit sRGB 62.2KB 0.000u 0:00.000

$> file wiz-stddensity.jpg wizard-test.jpg 
  wiz-stddensity.jpg: JPEG image data, JFIF standard 1.01, resolution (DPI), density 1x1, segment length 16, baseline, precision 8, 480x640, frames 3
  wizard-test.jpg:  JPEG image data, JFIF standard 1.01, aspect ratio, density 1x1, segment length 16, baseline, precision 8, 480x640, frames 3

$> exiftool wiz-stddensity.jpg | grep -E 'Resolution|Unit'
  Resolution Unit                 : inches
  X Resolution                    : 1
  Y Resolution                    : 1

$> exiftool wizard-test.jpg | grep -E 'Resolution|Unit'
  Resolution Unit                 : None
  X Resolution                    : 1
  Y Resolution                    : 1
File 'wizard-test.jpg' (the one Dirk linked to) works with LaTeX. The one created by the most simple 'convert' command available does not work. 'xelatex' and 'lualatex' stop with error messages when generating a PDF that should include the JPEG, while 'pdflatex' emits a seemingly empty page consuming ~60 KiBytes.

xelatex:

Code: Select all

$> xelatex mini.tex
  [....]
  ! Dimension too large.
  <to be read again> 
                     b
  l.7 \includegraphics{./testwiz.jpg}
                                     
  ? 
lualatex:

Code: Select all

$> lualatex mini.tex
  [....]
  LuaTeX warning: arithmetic: number too big
  ! Dimension too large.
  <recently read> \dimen@ 
          
  l.7 \includegraphics{./testwiz.jpg}
                                   
  ?
pdflatex:

Code: Select all

$> pdflatex mini.tex
  [....]
  pdfTeX warning: pdflatex: arithmetic: number too big
  <./testwiz.jpg, id=1, --32768.0pt x 0.0pt> <use ./testwiz.jpg> [1 <./testwiz.jpg>] (./mini.aux) )
  Output written on mini.pdf (1 page, 63350 bytes).
  Transcript written on mini.log.
So with 'pdflatex' a PDF is created, but the emitted file only shows an empty white page. (The image IS embedded -- I could even extract it from the PDF in one piece, but it was located way off the actual PDF page dimensions. I can hand-edit the PDF file and just change either the `/MediaBox` dimensions, or the `cm` operator parameters for scaling [or both] to make it visible.... However this is not a concern for *this* forum...)

For those interested, the `mini.tex` file used by the commands above is only 9 lines of LaTeX code:

Code: Select all

\documentclass[a4paper]{article}
\usepackage{graphicx}
\begin{document}
\thispagestyle{empty}
\begin{figure}[htbp]
\centering
\includegraphics{./testwiz.jpg}
\end{figure}
\end{document}