Problems With IM 6.4.0-3 and 6.4.0-7 both Q16 HDRI

[fmw42]

Hi,

While testing Sean Burke's FFT routines in IM 6.4.0-3 Q16-hdri, I ran into several issues that I wanted to bring to your attention. See note at bottom as I cannot get the FFT to run under IM 6.4.0-7 and had to back down to IM 6.4.0-3.

1. I have an image that represents the magnitude of the FFT of a motion blur spatial filter.
(The spatial blur filter before the FFT can be viewed at
http://www.fmwconcepts.com/misc_tests/F ... lt_h15.png
It is just a 15x1 pixel white line in a black background)

The magnitude image will be very dark as it has very small values. The DC (center point), which will be the largest valued pixel, will have a value that is the average of the original image, which is mostly black, thus a very small value. The magnitude is stored in MIFF format to preserve the small floating point values. The magnitude of the FFT of a 15x1 pixel constant-value line should be a sinc function in the horizontal direction. So I wanted to scale the image for viewing and to make a profile to check it out.

Here is the magnitude image:
http://www.fmwconcepts.com/misc_tests/F ... fft-0.miff

I therefore used -contrast-stretch 0% to stretch the image linearly to make the minimum value go to 0 and the maximum value go to quantum-max. Please correct me if I misunderstand what -contrast-stretch 0% is supposed to do. I then used my profile script to plot the profile. (You can get my profile script at http://www.fmwconcepts.com/imagemagick/index.html, if you desire).

Here is the resulting profile:
http://www.fmwconcepts.com/misc_tests/F ... rofile.gif

Note that the peaks are clipped flat. This implies to me that in HDRI, -contrast-stretch 0% is NOT preserving the floating point values properly.

Therefore, to verify that assumption, I scaled the magnitude by getting the minimum and maximum values from -verbose info and used -fx. This produced a proper looking profile:
http://www.fmwconcepts.com/misc_tests/F ... rofile.gif

So the question is should -contrast-stretch 0%, when used in Q16-hdri environment, preserve the floating point values properly while scaling. If it should, can you look into correcting this?

2) I have another image which is just a white 31x31 square in a black background.
http://www.fmwconcepts.com/misc_tests/F ... uare31.png

I did an FFT on this and wanted to look at the phase image. This image will have values that range within -pi to +pi (-3.14 to +3.14). Thus they are extremely small values in a Q16 environment, even with HDRI. I wanted to scale this image for viewing, but ran into trouble using both bc and fx to compute the scaling coefficients, because they get reported by IM fx in scientific numbers (after dividing by quantum-max) and computing the difference gives meaningless numbers. Here is the unscaled phase image:
http://www.fmwconcepts.com/misc_tests/F ... fft-1.miff


If I scale it using -contrast-stretch 0%, I get:
http://www.fmwconcepts.com/misc_tests/F ... -1_cs0.png
which shows diagonal banding that should not be there.

So I attempted to get the scaling values to use with -fx as follows:

from -verbose info I got:
min: -3.12932 (-4.77504e-05)
max: 3.12932 (4.77504e-05)

But when I tried to automate this as follows, I got:

freds-mac-mini:~ fred$ config=`convert -list configure`
freds-mac-mini:~ fred$ quantum=`echo "$config" | sed -n 's/^QuantumDepth[ ]*\([0-9]*\)$/\1/ p'`
freds-mac-mini:~ fred$ qmax=`echo "scale=0; (2^$quantum - 1) / 1" | bc`
freds-mac-mini:~ fred$ echo "qmax=$qmax"
qmax=65535
freds-mac-mini:~ fred$ min=`convert $phase -format "%[min]" info:`
freds-mac-mini:~ fred$ max=`convert $phase -format "%[max]" info:`
freds-mac-mini:~ fred$ min=`convert xc: -format "%[fx:$min/$qmax]" info:`
freds-mac-mini:~ fred$ max=`convert xc: -format "%[fx:$max/$qmax]" info:`
freds-mac-mini:~ fred$ diff=`convert xc: -format "%[fx:($max-$min)]" info:`
freds-mac-mini:~ fred$ echo "min=$min"
min=-4.77504e-05
freds-mac-mini:~ fred$ echo "max=$max"
max=4.77504e-05
freds-mac-mini:~ fred$ echo "diff=$diff"
diff=15.9598

Note the min and max values are reported in agreement with -verbose info, but the diff is nonsense. I assume this occurs because min and max are in scientific notation and -fx perhaps cannot handle that as input values. Can you confirm if this should work or not? If it should, can this be easily fixed.

To confirm, I therefore did the following by re-expressing the values as decimal values:

freds-mac-mini:~ fred$ min=-0.0000477504
freds-mac-mini:~ fred$ max=0.0000477504
freds-mac-mini:~ fred$ diff=`convert xc: -format "%[fx:($max-$min)]" info:`
freds-mac-mini:~ fred$ echo "diff=$diff"
diff=9.55008e-05

Which now reports the difference correctly.

Therefore, I converted this diff to decimal value and used the min in decimal value to do the scaling in -fx:
convert $phase -fx "(u+0.0000477504)/0.0000955008" square31_fft-1_csfx.png
which produced the following smooth and correctly scaled phase image:
http://www.fmwconcepts.com/misc_tests/F ... 1_csfx.png

(Note: I believe that I can use -evaluate in place of -fx to do the scaling as the former uses values 0 to quantum-max rather than 0 to 1 and therefore the values reported by -fx for the min and max of the image using %[min] and %[max] need not be divided by quantum-max. Therefore the diff should be properly computed.)


Nevertheless, the bottom line is that there are two issues:
1) -contrast-stretch 0% is not preserving the floating point values properly while scaling in an HDRI configuration.
2) -fx calculations do not seem to work with scientific notation variables


Thanks.

Fred

P.S.

I have not been able to get Sean's same FFT routine to work in IM 6.4.0-7. It hangs while trying to read the image. He is looking further into it. But I was wondering if you know of anything that might have changed since IM 6.4.0-3 that might be causing this. (Perhaps something to do with another change with Ghostscript, as with a previous problem that Sean reported just before IM 6.4.0-3)

[magick]

The Fx option does not support scientific notation. We consider this a bug and will have a patch in the near future.

[magick]

Its possible a few of the algorithms need tweaking to properly support HDRI. The contrast stretch algorithm generates a histogram map where values are clamped to [0 .. QuantumRange]. We will need to enhance the algorithm to support out-of-range pixel values.

[fmw42]

With regard to -contrast-stretch, I don't think it is an issue with out-of-range, but one of not preserving the fractional component of the number. Correct me if I am wrong, but by histogramming, are you then only using integer ranges for bins or integers for the bins, thus losing the fractional component by outputting a limited set of values? If this is the case, then a totally different approach is needed to scale/stretch floating point values, which is not consistent with your current approach. Thus, I should not consider this something that will be changed or changed in the near future to provide such a different approach/function and should continue to use my workaround with -evaluate when in HDRI?

Also, do you have any idea what might have changed between IM 6.4.0-3 and 6.4.0-7 such that I cannot run Sean's FFT function in 6.4.0-7, but can in 6.4.0-3?

[magick]

The contrast algorithm does do binning so it does lose the fractional component of a pixel.

We would need more insight into exactly what problem you are having with the FFT routines before we could offer any help. Is there any way to reproduce the problem from the command line? If not, you will need to post detailed instruction on how to reproduce the problem.

[seanburke1979]

Fred's issue seems to be connected to the addition of his ghostscript font directory in the configure step. Fred, if you have a moment, it may help for you to post your standard config command. The FFT routines compile fine on all recent versions of IM that do not have any additional config alterations concerning ghostscript. On Fred's system, the program hangs when calling the function "ReadImages" for the MagickCore API.
Hope this helps.

Sean

[fmw42]

Here is my current configuration under IM 6.4.0-3. But it works here. I will rebuild under IM 6.4.0-7 and provide my configuration under that.

convert -list configure

Path: /usr/local/lib/ImageMagick-6.4.0/config/configure.xml

Name Value
-------------------------------------------------------------------------------
CC gcc
CFLAGS -I/usr/local/include/lqr-1 -I/usr/local/include/glib-2.0 -I/usr/local/lib/glib-2.0/include -g -O2 -Wall -W -D_THREAD_SAFE
CONFIGURE ./configure --enable-delegate-build --enable-shared --disable-static --with-modules --with-quantum-depth=16 --enable-hdri --without-wmf --enable-libtool-verbose --disable-dependency-tracking --with-gs-font-dir=/usr/local/share/ghostscript/fonts/ --with-lqr
COPYRIGHT Copyright (C) 1999-2008 ImageMagick Studio LLC
CPPFLAGS -I/usr/local/include/ImageMagick
CXX g++
CXXFLAGS -g -O2 -Wall -W -D_THREAD_SAFE
DEFS -DHAVE_CONFIG_H
DELEGATES bzlib fontconfig freetype gs jpeg jp2 lcms lqr openexr png tiff x11 xml zlib
DISTCHECK_CONFIG_FLAGS --disable-deprecate --with-quantum-depth=16 --with-djvu=no --with-umem=no --with-wmf=no --with-fontpath= --with-gs-font-dir=/usr/local/share/ghostscript/fonts/
EXEC-PREFIX /usr/local
HOST powerpc-apple-darwin8.11.0
LDFLAGS -L/usr/local/lib -L/Users/fred/Applications/ImageMagick-6.4.0-3source/magick -L/Users/fred/Applications/ImageMagick-6.4.0-3source/wand -L/usr/X11R6/lib -L/usr/local/lib -lfreetype -lz
LIB_VERSION 0x640
LIB_VERSION_NUMBER 6,4,0,3
LIBS -lMagickCore -llcms -ltiff -lfreetype -ljpeg -L/usr/local/lib/lqr-1 -L/usr/local/lib -llqr-1 -lglib-2.0 -lintl -liconv -L/usr/local/lib -lfontconfig -lXext -lSM -lICE -lX11 -lXt -lbz2 -lz -lm -lpthread
NAME ImageMagick
PCFLAGS
PREFIX /usr/local
QuantumDepth 16
RELEASE_DATE 04/19/08
VERSION 6.4.0
WEBSITE http://www.imagemagick.org

[fmw42]

I just tried to install IM 6.4.3-8 and am getting error messages at the make stage.

Here is my configure:

./configure --enable-delegate-build --enable-shared --disable-static \
--with-modules --with-quantum-depth=16 --enable-hdri --without-wmf \
--enable-libtool-verbose --disable-dependency-tracking \
--with-gs-font-dir=/usr/local/share/ghostscript/fonts/ --with-lqr


Here is the errors at the end of the make:

/usr/bin/libtool: internal link edit command failed
make[1]: *** [magick/libMagickCore.la] Error 1
make: *** [all] Error 2


I will go back to IM 6.4.0-7.

[fmw42]

Here is my configuration under IM 6.4.0-7 (as I cannot build IM 6.4.0-8) and after rebuilding 6.4.0-7 and rebuilding imfft, I still cannot run imfft without it hanging.

convert -list configure

Path: /usr/local/lib/ImageMagick-6.4.0/config/configure.xml

Name Value
-------------------------------------------------------------------------------
CC gcc
CFLAGS -I/usr/local/include/lqr-1 -I/usr/local/include/glib-2.0 -I/usr/local/lib/glib-2.0/include -g -O2 -Wall -W -D_THREAD_SAFE
CONFIGURE ./configure --enable-delegate-build --enable-shared --disable-static --with-modules --with-quantum-depth=16 --enable-hdri --without-wmf --enable-libtool-verbose --disable-dependency-tracking --with-gs-font-dir=/usr/local/share/ghostscript/fonts/ --with-lqr
COPYRIGHT Copyright (C) 1999-2008 ImageMagick Studio LLC
CPPFLAGS -I/usr/local/include/ImageMagick
CXX g++
CXXFLAGS -g -O2 -Wall -W -D_THREAD_SAFE
DEFS -DHAVE_CONFIG_H
DELEGATES bzlib fontconfig freetype gs jpeg jp2 lcms lqr openexr png tiff x11 xml zlib
DISTCHECK_CONFIG_FLAGS --disable-deprecate --with-quantum-depth=16 --with-djvu=no --with-umem=no --with-wmf=no --with-fontpath= --with-gs-font-dir=/usr/local/share/ghostscript/fonts/
EXEC-PREFIX /usr/local
HOST powerpc-apple-darwin8.11.0
LDFLAGS -L/usr/local/lib -L/Users/fred/Applications/ImageMagick-6.4.0-7source/magick/.libs -L/Users/fred/Applications/ImageMagick-6.4.0-7source/magick -L/Users/fred/Applications/ImageMagick-6.4.0-7source/wand/.libs -L/Users/fred/Applications/ImageMagick-6.4.0-7source/wand -L/usr/X11R6/lib -L/usr/local/lib -lfreetype -lz
LIB_VERSION 0x640
LIB_VERSION_NUMBER 6,4,0,7
LIBS -lMagickCore -llcms -ltiff -lfreetype -ljpeg -L/usr/local/lib/lqr-1 -L/usr/local/lib -llqr-1 -lglib-2.0 -lintl -liconv -L/usr/local/lib -lfontconfig -lXext -lSM -lICE -lX11 -lXt -lbz2 -lz -lm -lpthread
NAME ImageMagick
PCFLAGS
PREFIX /usr/local
QuantumDepth 16
RELEASE_DATE 04/21/08
VERSION 6.4.0
WEBSITE http://www.imagemagick.org

[magick]

Did you do a 'make -k -i clean' first? We tried ImageMagick 6.4.0-8 under Tiger just now and it compiled/linked/installed without complaint.

[fmw42]

No, I did not know I had to do that. But IM 6.4.0-7 installed fine after trying IM 6.4.0-8. I will try the make -k - i clean and report back later. I presume I just do:

cd to current build folder
make -k -i clean
cd to IM 6.4.0-8
.configure...
make
sudo make install

Please correct me as I have never done a make -k -i clean before.

But what about the issue with not being able to run Sean's imfft on 6.4.0-7? See Sean's comments in prevous posts above and my configure files. Thanks.

[magick]

Set MAGICK_DEBUG to 'trace,all' and post the last 10 or so lines of the output. That should clue us in where the program is hanging.

[fmw42]

How do you do this? I tried:

freds-mac-mini:~ fred$ export MAGICK_DEBUG "trace,all"
-bash: export: `trace,all': not a valid identifier

Perhaps you would walk me through this as I have no idea how to do this?

Thanks

[magick]

Try

• export MAGICK_DEBUG="trace,all"

[fmw42]

Thanks. That did the trick. Here is the Debug info:

freds-mac-mini:~ fred$ export MAGICK_DEBUG="trace,all"
freds-mac-mini:~ fred$ Applications/imfft/trunk13/demo f twig.png twig_fft.miff
2008-04-21T20:24:37-07:00 0:01 0.010u 6.4.0 Trace ImageMagick[28339]: utility.c/unknown/947/Trace
Applications/imfft/trunk13/demo
2008-04-21T20:24:37-07:00 0:01 0.010u 6.4.0 Trace demo[28339]: utility.c/unknown/947/Trace
Applications/imfft/trunk13/demo
2008-04-21T20:24:37-07:00 0:01 0.010u 6.4.0 Configure demo[28339]: magick.c/unknown/1099/Configure
Register handler for signal: 6
2008-04-21T20:24:37-07:00 0:01 0.010u 6.4.0 Configure demo[28339]: magick.c/unknown/1099/Configure
Register handler for signal: 8
2008-04-21T20:24:37-07:00 0:01 0.010u 6.4.0 Configure demo[28339]: magick.c/unknown/1099/Configure
Register handler for signal: 1
2008-04-21T20:24:37-07:00 0:01 0.010u 6.4.0 Configure demo[28339]: magick.c/unknown/1099/Configure
Register handler for signal: 2
2008-04-21T20:24:37-07:00 0:01 0.010u 6.4.0 Configure demo[28339]: magick.c/unknown/1099/Configure
Register handler for signal: 3
2008-04-21T20:24:38-07:00 0:01 0.010u 6.4.0 Configure demo[28339]: magick.c/unknown/1099/Configure
Register handler for signal: 15
2008-04-21T20:24:38-07:00 0:01 0.010u 6.4.0 Configure demo[28339]: magick.c/unknown/1099/Configure
Register handler for signal: 24


... to last few lines ...


2008-04-21T20:24:38-07:00 0:01 0.020u 6.4.0 Trace demo[28339]: string.c/unknown/771/Trace
...
2008-04-21T20:24:38-07:00 0:01 0.020u 6.4.0 Trace demo[28339]: string.c/unknown/1637/Trace
...
2008-04-21T20:24:38-07:00 0:01 0.020u 6.4.0 Trace demo[28339]: string.c/unknown/1637/Trace
...
2008-04-21T20:24:38-07:00 0:01 0.020u 6.4.0 Trace demo[28339]: string.c/unknown/1637/Trace
...
2008-04-21T20:24:38-07:00 0:01 0.020u 6.4.0 Trace demo[28339]: string.c/unknown/1637/Trace
...
2008-04-21T20:24:38-07:00 0:01 0.020u 6.4.0 Trace demo[28339]: string.c/unknown/1637/Trace
...
2008-04-21T20:24:38-07:00 0:01 0.020u 6.4.0 Trace demo[28339]: string.c/unknown/1637/Trace
...
2008-04-21T20:24:38-07:00 0:01 0.030u 6.4.0 Trace demo[28339]: string.c/unknown/1637/Trace
...
2008-04-21T20:24:38-07:00 0:01 0.030u 6.4.0 Trace demo[28339]: signature.c/unknown/227/Trace
...
2008-04-21T20:24:38-07:00 0:01 0.030u 6.4.0 Trace demo[28339]: signature.c/unknown/329/Trace
...
2008-04-21T20:24:38-07:00 0:01 0.030u 6.4.0 Trace demo[28339]: string.c/unknown/1596/Trace
...
2008-04-21T20:24:38-07:00 0:01 0.030u 6.4.0 Trace demo[28339]: string.c/unknown/771/Trace
...
2008-04-21T20:24:38-07:00 0:01 0.030u 6.4.0 Trace demo[28339]: registry.c/unknown/218/Trace
temporary-path
2008-04-21T20:24:38-07:00 0:01 0.030u 6.4.0 Resource demo[28339]: resource.c/unknown/428/Resource
/var/tmp/magick-rOwHWThW

This is where it hung.

Fred


P.S. I also tried to build IM 6.4.0-8 again after doing make -k -i clean, but it still ended with errors at the make stage. See my other post. What can I provide you to help?