Imagemagick deepzoom - tiling and resizing huge pictures

[kaefert]

Hi there!

I've got 6 pictures with dimensions of 111.394 width & height in pixels (12.408.623.236 pixels) which are walls of a panorama cube.

Code: Select all

$ identify wall1-front-merged.tif 
wall1-front-merged.tif TIFF 111394x111394 111394x111394+0+0 8-bit DirectClass 3.7423GB 0.000u 0:00.000

To display panoramas I like to use the open source flash software SaladoPlayer http://panozona.com/wiki/SaladoPlayer
This software nees those panorama cube walls in a deep zoom tiled format similar to the format defined by Microsoft (easily convertable by just resorting the tiles with for example a shell script).

The problem I have, is that both the tools that I previously used to do the deep zoom tiling
(SaladoConverter http://panozona.com/wiki/SaladoConverter & imgcnv http://biodev.ece.ucsb.edu/projects/imgcnv) have failed do to the huge size of pictures.

At first I found this deepzoom tool http://search.cpan.org/dist/Graphics-DZI/ though it didn't work as I hoped:

Code: Select all

$ deepzoom wall1-front-merged.tif
something is wrong with 'wall1-front-merged.tif' at /usr/local/bin/deepzoom line 189

Next I thought that even if the all-in-one tool doesn't work, Imagemagick itself can both scale down and tile pictures, so I could just do those things manually after each other in a shellscript. So a few days ago I started this command:

Code: Select all

$ convert ../wall1.tif -crop -871x871 -set filename:tile "%[fx:page.x/871+1]_%[fx:page.y/871+1]" +repage +adjoin "tiles_%[filename:tile].jpg"

And I haven't gotten any output yet. At first it wrote a 93GB file to /tmp/magick-f-OzEAmq quite fast, and since then it does use 100% of one of my CPU-cores and very slowly fills my memory. Now after 80 hours of CPU-usage (according to htop) it has climbed up to 56.2GB of RAM usage (which I provided by creating and mounting multiple swap files). Will this command ever finish and provide the output I hoped for? (hope = tiles with a size of 871 pixel width&height)

How can I optimize the tiling and resizing of such huge pictures?
I've found http://www.imagemagick.org/Usage/files/#massive but I have a hard time judging which of the described methods is the best one for my situtation.

edit: ps: I know that Imagemagick can successfully read and interpret my image, since this command:

Code: Select all

convert wall1-front-merged.tif -resize 870x870 wall1-front-merged_870px.jpg

resulted in a sane resized small variant of the input picture.

[fmw42]

I am not an expert on this, but see
http://www.imagemagick.org/Usage/files/#massive
http://www.imagemagick.org/script/resou ... nvironment
http://www.imagemagick.org/script/archi ... tera-pixel

Also IM can convert your images to PTIF format that is multi-resolution and tiled. see the TIFF and PTIF sections of
http://www.imagemagick.org/script/formats.php#supported

[kaefert]

thanks for the response fmw42!

This PTIF format could be helpful, but my final outcome needs to be tiled zoom steps with each tile being a simple jpeg file inside a folder with all other tiles of the same zoom step.

Also thanks for all those links, especially I found it helpful to learn about the "-monitor" parameter, I will definitly use this one in the future to see whats going with my process.

I still would be very happy if somebody with experience in processing huge pictures could give me an idea of whats the best way to go in my situation.

And an update on my running tiling process --> still no output, but it seems he now reached a step where he want's to read all those gigabytes of stuff he wrote into the RAM, since he's now swapping pages between my 10GB true RAM and 70GB SSD-based swap-files a lot. It also seems that he is not doing this big junks at a time - since the read speed is nowhere near the maximum of my SSD but the wait percentage of the CPU is still quite high.

[snibgo]

There is a minor error in your command. "-crop -871x871" should be "-crop 871x871" (without "-" before the number). An experiment suggests that this doesn't cause problems, but I suggest you modify your script.

Just for fun, I'm replicating your command on my computer. But I don't expect any great revelation.

[kaefert]

Hi there snibgo! Thanks for the reply!

I think I've read somewhere that "-crop 871x871" would interpret the numbers as the number of tiles that the height and width should be split into, and "-crop -871x871" will intrepret the numbers as the pixel dimensions of the target tiles and create as many tiles as necessary.

But maybe that guy was wrong...

[snibgo]

I don't think that explanation of "-" is correct. Leaving off "-" always interprets the numbers as pixels, not tiles, unless "@" is used. The documentation (http://www.imagemagick.org/script/comma ... s.php#crop) says "@" after the crop means the numbers are of tiles, not pixels.

[kaefert]

ups.. okey, thanks snibgo for the clarification.

I'm unsure if I should cancel my running imagemagick tiling process, it's mostly just moving pages between the RAM and the swap now and therefore waiting for IO. It didn't print any console output and it didn't create any file but the one 90GB file in /tmp/

Do you think it will ever finish and give me a usable result? Anybody got concrete tips for how to optimize the process so that it doesn't take weeks? Other tools just take a few hours for that kind of task.

[snibgo]

I'm still trying to create the 111394x111394 image, let alone cropping into tiles.

My task failed trying to create an output buffer, so I'm retrying with "-define tiff:rows-per-strip=1". If all the rows are in a single strip which won't fit into physical RAM this might clobber the speed of each individual crop. So it might be worth you trying this.

Another thought: when IM does the "-crop 871x871", each crop will be saved in memory (as a pixel cache), and only written to disk at the end of the command. Each image is small but there will be 16384 of them. It may be best to ensure that no pixel caches are stored in RAM with "-limit memory 32 -limit map 32". I haven't tried this.

Incidentally, a specialist cropping tool probably writes each crop to a file as it is created, instead of holding them all in memory. You could force IM to do this, but it would need 16384 lines like:

Code: Select all

( +clone -crop 871x871+{offsx}+{offsy} -write {filename} +delete )

[kaefert]

if you want one of the walls as example I can put them on a webserver and send you a link via PM, just say a word if you're interested!

About you're suggestions, I've tried to put them into an example - did I understand you correctly, would this be a good idea for my next try? (I'm still running the line from my first post)

Code: Select all

convert wall1-front-merged.tif -monitor -limit memory 32 -limit map 32 \
( +clone -crop 871x871+0+0 -write w1t_0_0.jpg +delete ) \
( +clone -crop 871x871+871+0 -write w1t_1_0.jpg +delete ) \
( +clone -crop 871x871+1742+0 -write w1t_2_0.jpg +delete ) \
( +clone -crop 871x871+0+871 -write w1t_0_1.jpg +delete ) \
( +clone -crop 871x871+871+871 -write w1t_1_1.jpg +delete ) \
( +clone -crop 871x871+0+1742 -write w1t_0_2.jpg +delete )
# ... (just a manually written example, I would probably create the list with a shell script if you think thats the way to go.)

and what was that with the "-define tiff:rows-per-strip=1" ? how does that work? what will that accomplish? I mean he will need 871 rows of the picture for each tile, won't he?

[snibgo]

Yes, if you send me a link by PM, I can try and download it to experiment.

About you're suggestions, I've tried to put them into an example - did I understand you correctly, would this be a good idea for my next try?

Yes, that script is what I meant. It might be quicker. Or it might not. For Unix, parentheses must be escaped so change "(" to "\(" and ")" to "\)".

and what was that with the "-define tiff:rows-per-strip=1" ? how does that work? what will that accomplish? I mean he will need 871 rows of the picture for each tile, won't he?

Yes, it will still need 871 rows. If the input tiff file has 1 row per strip, it need to read 871 strips. If all the rows are in 1 strip then perhaps it needs to read all the rows for every crop. But this is just a guess. Creating the input tiff file with "-define tiff:rows-per-strip=1" might make cropping faster. Or it might not.

Another thought: Creating the input tiff file with "-define tiff:rows-per-strip=1" and "-define tiff:tile-geometry=871x871" might make cropping faster. Or it might not.

[kaefert]

Code: Select all

convert -define tiff:rows-per-strip=1 wall1-front-merged.tif -monitor -limit memory 32 -limit map 32 \
\( +clone -crop 871x871+0+0 -write w1t_0_0.jpg +delete \) \
\( +clone -crop 871x871+871+0 -write w1t_1_0.jpg +delete \) \
\( +clone -crop 871x871+1742+0 -write w1t_2_0.jpg +delete \) \
\( +clone -crop 871x871+0+871 -write w1t_0_1.jpg +delete \) \
\( +clone -crop 871x871+871+871 -write w1t_1_1.jpg +delete \) \
\( +clone -crop 871x871+0+1742 -write w1t_0_2.jpg +delete \)

Or should we maybe try to do it like that?

Code: Select all

convert wall1-front-merged.tif image.mpc
convert image.mpc -crop 871x871+0+0 +repage w1t_0_0.jpg
convert image.mpc -crop 871x871+871+0 +repage w1t_1_0.jpg
convert image.mpc -crop 871x871+1742+0 +repage w1t_2_0.jpg

(taken from http://www.imagemagick.org/script/archi ... tera-pixel )

[snibgo]

Either should work. The second version will re-read the MPC file for every output, which might make it very slow. "+repage" isn't needed when the output is JPG, but it is harmless. In the first version, the "-define tiff:rows-per-strip=1" is redundant as the output isn't tiff. It also needs a final output file, but this can be NULL:.

Code: Select all

convert -define tiff:rows-per-strip=1 wall1-front-merged.tif -monitor -limit memory 32 -limit map 32 \
\( +clone -crop 871x871+0+0 -write w1t_0_0.jpg +delete \) \
\( +clone -crop 871x871+871+0 -write w1t_1_0.jpg +delete \) \
\( +clone -crop 871x871+1742+0 -write w1t_2_0.jpg +delete \) \
\( +clone -crop 871x871+0+871 -write w1t_0_1.jpg +delete \) \
\( +clone -crop 871x871+871+871 -write w1t_1_1.jpg +delete \) \
\( +clone -crop 871x871+0+1742 -write w1t_0_2.jpg +delete \) \
NULL:

Try scripts on small files first!

[kaefert]

Ah okey, with creating the input image you really meant creating the input file in a previous step. I've created it with gimp, and I don't know if and where gimp would allow to set such parameters. I've used LZW compression to save it, maybe it would have been better to use uncompressed tiff.

But if you think my input file might be a problem for efficient cropping I could first convert it with imagemagick in a previous step before cropping, maybe I should create an mpc file and use that one? according to the documentation I've read this is the most efficient one to read for imagemagick. Or do you think a tiff created with this "-define tiff:rows-per-strip=1" parameter would be better?

[snibgo]

I'm sorry, I don't know. My guess is that cropping from an MPC file will be best. So, with either method, you could convert to MPC first, and crop from there.

Or maybe a tiled TIFF will be best.

I'm doing tests, and will do with your file when it downloads, but every test takes hours (or days or weeks...).

[kaefert]

I'm doing tests, and will do with your file when it downloads, but every test takes hours (or days or weeks...).

Yep I know.. Really huge thanks from me for helping me with this!