Imagemagick deepzoom - tiling and resizing huge pictures

[snibgo]

Another failed test, which I have reported as a bug: viewtopic.php?f=3&t=24653

[kaefert]

hmm, well I guess the developers of most tools don't test their creations with pictures of such dimensions, probably because of the incredible time requirements to do so...

[snibgo]

I've downloaded your wall. As I haven't yet successfully created an image of the same size, I'll use yours.

I did create a tenth-size image:

Code: Select all

%IM%convert ^
  -verbose ^
  -monitor ^
  -synchronize ^
  -limit memory 32 -limit map 32 ^
  -define registry:temporary-path=h:\temp ^
  -size 111394x11139 xc:white ^
  h:\huge\huge.mpc

%IM%convert ^
  -verbose ^
  -monitor ^
  -define registry:temporary-path=h:\temp ^
  h:\huge\huge.mpc ^
  -crop 871x871 ^
  -set filename:tile "%%[fx:page.x/871+1]_%%[fx:page.y/871+1]" ^
  +repage ^
  +adjoin "h:\huge\tiles_%%[filename:tile].jpg"

With "-monitor" I could see it was creating one crop in memory every 10 seconds or so. "-monitor" does slow things down, but if we assume 10 seconds, then 16384 crops will take 45.5 hours.

I have started this command ...

Code: Select all

%IM%convert ^
  -verbose ^
  -synchronize ^
  -limit memory 0 -limit map 0 ^
  -define registry:temporary-path=h:\temp ^
  h:\wall\wall1-front-merged.tif ^
  -crop 871x871 ^
  -set filename:tile "%%[fx:page.x/871+1]_%%[fx:page.y/871+1]" ^
  +repage ^
  +adjoin "h:\wall\tiles_%%[filename:tile].jpg"

... and we will see what happens. It takes hardly any CPU. The disk is external, on USB, I forget which version.

[kaefert]

well 45 hours are long past since I've started my cropping.
htop tells me my convert process has used over 94 hours of processing time, and I think it have been around 5 days in real time.
Next time I'll be sure to use -verbose and -monitor to know whats going on...

ps.: what does the -synchronize parameter do?
pps: for me it should be faster, since I store everything on my SSD which should give around 500MB/sec read & write speed.. Though with the currently running process I guess he is severly slowed down because of the huge memory usage that makes him swap to disk a lot.

[snibgo]

See http://www.imagemagick.org/script/comma ... ynchronize

When storing an image cache for a large image on disk, if the disk is out of space IM should fail when the cache is created, instead of slowly filling it up an failing a few hours later.

My temp area now has a few magick-* files sized 6,096,128 bytes. 871*871*8 = 6,096,128, so these are the individual crops. Sadly, they are being created slightly slower than one per minute. 16384 minutes = 11.4 days. Oh dear.

[snibgo]

... And it took 2 hours 9 minutes to read the tiff image. So one convert per output file is a very bad idea; it would take 16384 * 2 hours = nearly 4 years!!

Maybe one convert, read the file once, then make each individual crop, as I outlined above.

[fmw42]

... And it took 2 hours 9 minutes to read the tiff image. So one convert per output file is a very bad idea; it would take 16384 * 2 hours = nearly 4 years!!

Maybe one convert, read the file once, then make each individual crop, as I outlined above.

Did you convert the image to MPC format first and then try one convert for each. Once the MPC file is written, it is memory mapped and reading becomes very fast. But it takes a little longer to create the MPC than other formats.

If you use clones, you will need to have two copies in memory, the original and the clone. With the MPC, you only have to read it, which should be fast.

I am undecided which approach might be best.

Perhaps a test on a large image (but not as large as you have now) would identify which approach is faster.

[snibgo]

I haven't yet tried cropping from mpc. Experiments are ongoing.

I have successfully created a large png. See viewtopic.php?f=3&t=24653

Making the 16384 crops from the tiff, it has created 1697 temporary files in 13 hours 7 minutes, which is 2.16 per minute. Looking at the timestamps they are created either at 1 per minute or in clusters of 6 per minute. I don't know what causes this variation, or if it might help in the quest for a fast solution.

16384 at 2.16/minute is 126 hours, which is 5.27 days. It will also need time to create the cropped files.

[kaefert]

cropping the mpc is increadibly fast for me, with verbose and monitor it takes around 6 seconds, without them half a second -->

Code: Select all

$ date +%H:%M:%S:%N
07:24:38:563228784
$ convert wall1-front-merged.mpc -crop 871x871+1742+1742 +repage w1t_2_2.jpg
$ date +%H:%M:%S:%N
07:24:38:986028290

[fmw42]

Did you try adding -limit memory 32 -limit map 32 to see if that made it even faster?

[kaefert]

Did you try adding -limit memory 32 -limit map 32 to see if that made it even faster?

I did now, it seems it does! thanks for the hint!

Code: Select all

$ date +%H:%M:%S:%N
07:30:47:174675193
$ convert -limit memory 32 -limit map 32 wall1-front-merged.mpc -crop 871x871+1742+1742 +repage w1t_2_2.jpg
$ date +%H:%M:%S:%N
07:30:47:230184068

It seems this makes it possible to do a crop in 55 milliseconds!

[kaefert]

hmmm, what's wrong here? I can't finde the mistake - the first line works, the second one fails

Code: Select all

$ convert -limit memory 32 -limit map 32 wall1-front-merged.mpc -crop 871x871+1742+1742 +repage w1t_2_2.jpg
$ convert -limit memory 32 -limit map 32 wall1-front-merged.mpc -crop 871x871+85358+41808 +repage w1t_98_48.jpg
convert.im6: invalid argument for option `-crop': 871x871+85358+41808 +repage @ error/convert.c/ConvertImageCommand/1093.

update: lol, found it, the space before "+repage" in the first line is the default hex code 20 it should be, and in the second line its hex "C2 A0" and I don't know why gedit (my texteditor) will show it as space. I've produced the second line by copying it out of libreoffice calc and removing the tabulators by search and (empty) replace.

update: doing all the crops at the same time and each just once (no cheating by caching because of repeatedly doing the same crop) is a bit slower.

here's the output of a little monitor loop I've written - first it prints the time second the count of tiles already created (and it only prints a line if the count changed)

Code: Select all

08:22:53:283630889 - 2960
08:22:56:623716803 - 2961
08:22:56:723530474 - 2962
08:22:56:806320680 - 2963
08:22:56:888894938 - 2964
08:22:56:940312370 - 2965
08:22:57:093029277 - 2967
08:22:57:135605990 - 2969
08:22:57:180598653 - 2970
08:22:57:316549390 - 2971
08:22:57:379145968 - 2972
08:22:57:416258997 - 2973
08:22:57:486911767 - 2974
08:22:59:994201486 - 2975
08:23:00:084575967 - 2976
08:23:00:198056599 - 2977
08:23:00:297023653 - 2978
08:23:00:355776166 - 2979
08:23:00:431515756 - 2980
08:23:00:505943377 - 2981
08:23:00:576497461 - 2982
08:23:00:660770677 - 2983
08:23:00:727945071 - 2985
08:23:00:762333034 - 2986
08:23:00:814077603 - 2987
08:23:00:940853012 - 2988
08:23:01:059324921 - 2989
08:23:01:211528075 - 2990
08:23:01:274242684 - 2991
08:23:01:307822862 - 2992
08:23:01:331004720 - 2993

[snibgo]

Good stuff. So the trick is to crop from mpc? Excellent.

(My poor old laptop is still struggling to create the mpc! Ha!)

[kaefert]

Good stuff. So the trick is to crop from mpc? Excellent.

(My poor old laptop is still struggling to create the mpc! Ha!)

I did it over night, I guess it took around 6 hours. (on my old laptop, my new one is still working on my first try (see first post)
I think I'm gonna cancel that one now

[kaefert]

okey, so the route to tiling the original image is clear, now I need to make it a pyramid - so tiles that are downscaled so that one tile will represent 4 tiles of the layer above it

So the quality wise best route would surly be to resize the orignal image like that:

Code: Select all

convert -verbose -monitor -limit memory 2G -limit map 4G wall2-right_merged.mpc -resize 55697x55697 wall2-right_merged_55697px.mpc

and then do the tiling the same way we descussed above.

But I'm wondering - If I would not mind the quality loss from multiple lossy jpeg encodings - would it be quicker to join 4 tiles of the first layer and resize the result?