Re: Imagemagick deepzoom - tiling and resizing huge pictures
Posted: 2013-12-17T01:36:31-07:00
Another failed test, which I have reported as a bug: viewtopic.php?f=3&t=24653
Legacy ImageMagick Discussions Archive
Use https://github.com/ImageMagick/ImageMagick/discussions instead.
https://imagemagick.com/discourse-server/
Imagemagick deepzoom - tiling and resizing huge pictures
https://imagemagick.com/discourse-server/viewtopic.php?t=24644
Page 2 of 3
Re: Imagemagick deepzoom - tiling and resizing huge pictures
Posted: 2013-12-17T01:47:47-07:00
hmm, well I guess the developers of most tools don't test their creations with pictures of such dimensions, probably because of the incredible time requirements to do so...
Re: Imagemagick deepzoom - tiling and resizing huge pictures
Posted: 2013-12-17T08:19:32-07:00
I've downloaded your wall. As I haven't yet successfully created an image of the same size, I'll use yours.
I did create a tenth-size image:
With "-monitor" I could see it was creating one crop in memory every 10 seconds or so. "-monitor" does slow things down, but if we assume 10 seconds, then 16384 crops will take 45.5 hours.
I have started this command ...
... and we will see what happens. It takes hardly any CPU. The disk is external, on USB, I forget which version.
I did create a tenth-size image:
Code: Select all
%IM%convert ^
-verbose ^
-monitor ^
-synchronize ^
-limit memory 32 -limit map 32 ^
-define registry:temporary-path=h:\temp ^
-size 111394x11139 xc:white ^
h:\huge\huge.mpc
%IM%convert ^
-verbose ^
-monitor ^
-define registry:temporary-path=h:\temp ^
h:\huge\huge.mpc ^
-crop 871x871 ^
-set filename:tile "%%[fx:page.x/871+1]_%%[fx:page.y/871+1]" ^
+repage ^
+adjoin "h:\huge\tiles_%%[filename:tile].jpg"I have started this command ...
Code: Select all
%IM%convert ^
-verbose ^
-synchronize ^
-limit memory 0 -limit map 0 ^
-define registry:temporary-path=h:\temp ^
h:\wall\wall1-front-merged.tif ^
-crop 871x871 ^
-set filename:tile "%%[fx:page.x/871+1]_%%[fx:page.y/871+1]" ^
+repage ^
+adjoin "h:\wall\tiles_%%[filename:tile].jpg"Re: Imagemagick deepzoom - tiling and resizing huge pictures
Posted: 2013-12-17T08:31:53-07:00
well 45 hours are long past since I've started my cropping.
htop tells me my convert process has used over 94 hours of processing time, and I think it have been around 5 days in real time.
Next time I'll be sure to use -verbose and -monitor to know whats going on...
ps.: what does the -synchronize parameter do?
pps: for me it should be faster, since I store everything on my SSD which should give around 500MB/sec read & write speed.. Though with the currently running process I guess he is severly slowed down because of the huge memory usage that makes him swap to disk a lot.
htop tells me my convert process has used over 94 hours of processing time, and I think it have been around 5 days in real time.
Next time I'll be sure to use -verbose and -monitor to know whats going on...
ps.: what does the -synchronize parameter do?
pps: for me it should be faster, since I store everything on my SSD which should give around 500MB/sec read & write speed.. Though with the currently running process I guess he is severly slowed down because of the huge memory usage that makes him swap to disk a lot.
Re: Imagemagick deepzoom - tiling and resizing huge pictures
Posted: 2013-12-17T09:35:20-07:00
See http://www.imagemagick.org/script/comma ... ynchronize
When storing an image cache for a large image on disk, if the disk is out of space IM should fail when the cache is created, instead of slowly filling it up an failing a few hours later.
My temp area now has a few magick-* files sized 6,096,128 bytes. 871*871*8 = 6,096,128, so these are the individual crops. Sadly, they are being created slightly slower than one per minute. 16384 minutes = 11.4 days. Oh dear.
When storing an image cache for a large image on disk, if the disk is out of space IM should fail when the cache is created, instead of slowly filling it up an failing a few hours later.
My temp area now has a few magick-* files sized 6,096,128 bytes. 871*871*8 = 6,096,128, so these are the individual crops. Sadly, they are being created slightly slower than one per minute. 16384 minutes = 11.4 days. Oh dear.
Re: Imagemagick deepzoom - tiling and resizing huge pictures
Posted: 2013-12-17T09:40:50-07:00
... And it took 2 hours 9 minutes to read the tiff image. So one convert per output file is a very bad idea; it would take 16384 * 2 hours = nearly 4 years!!
Maybe one convert, read the file once, then make each individual crop, as I outlined above.
Maybe one convert, read the file once, then make each individual crop, as I outlined above.
Re: Imagemagick deepzoom - tiling and resizing huge pictures
Posted: 2013-12-17T10:55:15-07:00
Did you convert the image to MPC format first and then try one convert for each. Once the MPC file is written, it is memory mapped and reading becomes very fast. But it takes a little longer to create the MPC than other formats.snibgo wrote:... And it took 2 hours 9 minutes to read the tiff image. So one convert per output file is a very bad idea; it would take 16384 * 2 hours = nearly 4 years!!
Maybe one convert, read the file once, then make each individual crop, as I outlined above.
If you use clones, you will need to have two copies in memory, the original and the clone. With the MPC, you only have to read it, which should be fast.
I am undecided which approach might be best.
Perhaps a test on a large image (but not as large as you have now) would identify which approach is faster.
Re: Imagemagick deepzoom - tiling and resizing huge pictures
Posted: 2013-12-17T21:23:54-07:00
I haven't yet tried cropping from mpc. Experiments are ongoing.
I have successfully created a large png. See viewtopic.php?f=3&t=24653
Making the 16384 crops from the tiff, it has created 1697 temporary files in 13 hours 7 minutes, which is 2.16 per minute. Looking at the timestamps they are created either at 1 per minute or in clusters of 6 per minute. I don't know what causes this variation, or if it might help in the quest for a fast solution.
16384 at 2.16/minute is 126 hours, which is 5.27 days. It will also need time to create the cropped files.
I have successfully created a large png. See viewtopic.php?f=3&t=24653
Making the 16384 crops from the tiff, it has created 1697 temporary files in 13 hours 7 minutes, which is 2.16 per minute. Looking at the timestamps they are created either at 1 per minute or in clusters of 6 per minute. I don't know what causes this variation, or if it might help in the quest for a fast solution.
16384 at 2.16/minute is 126 hours, which is 5.27 days. It will also need time to create the cropped files.
Re: Imagemagick deepzoom - tiling and resizing huge pictures
Posted: 2013-12-17T23:25:48-07:00
cropping the mpc is increadibly fast for me, with verbose and monitor it takes around 6 seconds, without them half a second -->
Code: Select all
$ date +%H:%M:%S:%N
07:24:38:563228784
$ convert wall1-front-merged.mpc -crop 871x871+1742+1742 +repage w1t_2_2.jpg
$ date +%H:%M:%S:%N
07:24:38:986028290Re: Imagemagick deepzoom - tiling and resizing huge pictures
Posted: 2013-12-17T23:29:37-07:00
Did you try adding -limit memory 32 -limit map 32 to see if that made it even faster?
Re: Imagemagick deepzoom - tiling and resizing huge pictures
Posted: 2013-12-17T23:31:27-07:00
I did now, it seems it does! thanks for the hint!fmw42 wrote:Did you try adding -limit memory 32 -limit map 32 to see if that made it even faster?
Code: Select all
$ date +%H:%M:%S:%N
07:30:47:174675193
$ convert -limit memory 32 -limit map 32 wall1-front-merged.mpc -crop 871x871+1742+1742 +repage w1t_2_2.jpg
$ date +%H:%M:%S:%N
07:30:47:230184068Re: Imagemagick deepzoom - tiling and resizing huge pictures
Posted: 2013-12-17T23:54:57-07:00
hmmm, what's wrong here? I can't finde the mistake - the first line works, the second one fails
update: lol, found it, the space before "+repage" in the first line is the default hex code 20 it should be, and in the second line its hex "C2 A0" and I don't know why gedit (my texteditor) will show it as space. I've produced the second line by copying it out of libreoffice calc and removing the tabulators by search and (empty) replace.
update: doing all the crops at the same time and each just once (no cheating by caching because of repeatedly doing the same crop) is a bit slower.
here's the output of a little monitor loop I've written - first it prints the time second the count of tiles already created (and it only prints a line if the count changed)
Code: Select all
$ convert -limit memory 32 -limit map 32 wall1-front-merged.mpc -crop 871x871+1742+1742 +repage w1t_2_2.jpg
$ convert -limit memory 32 -limit map 32 wall1-front-merged.mpc -crop 871x871+85358+41808 +repage w1t_98_48.jpg
convert.im6: invalid argument for option `-crop': 871x871+85358+41808 +repage @ error/convert.c/ConvertImageCommand/1093.update: doing all the crops at the same time and each just once (no cheating by caching because of repeatedly doing the same crop) is a bit slower.
here's the output of a little monitor loop I've written - first it prints the time second the count of tiles already created (and it only prints a line if the count changed)
Code: Select all
08:22:53:283630889 - 2960
08:22:56:623716803 - 2961
08:22:56:723530474 - 2962
08:22:56:806320680 - 2963
08:22:56:888894938 - 2964
08:22:56:940312370 - 2965
08:22:57:093029277 - 2967
08:22:57:135605990 - 2969
08:22:57:180598653 - 2970
08:22:57:316549390 - 2971
08:22:57:379145968 - 2972
08:22:57:416258997 - 2973
08:22:57:486911767 - 2974
08:22:59:994201486 - 2975
08:23:00:084575967 - 2976
08:23:00:198056599 - 2977
08:23:00:297023653 - 2978
08:23:00:355776166 - 2979
08:23:00:431515756 - 2980
08:23:00:505943377 - 2981
08:23:00:576497461 - 2982
08:23:00:660770677 - 2983
08:23:00:727945071 - 2985
08:23:00:762333034 - 2986
08:23:00:814077603 - 2987
08:23:00:940853012 - 2988
08:23:01:059324921 - 2989
08:23:01:211528075 - 2990
08:23:01:274242684 - 2991
08:23:01:307822862 - 2992
08:23:01:331004720 - 2993Re: Imagemagick deepzoom - tiling and resizing huge pictures
Posted: 2013-12-18T01:33:09-07:00
Good stuff. So the trick is to crop from mpc? Excellent.
(My poor old laptop is still struggling to create the mpc! Ha!)
(My poor old laptop is still struggling to create the mpc! Ha!)
Re: Imagemagick deepzoom - tiling and resizing huge pictures
Posted: 2013-12-18T01:34:38-07:00
I did it over night, I guess it took around 6 hours. (on my old laptop, my new one is still working on my first try (see first post)snibgo wrote:Good stuff. So the trick is to crop from mpc? Excellent.
(My poor old laptop is still struggling to create the mpc! Ha!)
I think I'm gonna cancel that one now
Re: Imagemagick deepzoom - tiling and resizing huge pictures
Posted: 2013-12-18T04:04:03-07:00
okey, so the route to tiling the original image is clear, now I need to make it a pyramid - so tiles that are downscaled so that one tile will represent 4 tiles of the layer above it
So the quality wise best route would surly be to resize the orignal image like that:
and then do the tiling the same way we descussed above.
But I'm wondering - If I would not mind the quality loss from multiple lossy jpeg encodings - would it be quicker to join 4 tiles of the first layer and resize the result?
So the quality wise best route would surly be to resize the orignal image like that:
Code: Select all
convert -verbose -monitor -limit memory 2G -limit map 4G wall2-right_merged.mpc -resize 55697x55697 wall2-right_merged_55697px.mpcBut I'm wondering - If I would not mind the quality loss from multiple lossy jpeg encodings - would it be quicker to join 4 tiles of the first layer and resize the result?