Resizing .mpc get very slow halfway.

[mrudelle]

Hi everyone,

I'm not sure this is the right section but I am facing a very weird issue with ImageMagick.

I am working on a huge image (1.7GB ~ 53,000x18,000px) for a virtual texturing project. I first convert my image to mpc with the following command

Code: Select all

convert -monitor -debug cache -limit area 8192 -limit memory 8192 "$src" "$mpc"

Then I reduce the image by half it's size for every stage of the process with the following command

Code: Select all

convert -monitor -debug cache -limit area 8192 -limit memory 8192 "$mpc" -resize 50% "$mpc"

and my problem is that the latter handles "quickly" 73% of the resize and then it starts to be very very slow ! so I added -debug cache to understand what happen and the problem seem to happen just after the cache is extended but I can't find what it is ...

Here is the interesting part of the log:

2015-05-19T22:33:07+02:00 3:32.350 95.090u 6.7.7 Cache convert.im6[24764]: cache.c/WritePixelCachePixels/5673/Cache, 72% complete
/media/matt/18A2AFBFA2AFA02C/cropping/AfricanWoman/temp_tiler/temp_ptm_tiling_job.mpc[0][18718x1+0+25970]
2015-05-19T22:33:09+02:00 3:34.320 96.020u 6.7.7 Cache convert.im6[24764]: cache.c/WritePixelCachePixels/5673/Cache, 72% complete
/media/matt/18A2AFBFA2AFA02C/cropping/AfricanWoman/temp_tiler/temp_ptm_tiling_job.mpc[0][18718x1+0+26235]
2015-05-19T22:33:10+02:00 3:34.730 96.250u 6.7.7 Cache convert.im6[24764]: cache.c/ReadPixelCachePixels/4797/Cache4, 73% complete
/media/matt/18A2AFBFA2AFA02C/cropping/AfricanWoman/temp_tiler/temp_ptm_tiling_job.mpc[0][18718x8+0+52569]
2015-05-19T22:33:12+02:00 3:36.570 97.080u 6.7.7 Cache convert.im6[24764]: cache.c/WritePixelCachePixels/5673/Cache, 73% complete
/media/matt/18A2AFBFA2AFA02C/cropping/AfricanWoman/temp_tiler/temp_ptm_tiling_job.mpc[0][18718x1+0+26500]
2015-05-19T22:33:12+02:00 3:37.180 97.410u 6.7.7 Cache convert.im6[24764]: cache.c/WritePixelCachePixels/5673/Cache, 73% complete
/media/matt/18A2AFBFA2AFA02C/cropping/AfricanWoman/temp_tiler/temp_ptm_tiling_job.mpc[0][18718x1+0+26584]
2015-05-19T22:33:12+02:00 3:37.190 97.410u 6.7.7 Cache convert.im6[24764]: cache.c/ExtendCache/3951/Cache4 of 35944, 73% complete
extend /media/matt/18A2AFBFA2AFA02C/cropping/AfricanWoman/temp_tiler/temp_ptm_tiling_job.mpc[0] (/media/matt/18A2AFBFA2AFA02C/tmp_magick/magick-3T91f0RF[5], disk, 1.9905GB)
2015-05-19T22:33:12+02:00 3:37.190 97.410u 6.7.7 Cache convert.im6[24764]: cache.c/OpenPixelCache/4149/Cache
open /media/matt/18A2AFBFA2AFA02C/cropping/AfricanWoman/temp_tiler/temp_ptm_tiling_job.mpc[0] (/media/matt/18A2AFBFA2AFA02C/tmp_magick/magick-3T91f0RF[-1], memory-mapped, 9359x26585 1.8538GiB)
2015-05-19T22:33:13+02:00 3:37.850 97.560u 6.7.7 Cache convert.im6[24764]: cache.c/ReadPixelCachePixels/4797/Cache
/media/matt/18A2AFBFA2AFA02C/cropping/AfricanWoman/temp_tiler/temp_ptm_tiling_job.mpc[0][5x26585+0+0]
2015-05-19T22:36:14+02:00 6:39.420 97.840u 6.7.7 Cache convert.im6[24764]: cache.c/WritePixelCachePixels/5673/Cache
/media/matt/18A2AFBFA2AFA02C/cropping/AfricanWoman/temp_tiler/temp_ptm_tiling_job.mpc[0][1x26585+0+0]
2015-05-19T22:36:14+02:00 6:39.500 97.890u 6.7.7 Cache convert.im6[24764]: cache.c/ReadPixelCachePixels/4797/Cache4, 73% complete
/media/matt/18A2AFBFA2AFA02C/cropping/AfricanWoman/temp_tiler/temp_ptm_tiling_job.mpc[0][7x26585+0+0]
2015-05-19T22:36:19+02:00 6:43.710 98.000u 6.7.7 Cache convert.im6[24764]: cache.c/WritePixelCachePixels/5673/Cache
/media/matt/18A2AFBFA2AFA02C/cropping/AfricanWoman/temp_tiler/temp_ptm_tiling_job.mpc[0][1x26585+1+0]
2015-05-19T22:36:19+02:00 6:43.830 98.050u 6.7.7 Cache convert.im6[24764]: cache.c/ReadPixelCachePixels/4797/Cache4, 73% complete
/media/matt/18A2AFBFA2AFA02C/cropping/AfricanWoman/temp_tiler/temp_ptm_tiling_job.mpc[0][8x26585+1+0]
2015-05-19T22:36:21+02:00 6:45.870 98.180u 6.7.7 Cache convert.im6[24764]: cache.c/WritePixelCachePixels/5673/Cache
/media/matt/18A2AFBFA2AFA02C/cropping/AfricanWoman/temp_tiler/temp_ptm_tiling_job.mpc[0][1x26585+2+0]

Do you have any idea what can cause that ? I'm really stuck on this.

[magick]

You image exceeds your computer's memory resources so the pixel cache is created on disk and subsequently memory mapped. Its likely the slow down becomes apparent when the pixel cache is created on disk @ nearly 2GB. The pixels are then read from disk. Even with memory mapping, reading from disk is about 200 times slower than reading from memory.

[mrudelle]

Actually there is almost nothing in memory, because I use '-limit area 8192 -limit memory 8192' and we can see the program is alternatively reading and writing to cache (I think it's piped) so it can't be the slow access to disk because the first part is already reading and writing to disk (plus everything happen on my SSD)
Btw, when I say fast, it means the 70% are done in 10mins and the rest is likely to take 2 to 3 days at that speed (we can see only 26k pixels were processed in 6seconds

[magick]

Try just limiting the memory, not the area. With -limit area, the pixel cache is entirely read / written from disk. We tried your command on a 53000x18000 MPC image and

• convert -monitor -limit memory 8192 "image.mpc" -resize 50% "resize.mpc"

completed in 151 seconds.

You can also replace -resize with -scale. -resize iterates over the image twice, once horizontally and the other vertically. Scanning vertically on disk is quite inefficient and we suspect that is the source of your delay. With -scale, the command above ran in 21 seconds.

Making sure your temp area is on a fast disk would also be helpful. If you converting across NFS or a USB connected disk, that would be quite inefficient.

[mrudelle]

Thank you very much for your help here!

By switching to -scale I was able to avoid the problem (though just removing the -limit area did not have any effect, it must be a problem from my SSD or anything else). My script just finished running after 2 hours !

Indeed, if -resize scans vertically the image it might be the reason why it's so slow, but it's weird that it runs in 2.5 minutes on your setup.

Best