cropping very very large images

[abeazam]

i am passing in a massive 40000x40000 tif

it manges to load in the image into memory.

but it seems to crop each image and keep it in memory till the end of the process. which is no good cause in writes like 50gig to my swap disk then kills my computer.

ideally i would need to modify this so it writes out each tile to disk one at a time it might be a bit slow but thats cool i just need the process done

Code: Select all

#!/bin/bash
file=$1
function tile() {
convert -monitor -limit memory 2GiB -limit map 2GiB -limit area 2GB $file -scale ${s}%x -crop 256x256 \
-set filename:tile "%[fx:page.x/256]_%[fx:page.y/256]" \
+repage +adjoin "${file%.*}_${s}_%[filename:tile].png"
}
s=100
tile
s=50
tile

[anthony]

Using tile crop will generate a new image for each and every tile! Of course each will only be sized to fit that tiles data, but then you also make have a lotoof extra meta data associsated too.
Than means you at least double the memory usage when you do a tile crop.

My idea would be to crop one 'row' at a time from the large image using "stream" pipeing that into convert to
crop those into tiles.

For other techniques see Really Massive Image Handling
http://www.imagemagick.org/Usage/files/#massive

[abeazam]

Using tile crop will generate a new image for each and every tile! Of course each will only be sized to fit that tiles data, but then you also make have a lotoof extra meta data associsated too.
Than means you at least double the memory usage when you do a tile crop.

My idea would be to crop one 'row' at a time from the large image using "stream" pipeing that into convert to
crop those into tiles.

For other techniques see Really Massive Image Handling
http://www.imagemagick.org/Usage/files/#massive

i looked over the link and the help page was not great but the forum link that was there was broken. I searched the title of the post mentioned and found it. viewtopic.php?f=1&t=18315&p=70157&hilit ... es.#p70157
Must have been a really old link.

there was a script there i cleaned it up a bit, which had 2 problems.

The first was that it loads the large image into memory for every tile. This would take my script over 130days to complet lol.

The second was it error on writing the singe file it had to write.

Code: Select all

#!/bin/bash
src=$1
z=100
width=`identify -format %w $src`
limit=$[$width / 256]
echo "count = $limit * $limit = "$((limit * limit))" tiles"
limit=$((limit-1))
#~ k=1
for x in `seq 0 $limit`; do
  for y in `seq 0 $limit`; do
    tile=tiles/tile-$z-$x-$y.png
    echo -n $tile
    w=$((x * 256))
    h=$((y * 256))
    convert -debug cache -monitor $src -crop 256x256+$w+$h $tile
  done
done

What would help me is any one of three things

Tweak the above code as to keek the large image in memory.

or

get the code original post to not write so much to the swap disk and use so much ram

or

some other way to get the image cut up.

thx

[magick]

Convert your source image to MPC once then tile, something like

• convert -limit area 2mb srcimage srcimage.mpc
  for each tile
  convert srcimage.mpc -crop axb+c+d tile.png

[anthony]

However like I said above, cropping rows, then tile cropping the columns from that row, will also speed up the process.

[abeazam]

Convert your source image to MPC once then tile, something like

• convert -limit area 2mb srcimage srcimage.mpc
  for each tile
  convert srcimage.mpc -crop axb+c+d tile.png

This might sound like a silly question but is "for each tile" part of the script?

[abeazam]

THX magick .mpc image format sorted it.

Since the .mpc format is native image format used by imagemagick it does not need to convert the initial image it just cuts out the piece that it needs. This is the case with the second script I setup. See below how I got it working.

Lats say you have a 50000x50000 tif called myLargeImg.tif

First you take your large image and convert it to the native image format

Code: Select all

convert -monitor -limit area 2mb myLargeImg.tif myLargeImg.mpc

After the image is converted to mpc run the bellow bash script that will create the tiles.

Create a file named tiler.sh in the same folder as the mpc image and put the below script.

Code: Select all

#!/bin/bash
src=$1
width=`identify -format %w $src`
limit=$[$width / 256]
echo "count = $limit * $limit = "$((limit * limit))" tiles"
limit=$((limit-1))
for x in `seq 0 $limit`; do
  for y in `seq 0 $limit`; do
    tile=tile-$x-$y.png
    echo -n $tile
    w=$((x * 256))
    h=$((y * 256))
    convert -debug cache -monitor $src -crop 256x256+$w+$h $tile
  done
done

In your consol/terminal run the below command and watch the tiles apear one at at time into your folder.

Code: Select all

sh ./tiler.sh myLargeImg.mpc

[anthony]

Seems to me people are in need of a proper 'stream' oriented programs to break images up into tiles.
That is one that only uses pipelines, and memory (not disk, or swap).

Such streaming programs should only need to buffer one 'row' of tiles in memory, or even just one row of pixels but using multiple buffered pipelines, with one file handle per column to process each tile.

NOTE: a lot of emphasis I have seen is of breaking up large images into tiles, and not visa-versa. That is montaging tiles back together into a very large image.

The most promising has been PbmPLus, but its manuals are far from clear about its 'stream' processing handling for large images, and seems to me to fall short of the task.

[abeazam]

I agree and I dont think it would be that big of a change. If you keep the initial image in memory. Then save out each tile to file rather than storing it memory it would resolve a lot of the problems.

This seems like it will facilitate at least the taking apart of images.

[anthony]

Something that will be much easier to do with IMv7, which can run a IM command in background (holding the image), while a shell feeds it 'options' to be applied immediatally. The shell can do the loop, and get IM to process the image in exactly that way.

This would be a good example of IMv7 processing, and actually should be posible with the 'alpha' development version now!