Help efficiently build huge image from tiles

[ner0]

Hello,

I'm trying to build a really big image from 34.969 tiles, the bigger tiles are at 512x512 each, and the target image will be 95488x95488 pixels.
The reason why I need to build such a big image is because afterwards I need to resize it and then crop it into multiple tiles once again, because currently the tiles have varying widths and heights and I need to output a specific number of perfectly squared tiles. The only way I can think of to achieve this objective is to build the entire image first. My problem is that it is filling my HDD and failing because of it.

Here's the command I'm using:

Code: Select all

montage.exe @list_187x187.txt -mode Concatenate -tile 187x187 image.png

This process went on for 24 hours before filling 210GB of available HDD space and failing as a result.

I also tried building smaller chunks of the final image first, and so I did a montage of 187 images, using 187 horizontal tiles each, and finally the montage of 187 (95488x[H]) images into one.

Code: Select all

montage.exe @list_1x187.txt -mode Concatenate -tile 1x187 image.png

The result was pretty much the same with the difference that it reached the space limit in about 30 minutes as opposed to 24 hours.

Now I'm going to try using IM Q8, even though I wanted to use a portable version.
What I really wanted was for someone to comment on this and suggest a better approach.

Thank you!


SPECS:
OS: Windows 10 Pro (x64)
IM: ImageMagick-7.0.3-0-portable-Q16-x64
CPU: i7-4770K @ 3.5GHz
RAM: 16GB

[snibgo]

You might buy, beg, steal or borrow an external hard drive.

If the inputs are all 8 bits/channel/pixel, Q8 will do fine.

Perhaps you can do the job in pieces, eg build a quarter of the 95488x95488 image, then resize that and divide into the final tiles. Then do the next quarter.

Or do the job in horizontal strips, as you have tried, but don't join all the strips together. Just do a small batch, enough for the resizing and chopping into the finl tiles. Then do the nect batch.

[ner0]

You might buy, beg, steal or borrow an external hard drive.

If the inputs are all 8 bits/channel/pixel, Q8 will do fine.

Perhaps you can do the job in pieces, eg build a quarter of the 95488x95488 image, then resize that and divide into the final tiles. Then do the next quarter.

Or do the job in horizontal strips, as you have tried, but don't join all the strips together. Just do a small batch, enough for the resizing and chopping into the finl tiles. Then do the nect batch.

Thanks for the reply.

Q8 worked, it took 61 minutes to do the image.

But your idea of doing the work in pieces is much more appealing since, if I could just figure out the math, I could do without these big operations and just move to the final tiles by doing small batches. Unfortunately I'm not getting the math right for this...
I know that these 34.969 tiles will result in an image with 95488x95488, but the squared image needs to actually be 98304x98304 so that I can split it into 16.384 even tiles with 768x768. Now, if I want to process this in 4 batches... my head is melting...

If I do the full horizontals, I can resize each to the width needed, but then I'll be in trouble with the height, the same thing if I do verticals and resize them, later I'll me missing size horizontally.

[ner0]

Let's imagine this square image, for example:
428|256|296|512
256|428|512|296
296|512|256|428
512|296|428|256

This square is composed by 16 tiles, and I need them converted in a way so that I end up with only 2 tiles of 768.
As per the example above, the full image has a resolution of 1492. The way I'm doing this at the moment is to do a montage of the 8 tiles into a single image, then I rescale the resolution from 1492 to 1536, and then I crop the resulting image back into individual tiles of 768, this gives me 2 tiles.

This is a basic example with 16 tiles to explain the situation.
Assuming that I always know the original image resolution and the target resolution, how would I replace my approach so that I don't need to build the full image and rescale it as a whole? Would it even be possible?

I've given it some thought but don't seem to be able to make it work any other way.
I know that this isn't IM specific, but maybe someone had to do something along these lines at some point?

Thank you.

[snibgo]

You have a list of input images, and you know the width and height of each, and the required location of each. Correct?

If all the images were placed, the output would be 95488x95488, which you want to resize to 98304x98304, and chop into tiles.

You want output tiles 768x768. Because of resizing, this corresponds to input tiles of 746x746 exactly.

Assuming the list is ordered left-to-right, then top-to-bottom, read the list data, grabbing every file that has a y-coordinate in the range 0 to 745. This is the first row. If you have grabbed any pixels outside this range, trim it off. It has a size 95488x746. You can resize it to 98304x768, and chop it into 128 tiles that are 768x768.

Repeat for the next row, which will cover y=746 to 1491.

Keep going until you have done all the rows.

[ner0]

I'm going to try this something like that, but first, how did you extrapolate 746?

[snibgo]

Sorry, I should have said: 768 * 95488 / 98304 = 746.

EDIT: I should say, I modified my algorithm as I wrote it. It isn't necessary that the list of input files is ordered at all.

[ner0]

I'll try to do this again tomorrow.
I tried to apply this logic to a smaller resolution for testing and didn't go as expected.
144 tiles that build to an image of 5888x5888, needs to be 6144x6144 to end up with 64 tiles of 768x768.
768 * 5888 / 6144 = 736

What I am doing now is this:
1. Montage of rows 1 and 2: 5888x1024
2. Trim excess pixels with: -gravity South -chop 0x288 (result is now 5888x736)
3. Rescale image from 5888x736 to 6144x768
4. Crop image into 8 tiles of 768x768

5. Montage of rows 2 and 3: 5888x1024
6. Trim excess pixels with: -gravity North -chop 0x288 (result is now 5888x736)
7. Rescale image from 5888x736 to 6144x768
8. Crop image into 8 tiles of 768x768

I repeat this until rows 11 and 12, alternating the trim between North/South, but ended up with 88 tiles instead of 64.

Possibly I'm doing the trim wrong, among other things.
Also the last row is smaller than the others in terms of height, may also be throwing things off, I'll have to do more testing.

[ner0]

I have been trying for hours but can't seem to apply this idea.
I think I'm doing a different approach than the one you tried to explain, I have a very limited knowledge of IM.
Could you demonstrate a few command line examples to work this out in the way you suggested?

Btw, tiles' filenames are mapped into the image like this:
file_1_1|file_1_2|file_1_3|file_1_4
file_2_1|file_2_2|file_2_3|file_2_4
file_3_1|file_3_2|file_3_3|file_3_4
file_x_1|file_x_2|file_x_3|file_x_4

Thanks for the help.

[snibgo]

Could you demonstrate a few command line examples to work this out in the way you suggested?

I don't know what shell language you are using. I don't know where you get the data about the width, height and required position of each input tile.

If I was writing it, in "BAT" or C, there would be some initial calculations of the size of each row of tiles and so on. Then loop through each row, noting the first and last y-coordinate covered by the row. Within each row, loop through all the input tiles, looking at just the tile height and required y-offset. A simple calculation says whether that tile is required on this row.