Placing images by co-ordinates given in the file name?

[fmw42]

I defer to snibgo on the memory issue.

I think then the issue is converting his filename coordinates into pixels for the upper left corner of the tile. One may have to make a pass over the filenames to figure out how big the resulting image needs to be since he says there are many missing tiles. I also think one needs to compute some offset to his top left tile, since his coordinates are probably offset from his (0,0) point by a large amount. Perhaps this last is not needed if one uses -layers mosaic to put it all together. I think it keeps track of the virtual-canvas and so may crop to the real bounds of his data. I have not tested that to be sure.

I think I my earlier computation of the file size was in error by 8x.

529 tiles x 256px x 256px x 3 channels = 104 Mbytes, if I am not in error. But that assumes they all fit together. He says there are missing tiles. So it is not clear how big the real output would be. But even multiple times 104 Mbytes is still acceptable.

Correct me if I am wrong.

[snibgo]

If they all fitted together with no gaps, the output would need to be 529*256*256 = 34,668,544 pixels. At 8 bytes/pixel this is 277MB. (I believe IM always allows for at least 4 channels.) Doubled for in/out (I'm not sure if this is needed) = 554 MB. I assume Q16; Q8 needs half this memory. A Windows 7 computer probably isn't new, and might struggle with the memory requirements of my example above.

In my script above, sending the output to TIFF instead of PNG reduces the time from 21s to 12s.

Just for fun, I multiply linear dimensions by 4. My output is now 34000x28000 pixels (at 8 bytes/pixel, this is 7.6GB), and I write 9600 tiles. Making an MPC this size takes 3 minutes. Writing the 9600 tiles, with scale and rotation (-distort SRT would be faster?) and saving to a different MPC takes 11 min. The resulting .cache files are 7.6 GB each, which is no surprise.

(Yes, I did try saving to TIFF, and got an error:

Code: Select all

F:\web\im>c:\im\ImageMagick-6.8.9-5\convert m600.mpc @im_make600.txt m600.tiff
TIFFAppendToStrip: Maximum TIFF file size exceeded.
convert.exe: Maximum TIFF file size exceeded. `TIFFAppendToStrip' @ error/tiff.c/TIFFErrors/561.
convert.exe: Error flushing data before directory write. `TIFFWriteDirectorySec' @ error/tiff.c/TIFFErrors/561.

)

As I said, I don't understand the 6 given parameters for the files. Perhaps 2 of them must be multiplied by 3.2 to get the pixel coords of the top-left corner, taking y=0 as the bottom. Perhaps the tiles need no resizing or rotation. Or perhaps they do.

My script could be modified so, instead of looping through 600 integers, it would loop through a list of filenames. For each one, it would parse the data, do any calculations, and echo to file IM_SCR.

[fmw42]

Code: Select all

My script could be modified so, instead of looping through 600 integers, it would loop through a list of filenames. For each one, it would parse the data, do any calculations, and echo to file IM_SCR.

Yes, I think that is what is needed. But since tiles are missing, I think you need to use -layers mosaic to place them.

Thanks for correcting my calculations. I am not thinking well this late tonight.

[snibgo]

I don't know if it needs mosaic. I agree that tiles may need offsets, and that an initial pass may be needed to find the output size. If all the tiles have the same large offset, I'd ignore it, and "-repage" at the end.

Of course, the tiles may be very very sparse, giving a massive required output of 100,000x100,000 pixels (needing 80 GB) or more. I expect my revised script (with MPC) would still work, and it might even be the optimal method. Or a quicker method might be to build the output in smaller pieces, such that each piece fits in memory, and assemble the pieces together at the end. But some tiles might straddle two or more pieces, so processing becomes more awkward: loop through each piece and, for each one, loop though all the tiles to find the ones required for this piece.

Very large output files bring their own problems, of course. Software to read the images needs loads of memory.

[fmw42]

I was just thinking that by using -mosaic, the image need not be initialized to its full size at the beginning. -mosiac expands the image as needed by the page offsets used.

[Lumo]

The batch script certainly looks like it's on the right track There shouldn't need to be any scaling or rotation of the tiles.

Regarding memory, the files are indexed colour with a 256 colour palette.The machine running the job has 12Gig of RAM installed. The data tiles in the sample are the smallest set and I expect most of the tasks in the future would have at least ten times as many tiles. Therefore I think that the most memory-efficient process would be the most appropriate, even if it takes a long time.

[Bonzo]

I have been working on a similar problem and the composite takes a relatively long time compared to my other processes. Is there a more efficient way using layers or page?
I read the note about working in rows but my image is only 3 rows of 4 images.

This is my command:

Code: Select all

convert -size {$width}x{$height} xc:none  image-0.miff -geometry +15+4 -composite image-1.miff -geometry +215+13 -composite image-2.miff -geometry +417+6 -composite image-3.miff -geometry +605+3 -composite image-4.miff -geometry +13+188 -composite image-5.miff -geometry +204+187 -composite image-6.miff -geometry +404+191 -composite image-7.miff -geometry +611+187 -composite image-8.miff -geometry +13+361 -composite image-9.miff -geometry +215+359 -composite image-10.miff -geometry +413+364 -composite image-11.miff -geometry +616+367 -composite -trim combined.png

[fmw42]

You can avoid all the -composites if you use -page rather than -geometry and use -layers flatten. I cannot say which is faster.

If you have spaces between images, you can use -layers mosiac or -layers merge without the -size ... xc: and just -background and it will fill out the canvas as needed.

see
http://www.imagemagick.org/Usage/layers/#flatten
http://www.imagemagick.org/Usage/layers/#mosaic
http://www.imagemagick.org/Usage/layers/#merge

[Bonzo]

Thanks fmw42; I had a problem as convert was ignoring -page and stacking all the images one on top of the other. This was solved by moving the -page before the image.

Original code was 25 seconds; the -page and merge version was 10 seconds; flatten was about the same as merge. I should add this is on my home PC where threading is not working for some reason and should be faster on the server.

Code: Select all

convert  -page +14+8 image-0.miff -page +207+10 image-1.miff -page +410+11 image-2.miff -page +607+4 image-3.miff -page +11+182 image-4.miff -page +204+191 image-5.miff -page +414+185 image-6.miff -page +607+189 image-7.miff -page +9+361 image-8.miff -page +216+366 image-9.miff -page +409+362 image-10.miff -page +612+367 image-11.miff -background dodgerblue -layers merge +repage combined.png

[anthony]

Getting numbers from a filename is getting numbers from a string. IM can't currently do this.
I did plan to add some sub-string handling in percent escapes, but that was a very long way off.

Once you have the numbers from a filename, where do you put them!
You can store them into the file as file meta-data.

In fact that would have been the better place to put them if you had planned to later use them.

Images saved from IM in PNG or MIFF formats will save meta-data, whcih can then be extacted and used in later IM commands!
for example...

Code: Select all

  convert rose: -set my_X 35  -set my_Y 20 rose.png

now lets position the rose according to that offset on the granite image

Code: Select all

  convert rose.png -set page '+%[my_X]+%[my_Y]' granite: +swap -flatten show:

[Lumo]

Over on GIMPChat, Ofnuts kindly wrote me a script which works well. In fact it works perfectly.

The memory problem remains though, even with 12 Gig available.

Can anyone tell me how to get Snibgo's Batch script to work?

Thanks