Destroy Images in MSL

[ivan.savu]

I've been using conjure and MSL to combine bunch of images into one single large image with something like this:

Code: Select all

<group>
  <image id="0">
    <read filename="im1.jpg" />
  </image>
  <image id="1">
    <read filename="im2.jpg" />
  </image>
  <image id="2">
    <read filename="im3.jpg" />
  </image>
  <image id="3">
    <read filename="im4.jpg" />
  </image>
  <image size="12448x12448">
    <read filename="xc:" />
    <composite geometry="+0+0" image="0" />
    <composite geometry="+0+256" image="1" />
    <composite geometry="+0+512" image="2" />
    <composite geometry="+0+768" image="3" />
    <write filename="C:/out.png" />
  </image>
</group>

Now, when there's very large amount of images, conjure, first loads all these small images in memory, thus spending lots of ram thats gonna be needed for the last large image thats going to hold them all. Since every small image used only once, I'm wondering is there a way for me to load them one by one, only for drawing and then dispose them ?

Thank you

[magick]

Images are destroyed when they go out of scope-- in this case when the </group> tag is encountered. There is no way to explicitly destroy them, if you did bad things could happen in the script. If memory is an issue you can force the image pixels to disk with the -limit option, for example:

• conjure -limit area 2mb ...

FYI, if there is a tag you need that is not currently supported by MSL, let us know. MSL enhancements are on a on-demand basis.

[ivan.savu]

Memory itself is not a problem, since IM starts pushing stuff onto disk automatically as it runs out of memory, the main problem is performance.. Once IM starts pushing stuff to disk it becomes painfully slow.. Consider the following list of number of images process/ time taken to process

2 16s
512 35s
1024 1m
1500 1m 39s
2000 8m 36s

till 2000, time increases linearly, but with 2000 images, the target image ends on disk
(because its pushed out by a bunch of small images that are not being used),
making the process painfully slow.. btw conjure won't accept -limit argument

[magick]

We will need to think about ways to make your workflow memory efficient. We currently do not have an ETA on a solution.

[ivan.savu]

Ok, I found a workaround, I created multiple XMLs, each processing about 1000 images and applying the result to the last batch, it's a bit ugly and loses some time reading and writing big image, but it's faster then the one big batch version. This was a low priority project for me, and this solution is acceptable, so as far as I'm concerned, you don't have to fix it .

[anthony]

NOTE I have not really used conjure before. But...

It seems to me the big problem is that conjure provides no simple way to destroy an image you are finished using. If old images could be destroyed then you would have no problems.

This may have an easy solution, perhaps using scopes, whcih looks to be the way conjure works. So the question becomes can you have one <group> enbedded in another <group>?

Also what happens if you try to read a new image over an old image?

If the old image is replaced, then you have a solution. Just read each input image in composite it onto the canvas, then read the next image in over the top of the previous image.

Please let us (here on the forum) know how it goes, and watch the processes (and system) memory usage.

[ivan.savu]

Actually that's great idea, gonna give it a try immediately. I'd also have to note that MSL has a rather sparse documentation (when compared to the rest of the ImageMagick)

[anthony]

It has mostly been replaced by the exhancements to the "convert" command that came about in ImageMagick v6. That is its 'do it in option order' and the parenthesis, clone, and other options that were added.

See IM examples, Basics
http://www.imagemagick.org/Usage/basics/

and especially the section of parenthesis.


The big difference however between "convert' and "conjure" methods is that "convert tends to over destroy images, unless images are saved into MPC: (which has no destroy function). And does not have multiple image sequences. As a consequences 'cloning' into a parenthesis to do operations is much more common technique for complex processing tasks.

See Examples in
Combining and Debugging Complex Image Operations
http://www.imagemagick.org/Usage/basics/#seq_combine

[ivan.savu]

I have first tried using convert, but since convert gets input as command line arguments, I had trouble calling convert.exe with such a large number of arguments.
The suggested method of loading all images into the same canvas didn't work, because MSL doesn't support placing image tag into another image tag.

when I do something like this

Code: Select all

<image size="12448x12448">
  <read filename="xc:" />
  <image id="the_image">
	<read filename="im1.jpg" />
  </image>  
  <composite geometry="+0+0" image="the_image" />

  <image id="the_image">
	<read filename="im2.jpg" />
  </image>  
  <composite geometry="+256+0" image="the_image" />
    
  <write filename="out.png" />
</image>

The resulting image is same as the last image I tried to compose.
I suppose that's becuase <read> tag reads into the top image (the main image) instead into the_image canvas

[anthony]

I have first tried using convert, but since convert gets input as command line arguments, I had trouble calling convert.exe with such a large number of arguments.

Okay command line limit. Something that will be resolved for version 7 of IM, with the ability to read options from a file or pipeline. It could even allow for 'daemon' "convert" processes!

The suggested method of loading all images into the same canvas didn't work, because MSL doesn't support placing image tag into another image tag.

The resulting image is same as the last image I tried to compose.
I suppose that's becuase <read> tag reads into the top image (the main image) instead into the_image canvas

That is a shame.

I see the same effect with MPR named registers...
convert \( cyclops.gif -write mpr:this_image +delete \) \
\( dragon.gif -write mpr:this_image +delete \) \
mpr:this_image show:

I use to get the cyclops image, (the dragon was not stored) but now I see the dragon image.
What version of IM are you using?


Hmmm adjusting your test script I can't even see the compositions in the output..

Code: Select all

<image size="512x512">
  <read filename="xc:none" />
  <image id="the_image">
   <read filename="im1.jpg" />
  </image> 
  <composite geometry="+0+0" image="the_image" />

  <image id="the_image">
   <read filename="im2.jpg" />
  </image> 
  <composite geometry="+256+0" image="the_image" />
   
  <write filename="show:" />
</image>

But reordering it back to more like the original I see the compositions...

Code: Select all

<group>
  <image id="i1">
   <read filename="im1.jpg" />
  </image>
  <image id="i2">
   <read filename="im2.jpg" />
  </image>
  <image size="512x512">
    <read filename="xc:none" />
    <composite geometry="+0+0" image="i1" />
    <composite geometry="+256+0" image="i2" />
    <write filename="show:" />
  </image>
</group>

I have been playing with it but can not find any guide on making good use of it!

Also its error reporting sucks, with no indication on what line the error is on!

[anthony]

You also can't seem to have multiple <group> invocations. It either produced a syntax error, or a segmentation fault!

Also while you can read multiple images into a single image sequence, you can't seem to go back to a previous image sequence.

what we need is some better examples of MSL, and a guide to what it is actually doing!

[ivan.savu]

Okay command line limit. Something that will be resolved for version 7 of IM, with the ability to read options from a file or pipeline. It could even allow for 'daemon' "convert" processes!

That would be awesome

I'm using ImageMagick 6.5.0

[anthony]

In the mean time you can do the composition in batches.

Code: Select all

convert .....enough image processing, but short of command line limit... \
            temporary.mpc

then

Code: Select all

convert temporary.mpc  ... do more image processing... \
            temporary.mpc

repeating this as many times as needed, and finally

Code: Select all

convert temporary.mpc  ...do final image processing stuff...\
            result.png
rm temporary.mpc temporary.cache

The "MPC" images are save to disk but simply as a direct (paged) memory dump.
That is not formating during write. so it is quite fast.

When reading it is 'paged' directly into memory as needed, so while their is disk IO, no file parsing or conversion is needed. the data is just directly used. That make the MPC file format VERY fast in comparision to any other image file format.

WARNING: as it is a direct memory dump you can not use it as a permanent save format as the smallest change in commands (upgrade or different machine) will make it unusable and probably cause segmentation faults. But for a script it is perfectly safe.

Also note that two files are saved. A mapping information file (very small) and the memory dump itself (cache). Both need to be cleaned up when finished.

See IM Examples MPC
http://www.imagemagick.org/Usage/files/#mpc