miff mpr - what to use?

[geoland]

I have been reading up on miff mpr and mpc files in an attempt to make a fairly long and complicated bash script process more efficient. I'm not sure I'm on the right track.

Presently, the script reads, potentially, dozens of large tif files and converts them to miff. An operation is performed on the miff files and further operations on the resultant miff files. The final result is output as a tif file.

Using a single image example to simplify the process for illustration
First

Code: Select all

convert a.tif a.miff 
convert b.tif b.miff
convert c.tif c.miff

then

Code: Select all

convert a.miff b.miff -compose Minus_Src -composite d.miff 
convert d.miff c.miff -compose Divide_Src -composite final.tiff
rm -f *.miff

To avoid reading all images into memory; this, for example

Code: Select all

for i in `ls -v *.tif'; do any of the operations above; done

one image at a time.

Each image is processed once and then discarded. The resultant images are processed once and discarded. The final process combines the end result of the two processes, again discarding any temporary miff files.

Is it possible to use mpr in the above case; as follows

Read an image sequence - perform an operation on all the files and keep the result. Then, perform another operation on the result and keep the new result. Finally perform an operation on the new result and keep the final result... all without reading each result (miff file, as I am currently doing) into memory over and again with convert.

This seems to be the general idea behind mpr but I'm not sure its appropriate to the type of processing my script does. Fairly simple operations but lots of large files.

[snibgo]

convert a.miff b.miff -compose Minus_Src -composite d.miff
convert d.miff c.miff -compose Divide_Src -composite final.tiff
rm -f *.miff

In those commands, d.miff is a temporary result, used in the next command but never again. So why not combine them into one command?

Code: Select all

convert a.miff b.miff -compose Minus_Src -composite c.miff -compose Divide_Src -composite final.tiff
rm -f *.miff

[geoland]

Of course. Thanks. I have a bit more reading to do.

[muccigrosso]

Why convert to miff at all? Just compose the tiff files into mpr: (so they stay in memory and never get written out) and then only write the final product:

Code: Select all

convert a.tiff b.tiff -compose Minus_Src -composite mpr:c d.tiff -compose Divide_Src -composite final.tiff

(That might have to be divide_dst in the last step.)

[snibgo]

convert a.tiff b.tiff -compose Minus_Src -composite mpr:c d.tiff -compose Divide_Src -composite final.tiff

That won't work because "mpr:c" is a request to read mpr:c, but it hasn't been written. It could be "+write mpr:c". But there is no point in writing to mpr:c if it is never used.

When we want to improve performance of a sequence of operations, there are some obvious questions:

1. Are we doing more work than is necessary? Eg blurring a large image, then cropping it so most of the blurring is wasted effort.

2. Are we reading inputs more than once?

3. Are we writing temporary results to disk? If we really need to do this, are we using miff or mpc, or some slower format?

Reading and writing miff isn't much faster than tiff, but it is faster than png. So if the overall process needs to re-read a PNG image multiple times, it may be quicker to save that as miff or mpc the first time it is used, and then read from that miff/mpc in subsequent commands.

[muccigrosso]

convert a.tiff b.tiff -compose Minus_Src -composite mpr:c d.tiff -compose Divide_Src -composite final.tiff

That won't work because "mpr:c" is a request to read mpr:c, but it hasn't been written. It could be "+write mpr:c". But there is no point in writing to mpr:c if it is never used.

Good point! What I was trying to do is avoid writing anything along the way. Doesn't creating the miff result it getting written to the disk?

[snibgo]

Miff is a file format, so writing "abc.miff" writes to disk, so the image is available for subsequent commands.

Mpr is a memory location, so writing "mpr:abc" clones the image to memory, so it is available for subsequent operations within that command (by reading from "mpr:abc"), but not subsequent commands.

[muccigrosso]

Miff is a file format, so writing "abc.miff" writes to disk, so the image is available for subsequent commands.

Mpr is a memory location, so writing "mpr:abc" clones the image to memory, so it is available for subsequent operations within that command (by reading from "mpr:abc"), but not subsequent commands.

I had misread what you originally proposed, which is exactly what I was intending.

[geoland]

Thank you. Very informative to understanding how all this works.

It occurred to me, after starting this thread, that the processes involved are buried in conditional statements which provide the option of performing the task or not. It would require a major overhaul to write the script based on mpr, though some processes could be adapted. I will see what I can adapt and work from there. By then I should have a better understanding of how to apply the mpr concept in bash.

[snibgo]

"mpr:" is useful when we want to process the same image in multiple ways within a single magick or convert command. Internally it works in a similar way to "-clone". When we write a complex command we can often structure it using clones or mprs or both. The decision may be based on the logical structure and understandability of the command, or personal preference.

As a rule of thumb, when I write commands with three or more levels of parentheses, I realise it would be clearer if re-written with mpr.

[geoland]

Thanks. I was able to combine several commands as described in this thread. Largely, the do either / or nature of the work flow separates processes into individual commands, tied to user selectable options, the end result obtainable by several different pathways and processes.