Performance optimization tips

[neurosys]

Hello,

My requirement is perspective rotating a user submitted image into a pre-rendered background image (which is actually required per frame on a video).

The easiest was to use ImageMagick and I wrote a very crude simple bash script to achieve what I needed as follows:

Code: Select all

#!/bin/bash
# @author: neurosys
# @Description: Perspective transforms and projects an alpha image
# onto a background image.

if [ $# -ne 11 ]
then
    echo 'Usage: ./map_image.sh background.jpg image.png output.jpg x1 y1 x2 y2 x3 y3 x4 y4';
    exit;
fi

BG=$1
IMAGE=$2
DEST=$3
TEMP='temp.png'

BG_SIZE_W=$(convert $BG -print "%w\n" /dev/null)
BG_SIZE_H=$(convert $BG -print "%h\n" /dev/null)

IMAGE_W=$(convert $IMAGE -print "%w\n" /dev/null)
IMAGE_H=$(convert $IMAGE -print "%h\n" /dev/null)

X1=$4
Y1=$5

X2=$6
Y2=$7

X3=$8
Y3=$9

X4=${10}
Y4=${11}

OFFSET=15

TRANSFORM="$OFFSET,$OFFSET, $X1,$Y1  $(($IMAGE_W+$OFFSET)),$OFFSET $X2,$Y2  $OFFSET, $(($IMAGE_H+$OFFSET)) $X3,$Y3   $(($IMAGE_W+$OFFSET)), $(($IMAGE_H+$OFFSET)) $X4,$Y4"

echo "Transform matrix: $TRANSFORM"

convert $IMAGE -background transparent -extent $BG_SIZE_W\x$BG_SIZE_H-$OFFSET-$OFFSET $TEMP
convert $TEMP -background transparent -distort Perspective "$TRANSFORM" $TEMP
convert $BG $TEMP -composite $DEST
rm -f $TEMP

However it takes for about 4 seconds to produce the desired image on my computer as follows:

Code: Select all

[neuro@neuro-linux ~]$ time ./map_image.sh bg.png Hp-lovecraft.jpg output.jpg 494 108 579 120 494 183 576 196 && nomacs output.jpg
Transform matrix: 15,15, 494,108  195,15 579,120  15, 267 494,183   195, 267 576,196

real	0m3.852s
user	0m3.437s
sys	0m0.037s
[neuro@neuro-linux ~]$

The images used for the above example are,

• Background:
  
• User image:
  
• Output:
  

I'm wondering if there's a way to speed up this so much so that I can generate frames for a one minute video (25 fps * 60 sec) under a few seconds? I know this sounds a bit far fetched, but I might resort to writing an opengl program for this specifically, which I believe will be much faster given hardware leveraging.

Besides, the order of operations as well as the parameters I use in the above ImageMagick script might not be optimal. So any opinions or alternatives to achieve what I need are greatly welcome.

Thanks.

[snibgo]

You call that script once per frame, with the same BG and IMAGE in each frame, is that right?

For every frame, you find the dimensions of both images, using four convert commands and reading all pixels of both images twice. You can do this with only one identify per image (reading no pixels at all), and obviously outside the loop.

Then you have three converts that could be combined into one, saving two writes/reads of $TEMP.

But the first convert seems to be the same for all frames, so move that outside.

So the only remaining IM work per frame is:

Code: Select all

convert $TEMP -background transparent -distort Perspective "$TRANSFORM" \
 $BG -compose DstOver -composite $DEST

EDIT to add:

The result of the "-distort perspective" is the size IMAGE, but the distorted portrait occupies only about 1% of the output. So about 99% of that processing is wasted.

[fmw42]

convert $IMAGE -background transparent -extent $BG_SIZE_W\x$BG_SIZE_H-$OFFSET-$OFFSET $TEMP

Does this command extend the small image to the size of the large image with transparent background? If so, you do not need to do that. You should be processing the small image directly and using +distort perspective -layers merge +repage.

I had assumed that both images were changing in your script each time. So I do not see any speed up there.

Snibgo is right, in that you need to combine all your commands into one command line to avoid the time to read and write from one command to another.

[neurosys]

Thanks for input. I'll look into them now. In the meantime to clarify things:

You call that script once per frame, with the same BG and IMAGE in each frame, is that right?

No. BG is actually a frame of the video, so it changes each time. IMAGE doesn't change, but control points keep changing for every frame.

Does this command extend the small image to the size of the large image with transparent background? If so, you do not need to do that. You should be processing the small image directly and using +distort perspective -layers merge +repage.

Right, I understand that I don't need the first step. And it causes unnecessary distort execution later. So I'll remove it.

But it is distort perspective which takes the most of time.

For a one minute video it adds up to 60*24 = 1440 times this script needs to be executed. Also, I should note that merging part will be done in ffmpeg to produce a video with perspective distorted image overlayed on top.

[snibgo]

Yes, about 99% of the distort perspective processing is wasted, because most of its output is transparent.

[neurosys]

I'm still reading the documentation and trying to figure how I would achieve increasing canvas size while doing distort perspective with the following command:

Code: Select all

convert image.jpg -background transparent -distort Perspective "1,1, 494,108  201,1 579,120  1, 201 494,183   201, 201 576,196" output.png

Control points lie outside the source image's dimensions, therefore output image needs to be as large as to contain distorted image (which with the above command is not).

How could I do that?

[snibgo]

Use "+distort", with a plus "+".

[neurosys]

Thanks, it reduced to about 0.6 seconds. Bulk of it is likely reading/writing the image.

Code: Select all

convert image.png -background transparent +distort Perspective "1,1, 494,108  201,1 579,120  1, 201 494,183   201, 201 576,196" -compose DstOver -composite bg.png output.png

Except transparency doesn't work for some reason, and output image is all black around the distorted image.

[fmw42]

.

The perspective transform below takes me 0.13 seconds on my IM 6.9.3.8 Q16 Mac OSX running two cores. Only one command line for doing the image processing and I use +distort ... -layers merge +repage.

Code: Select all

tl_bg="494,108"
tr_bg="579,118"
br_bg="579,197"
bl_bg="494,181"
dim=`convert -ping image.jpg -format "%wx%h" info:`
ww=`echo $dim | cut -dx -f1`
hh=`echo $dim | cut -dx -f2`
right=$((ww-1))
bottom=$((hh-1))
echo "ww=$ww; hh=$hh; right=$right; bottom=$bottom;"
tl_im="0,0"
tr_im="$right,0"
br_im="$right,$bottom"
bl_im="0,$bottom"
time convert bg.png \
\( image.jpg -virtual-pixel transparent +distort perspective \
"$tl_im $tl_bg  $tr_im $tr_bg  $br_im $br_bg  $bl_im $bl_bg" \) \
-layers merge +repage bg_image_result.jpg

[snibgo]

convert image.png -background transparent +distort Perspective "1,1, 494,108 201,1 579,120 1, 201 494,183 201, 201 576,196" -compose DstOver -composite bg.png output.png

The syntax is bad. The input bg.png should come before "-composite". I would put it before "-compose".

[neurosys]

Thank you, this is exactly what I needed.

.

The perspective transform below takes me 0.13 seconds on my IM 6.9.3.8 Q16 Mac OSX running two cores. Only one command line for doing the image processing and I use +distort ... -layers merge +repage.

Code: Select all

tl_bg="494,108"
tr_bg="579,118"
br_bg="579,197"
bl_bg="494,181"
dim=`convert -ping image.jpg -format "%wx%h" info:`
ww=`echo $dim | cut -dx -f1`
hh=`echo $dim | cut -dx -f2`
right=$((ww-1))
bottom=$((hh-1))
echo "ww=$ww; hh=$hh; right=$right; bottom=$bottom;"
tl_im="0,0"
tr_im="$right,0"
br_im="$right,$bottom"
bl_im="0,$bottom"
time convert bg.png \
\( image.jpg -virtual-pixel transparent +distort perspective \
"$tl_im $tl_bg  $tr_im $tr_bg  $br_im $br_bg  $bl_im $bl_bg" \) \
-layers merge +repage bg_image_result.jpg

[fmw42]

You can remove the line:

echo "ww=$ww; hh=$hh; right=$right; bottom=$bottom;"

I forgot to take it out after debugging.

And also you can remove the "time" from the convert command, since it just reports the processing time.