How to stretch an image non-linearly in one dimension (à la Gopro Superview)

[RustyToad]

Hi all,

I'm trying to recreate the "Superview" effect of a go-pro camera in linux, from a camera that doesn't have this ability. A description of what I am going for is here, the critical bit is the image below:



The effect is created by recording in 4:3, then stretching out the edges of the image to fill out a 16:9 frame. The centre of the image is relatively unstretched.

I cannot find a way of distorting an video in this way in Linux, so intend to use ffmpeg to break the video into frames, then an imagemagick script to distort each frame, then re-build the video from the new frames.

However, I cannot find a distort filter which does what I want. I have tried using the polynomial and barrel distorts, but I can't work out how to get them, or any other distort method, to apply in this way - linear in the y axis, with a variable scale in the x axis.

Starting with 1440x1080, aiming for a 1920x1080, the closest I have got is the following, which simply stretches out the image uniformly. It feels like this should be close, but when I change the Order to 2, all hell breaks loose, and I'm not sure why:

Code: Select all

convert 4by3.jpg -define distort:viewport=1920x1080+0+0 -filter point -distort polynomial \
	'1.5 \
	0,0    0,0      576,0    816,0        720,0    960,0      864,0    1104,0      1440,0    1920,0  \
	0,1080 0,1080   576,1080 816, 1080    720,1080 960,1080   864,1080 1104,1080   1440,1080 1920,1080' \
	16by9.jpg

Any help or hints would be greatly appreciated.

[fmw42]

I am not sure what the issue is with the distort polynomial. Possibly because you are trying to do a 1D stretch with a 2D distort.

This may not be what you want, but here is another approach.

Input:

Code: Select all

convert image.png \
\( +clone -sparse-color barycentric "0,0 black  %[fx:w-1],0 white" -evaluate pow 2 \) \
-define compose:args=10,0 -compose displace -composite result.png

[snibgo]

I have GoPro 3 Black, not the 3+ Black with this weird "Superview". The only descriptions I can find online are too vague. I can't find any photos shot in both normal and superview.

There are some obvious ways to do this:

1. Crop the picture into three pieces, widen the left and right pieces, then append them together.

2. Resample so the centre column is unchanged, but there is progressive rescaling from there outwards, so the image widens by a factor of 1.333.

3. As (2) but many central columns are unchanged, and the progression starts further out so must be more severe.

[snibgo]

Source image, gpsv_g.png:


Method 1:

The sudden change in horizontal scale creates bends in diagonals.

Method 2:

The smooth change in horizontal scale makes curves in diagonals.

Method 3 is left as an exercise for the reader.

The Windows BAT code for IM v7.0.7-28, is:

Code: Select all

%IMG7%magick ^
  gpsv_g.png ^
  ( -clone 0 ^
    -gravity West -crop 25%%x100%%+0+0 +repage ^
    -resize "166.6666x100%%^!" ^
  ) ^
  ( -clone 0 ^
    -gravity Center -crop 50%%x100%%+0+0 +repage ^
  ) ^
  ( -clone 0 ^
    -gravity East -crop 25%%x100%%+0+0 +repage ^
    -resize "166.6666x100%%^!" ^
  ) ^
  -delete 0 ^
  +append +repage ^
  gpsv_a.jpg

set C=(4/3)
set A=(1-%C%)

%IMG7%magick ^
  gpsv_g.png ^
  -resize 133.333x100%% ^
  ( +clone ^
    -sparse-color Bilinear ^
0,0,#008,^
%%[fx:w-1],0,#f08,^
0,%%[fx:h-1],#0f8,^
%%[fx:w-1],%%[fx:h-1],#ff8 ^
    -channel Red ^
    -fx "XD=2*u-1;YD=%A%*XD*XD*XD+%C%*XD;(YD+1)/2" ^
    +channel ^
  ) ^
  -compose Distort -composite ^
  +repage ^
  gpsv_d.jpg

[RustyToad]

Thanks fmw42 and snibgo, great pointers.

Snibgo, I also don't know exactly what GoPro does, but the effect looks fantastic for fast moving, close-to-the-subject footage such as FPV drone racing. Your method 2 looks pretty close. I think using method 3 would lead to far too much distortion at the edges - method 2 already leaves the central third or so relatively unscathed.

I'll post back when I get this working as, unless my google-fu has failed me, there doesn't seem to be much discussion about recreating this effect.

[GeeMack]

I'm trying to recreate the "Superview" effect of a go-pro camera in linux, from a camera that doesn't have this ability. [...] Any help or hints would be greatly appreciated.

I can get the basic effect you describe by cropping a 1440x1080 image into halves, running two distortions on the halves, and re-assembling them at the end. The two distort operations are done first with perspective to get the graduated stretch, then with polynomial to reshape the resulting trapezoids back to rectangles. Those get appended to make the 1920x1080 output image. I tested this with IM6 6.9.10-11 on Windows. To translate it to Linux you should be able to just change the continued-line carets "^" to backslashes "\", and escape the parentheses "( )" with backslashes "\( \)".

Code: Select all

convert testin.png -background none -gravity center -extent 1920x1440 +gravity -crop 960x1440 ^
   ( -clone 0 -rotate 180 ) -delete 0 -virtual-pixel none ^
   -distort perspective "0,0 0,0   0,1440 0,1440   720,180 960,0   720,1260 960,1440" ^
   -distort polynomial "1.5   0,0 0,0   0,1440 0,1440   960,0 960,180   960,1440 960,1260" ^
   ( -clone 1 -rotate 180 ) -delete 1 +swap +append -shave 0x180 testout.png

[GeeMack]

I'm trying to recreate the "Superview" effect of a go-pro camera in linux, from a camera that doesn't have this ability. [...] Any help or hints would be greatly appreciated.

Another approach is to crop the image into vertical strips, use "+distort" to scale those strips progressively wider from the center ones to the outside, and reassemble the strips.

Code: Select all

convert testin.png -crop 2x1@ -distort srt %[fx:t?0:180] ^
   -append +repage -crop 8x1@ +distort srt "0,0 %[fx:1+(t*0.095)],1 0 0,0" -shave 1 ^
   +append +repage -crop 1x2@ -distort srt %[fx:t?0:180] +append -resize 1920x1080! testout2.png

I just estimated the number of strips and amount of increase of the scaling. It gets pretty close to the required 1920x1080 output size, but just to be certain it does a "-resize" at the end.

Again that is Windows syntax. Change the carets "^" to backslashes "\" for Linux.

[RustyToad]

GeeMack, thanks, that's another couple of great suggestions. I've just tried processing a short test video with the first of your two options, and it looks good. I want to try "in anger" with a proper video, but it will be the weekend before I have good 4x3 footage to experiment on.

I like the first option better as there is no sudden cutoffs within the image. As an object travels through the frame, it is important that it is handled smoothly without perspective changing suddenly. I think the invisible lines created by cutting the image into strips, as per your second example, may end up being quite visible.

Here is my lash-up of a Linux bash script to process the video, using your first option. Obviously this is quite slow to run - at 60fps, on my reasonably good laptop, it runs at about 30-60 mins per minute of footage.

Code: Select all

#!/bin/bash

ffmpeg -i $1 frames/%06d.jpg
mkdir frames
cd frames
mkdir superview_frames

for i in *.jpg
  do
  echo $i
  convert $i -background none -gravity center -extent 1920x1440 +gravity -crop 960x1440 \
    \( -clone 0 -rotate 180 \) -delete 0 -virtual-pixel none \
    -distort perspective "0,0 0,0   0,1440 0,1440   720,180 960,0   720,1260 960,1440" \
    -distort polynomial "1.5   0,0 0,0   0,1440 0,1440   960,0 960,180   960,1440 960,1260" \
    \( -clone 1 -rotate 180 \) -delete 1 +swap +append -shave 0x180 superview_frames/$i
  done

cd ..
ffmpeg -r 60 -i frames/superview_frames/%06d.jpg -r 60 -c:v libx264 -an $1_superview.mp4
rm -Rf frames/

[fmw42]

I have created a Unix Bash shell script to do this effect. It is called xpand. See my link below. It also includes an optional protect region whose scale will be preserved during the expansion process.

The script implements this as a 1D (one-sided) fx expression according to the following format:

Code: Select all

I=O+a*O^2; I=input and O=output
use Imax with Omax to evaluate "a"
I=O+O^2*(Imax-Omax)/(Omax)^2

This can be reformatted as:

Code: Select all

I=(1+a*O)*O

for which I believe can be interpreted as a linearly increasing stretch in proportion to the distance from the origin, where the scale factor is (1+a*O). Since the scale factor includes the first order O, it is linearly increasing.

[RustyToad]

@fmw42, I've only just seen this, sorry for the delay replying. This looks absolutely perfect for what I'm trying to do, but I can't get it to work...

First error I'm getting (no matter my usage) is --- INVALID NUMERATOR OF ASPECT RATIO DIMENSION ---

I think it was trying to get the aspect ratio from the wrong argument, so I've changed that, but then I'm getting --- OUTPUT WIDTH MUST BE LARGER THAN INPUT WIDTH ---

This error doesn't stop execution though, but the output is always the same as the input. I can't recreate any of the samples on your site.

I'm running as below, as per your examples (although I've tried all sorts of different things, I get the same errors):

Code: Select all

./xpand -d "4:3" -m horizontal checker_pattern.jpg test.jpg

[fmw42]

I am now getting your same error. So something has changed in IM. I will look into it and get back with a fix.

[fmw42]

I believe I have the script fixed, now. Download it again and let me know if you still have issues.

[RustyToad]

Many thanks Fred, that's working now.