Extracting change events from a video into separate images.

[RippleRatt]

Hi,

I been working on this for a while and finally decided to ask for help.
I identified the entire process so you can recommend changing any of them to allow for a better end result.

What i have is basically a high res color video traffic camera output files.
I need to extract images , jpgs, for each different event that happened,
zoomed in as much as possible around the event and touched up for lighting.


1. cut a video into one second snapshots.
I got this done already using ffmpeg gif png jpg any format and quality.

2. get differences between each snapshot using compare or diff. not sure best way.
best command so far is:
compare -fuzz 35% "video-1.frame-08.png" "video-1.frame-07.png" -highlight-color black -lowlight-color white compare.video-1.frames.07.08.fuzz35.png
it still has noise in it but the objects are there and they are dark the rest is white,
if the black and white idea is not right for the next step I can change that.

reference image: there can also be objects in both images i am using which messes up my compare. but cant use one reference image for an all day long sample.
was thinking of creating an "evolving reference image" by adding comparing several frames and discarding the difference areas to basically get a blank background image. as it existed just before the image to be sampled. so that it changes as the light and clouds of the day changes.



3. then extract/crop from the original image the major difference areas, with some padding to a 3:2 ratio, saving each to a file.

I have tried countless methods for many months., all brute force, fragile, and time consuming.
each involved either row column scanning looking for black in a white image or creating a matrix of files via crop, both then would use identify looking for dark tiles and return the position so i could open the original and corp it at that each location.

Just not working well.

seeing what you guys can do to images already there has to be a stable and intelligent way to do this?



Thanks
Ripple


windows 7 maybe unix later.
Version: ImageMagick 6.9.0-7 Q16 x86 2015-02-18
batch or ksh
android or tablet eventually maybe.

[snibgo]

To find what has changed between two frames, I use "-compose difference", like this. (Windows BAT syntax.)

Code: Select all

convert ^
  frame000001.png frame000002.png ^
  -compose Difference -composite ^
  {processing} ^
  diff.png

The {processing} step depends on what I'm looking for, and what I will do with the result. It could be a simple threshold, or a "-level". The result is black where pixels haven't significantly changed and white where they have. (But this can be negated, of course.) Some de-noising might be needed eg for vegetation that bends in the wind.

For the evolving reference image: some situations have constant movement. Other (like road junctions) have objects that move into frame, then are still for a period, then move out. We generally want the reference image to have no objects. Busy road junctions are tricky as they more often than not have objects. The trick then is to get say 36 frames at 5-second intervals, so covering 3 minutes. If two adjacent frames have a black difference pixel, that pixel is either the background or a vehicle that isn't moving. If a pixel is the same in frames 1 and 2 and 24 and 25, then it is either background or a vehicle that hasn't moved for 2 minutes. Adjust the timing as required.

Once you have an evolving reference image, the "-compose Difference -composite" between the reference and a frame gives us white where the objects are, so can be used as a mask to make the background transparent, so you can trim it away, or whatever you want to do.

All the above uses standard IM commands, and can be fast. A performance trick is to ensure that frames are not re-read. This can involve writing a script that builds a long command that processes 100 or so frames at once.

I have written (but not yet published) software that creates displacement maps. In this application, it could show how much a vehicle has moved between frames. However, a better approach might be to "break into" a video format that already contains motion vectors.

Sorry, I've rambled. Hope this helps.

[RippleRatt]

OK, I got the reference image part and getting the DIFF file to where i can clean it up some but i dont know what the next step will require so i cant clean the result of the previous step.

So, how to I get from creating the DIFFERENCE file, which is the the output of this convert command

Code: Select all

convert   frame000001.png frame000002.png   -compose Difference -composite   {processing}   diff.png

to getting the location of the difference areas and then cropping around those areas in the original file.?

its the method i am not even sure of any more. masking? matrix? nested loops taking samples?
is it possible to do in one step seeing how i have the source file loaded to run the convert already?
or is is a matter of getting the location then running a crop as a second step. ?

THanks

[fmw42]

So, how to I get from creating the DIFFERENCE file, to getting the location of the difference areas and then cropping around those areas in the original file.?

Take a look at connected-components labeling at http://magick.imagemagick.org/script/co ... onents.php

[snibgo]

So, how to I get from ...

I don't fully understand your questions.

We can usually merge multiple convert commands into one command, but first we need to decide what we are trying to do.

I suggest you put up some example frames. You can upload to somewhere like dropbox.com and paste the URLs here.

[RippleRatt]

mock up samples are here :
https://goo.gl/photos/nUmsPGVbN6VaDLgS6

end to end is step 1. source image from video to final crop images.

steps 2, 3, 4, 5 are my thinking on how to do the process step by step but if you can come up with something else please do.

[fmw42]

You can extract the bounding boxes of the two dark areas in image "cleaned diff image.jpg" by using connected-components labeling as per my note above. You should threshold that image to binary before processing it with CCL

[RippleRatt]

So, how to I get from creating the DIFFERENCE file, to getting the location of the difference areas and then cropping around those areas in the original file.?

Take a look at connected-components labeling at http://magick.imagemagick.org/script/co ... onents.php

Exactly what I have been needing. !!!!!

I started research two years ago by reading about every feature and flag and dont think i ever saw this one.
when did this go in?

this will also work with my crop process I been trying to use with non-tripod mounted still images, photographer hand held where compare cant be used but i still need to find and extract images the same way..
Its really about finding the foreground image from the background on those human taken photos then crop around it. it works but its fragile and needs to be monitored due to constant changes but this will help in the stability side of the problem list so i can focus on the other problems.

Lemme play with this a bit and see, i sure i will need help to get it right.

I knew i should have asked for help sooner but dam pride gets in the way.

Thanks a bunch.

-- there is finally a light at the end of the tunnel and it's not a train.

[RippleRatt]

I see the syntax and how to pass the params from the data collected by CC

so all i need to do is ....
pass the coordinates found in the diff image using CC ..
to a crop command on the source image..
for several found areas ..
and put that ALL ON ONE COMMAND LINE.
ya right... LOL... that is way beyond my ability to even begin to do on my own.
can you save me some time with a quick example of how to do that.

actually, I will have to pad the coordinates of the retrieved CC data before doing the actual cropping cause the cleaning process made a smaller overall target to detect and will vary depending on level of noise in the background. so the detected area could become quite small and need a lot of padding on some and not so much for a blue sky background.

so i am thinking i may have to still do this one command at at time so i can intervene with changes first.
unless yo know of a way to pad the CC results on the command line can you tell me how to:
export the CC data to a CSV or other text file , then I can grab the data and pad it in a separate command step.

thanks,

[fmw42]

I could show you how to write a loop to crop the image from the CCL data, but only for unix. One of the Windows users would need to help for your platform.

[RippleRatt]

I am well versed in Unix and Ksh as well as batch and windows. I end up with a mix of both on the pc with the more complex operations in a call to sh. I thinking I will have to port all to unix sh eventually if i want to run from tablet or android if that is even possible.

but ya, I am thinking i have to do the entire process twice if heavy clean up is needed to remove the background noise. I got that but thanks for offering.

I gonna go play with the CC command and let you know how it worked out.

OH, what about how to save the data from CC to a file or some how passing it to the calling shell?

I will be back if i run into problems a lot sooner instead of spending countless hours trying things.. best to just ask the experts

what would we do with out you guys.?

has anyone tried to create a GUI wrapper or matrix pages in html to simply see results of applying different values with a set of predetermined levels. then display code that it used to create the displayed image.
it would never be able to cover the wide range of usages but it may help the learning curve for new people so they dont use up your time with simple things.

Thanks again for the help and advice..
your the best.

Ripple Ratt
Phat Ratt Photo

[snibgo]

a GUI wrapper ... it would never be able to cover the wide range of usages but it may help the learning curve for new people so they dont use up your time with simple things.

In my view, people who are new to a product should do two things: read the manual, and experiment. A GUI that generates IM commands is useful (I use one myself), but I'm not sure it helps learning curves. On the contrary, it's like asking an expert to write a script to do something. Okay, so the learner then has a script, but have they learnt anything? Not unless they take the time to study the script.

IM is large and complex. But there is plenty of documentation that gives example usage, and reference pages, and commented source code for those who want to look under the hood.

And GUIs have limitations. For example, mine records all the editing operations as equivalent IM script commands. It creates code that works, of course, but not written in a way a human would.