Speed up “movie barcode” generation w/ IM

[killmoms]

So I attempted to use "stream" to extract portions of the frame dumped images, but it seems that it's still having to read the entirety of the source frame off the disk to do this—I sat there and watched disk activity in Activity Monitor and saw it reading GBs worth of data. I have a code snippet that makes a barcode successfully…

Code: Select all

FILES=($(find "${1}" -type f | egrep -i "\.bmp$" | sort))

for f in ${FILES[*]}; do
	stream -map rgb -storage-type char -extract 1x1080+959+0 ${f} -
done |\
convert -depth 8 -size 1080x1920 -flip rgb:- barcode.png

…but it doesn't seem to be any faster. I'm guessing at this point I'm I/O limited. (I'm not sure why this comes out flipped; probably due to the vagaries of dealing with raw data? Or something related to bitmaps and stream specifically?)

Digging through other threads, I found this interesting topic. This post in particular caught my eye:

viewtopic.php?f=2&t=18320#p70984

Could it be possible to pipe FFMPEG's output (in image2pipe/PPM mode) into a convert command and dump out a .mpc file, and then use stream to extract from that, with offsets to hit each original "frame" in what would be just one long stream of image data? This would avoid having to load full portions into RAM, nor need to read through the entire chunk of data on my hard drive, right? And the Perl script, pulling one input "image" from ffmpeg at a time, would never overload my available RAM? Maybe?

Alternatively (since it seems one can stream from JPEGs), maybe it'd make sense to go back to JPEG-compressed images so the total amount of data to be read on every iteration is smaller? Or use JPEGs with the JPEG size hint option as mentioned earlier in the thread? So many ways to skin this cat!

[fmw42]

Could it be possible to pipe FFMPEG's output (in image2pipe/PPM mode) into a convert command and dump out a .mpc file, and then use stream to extract from that, with offsets to hit each original "frame" in what would be just one long stream of image data?

This only makes sense to me if you have ffmpeg do the resize and crop. Otherwise, your stream would be huge. FFMPEG has special streams for output as I recall, though I have never used them. Search Google or FFMPEG documentation. Note I know little about streaming in IM.

[killmoms]

Could it be possible to pipe FFMPEG's output (in image2pipe/PPM mode) into a convert command and dump out a .mpc file, and then use stream to extract from that, with offsets to hit each original "frame" in what would be just one long stream of image data?

This only makes sense to me if you have ffmpeg do the resize and crop. Otherwise, your stream would be huge. FFMPEG has special streams for output as I recall, though I have never used them. Search Google or FFMPEG documentation. Note I know little about streaming in IM.

A stream -extract wouldn't be huge though, would it? If the idea is that an MPC file is a memory-mapped file written to disk, then a stream -extract will just jump to the specified offset, return the requested number of pixels, and exit, right? Obviously the MPC file itself would be enormous for HD sources (I'd probably HAVE to use the Q8 version of ImageMagick so that the MPC was roughly the same size on disk as the 11.9GB worth of BMPs I was dumping out before). But that's fine—it's a temporary file, albeit potentially a big one. If assembling the barcodes from that is enough faster (which is to say, much faster than "two and a half days"), it's worth doing. And using ffmpeg to assemble the barcodes in one shot would be even slower—it takes almost 15 minutes just to parse through the movie file to dump out the frames (and that's NOT I/O limited, it's CPU-limited on decode).

EDIT: Maybe I phrased it poorly the first time. This would still be a two step process; ffmpeg would dump all requested frames sequentially into a massive single MPC file, and then I'd design a loop to iteratively stream -extract the rows of pixels I want out of that big MPC—calling that from a loop to adjust the offset to make all the desired barcodes—rather than having to read every byte of 1920 BMP files 1920 times just to do the same thing.

[snibgo]

I'll throw in another possibility: when you see a frame, crop it into one file per vertical strip. With 2000 (yeah I know it's 1920) frames each with 2000, you then have 4,000,000 images, but each one is small. The building process just reads the appropriate images and appends them.

I don't know how fast random access is to the n-th image in a tiff. Maybe putting all the strips from a frame into a multi-image tiff would work well.

[fmw42]

I'll throw in another possibility: when you see a frame, crop it into one file per vertical strip. With 2000 (yeah I know it's 1920) frames each with 2000, you then have 4,000,000 images, but each one is small. The building process just reads the appropriate images and appends them

I think that is how he started. Loop over each image and crop and save to multiframe miff. Then append all the strips. He used montage, but convert append would work the same.

The time issue, it seems to me, is the reading of all the images, one at a time and then resizing and cropping.

[killmoms]

Ooh, what about this, then? Is there some way to read ONE frame in, then throw out each one of its columns into one file per column? And then, read the next frame in, and do the same thing—except instead of making their own files, appending them to the proper files from the previous iteration? Basically, appending the correct column to the correct output barcode, one input frame at a time? That way you'd only have to read each of the source frames once—after that it'd just be a matter of writing out the strips to the correct final barcode file.

So:

1) Read frame 1 into RAM. Output column 1 to barcode_0001. Output column 2 to barcode_0002. … Output column 1920 to barcode_1920.
2) Read frame 2 into RAM. Append column 1 to barcode_0001. Append column 2 to barcode_0002. … Append column 1920 to barcode_1920.
…
1920) Read frame 1920 into RAM. Append column 1 to barcode_0001. Append column 2 to barcode_0002. … Append column 1920 to barcode_1920.

DONE.

Would that work?

If doing this extraction/build process in rows would be easier/faster than columns, again, currently it's FAR faster to pre-size/rotate frames during their export from ffmpeg into the most efficient form for IM processing, since there so much more IM processing to be done than ffmpeg processing.

EDIT: Reading over this again, this is basically snibgo's idea above, except without trying to write ~4mil tiny files to the filesystem (which could bring its own slowdown). Unless, of course, doing 4mil appends is just as slow. But right now the slowdown is definitely that I'm trying to read 11.9GB of data 1920 times—a quick back-of-the-napkin calculation shows that my time estimate from multiplying the approximate time it was taking to assemble a barcode by the desired number of barcodes lines up with a time estimate of multiplying that data size by 1920 and dividing by the average read speed of my SSD.

Then again, that doesn't really explain why doing it with JPGs was still really slow. Hmm.

[fmw42]

I am confused. I thought you were only extracting one column from each frame (after resizing). Are you trying to get every column of each resized frame. Can you outline your processing steps again to clarify what you are doing?

You can use -crop 1xH to get every column of an image and write it to miff (or possibly mpc) format. See tile cropping at http://www.imagemagick.org/Usage/crop/#crop_tile. Also there is an in memory format (mpr:) that keeps images in memory for later use in the same command line. But I do not see how that would help. see http://www.imagemagick.org/Usage/files/#mpr

[fmw42]

Have you considered using ffmpeg to write a stream of images to ppm format and then reading that into IM for your convert loop saving to miff. See http://www.imagemagick.org/Usage/files/#image_streams

[killmoms]

I am confused. I thought you were only extracting one column from the image (after resizing). Are you trying to get every column of each resized image.

In a word, yes. Lemme try to break it down.

The first thing I do is dump out a number of frames from the source video. Basically I give ffmpeg a "frame rate" for conversion to output images that is much lower than the original frame rate—this skips many frames of the movie and just outputs a single still image every n frames. Since each column of pixels in a barcode, regardless of mode, represents one frame from the movie, I tell the script how many frames I want by specifying the final desired width of the barcode(s). So if I want a barcode that's 1920 pixels wide, the script calculates the frame rate to dump 1920 frames out of the total number of frames in the original movie, evenly spaced out across its entire duration.

My script then has the ability to create a barcode (or barcodes, I'll get to that in a second) in one of a couple different ways. Its original mode, "resize" (the one that generates barcodes that look like the ones on the Tumblr I linked), creates a single barcode by resizing each dumped frame horizontally to 1 pixel wide (averaging it horizontally) and then appends those "slices" together to make a barcode. That takes a while, because it HAS to read and process every frame to make each slice, but that's fine. It only makes ONE barcode. I'm not talking about that mode.

Its second mode, "crop", makes more detailed, vibrant, and "noisy" barcodes by cropping out a single column of pixels from each dumped frame (usually one at the center of the frame, horizontally—say, column 960 in a 1920px wide video). This is also "slow" when creating a single barcode, but again, that's not that big a deal—crop mode is also only making ONE barcode. I'm not REALLY talking about this mode either, exactly.

The third mode is the one I'm trying to speed up. This mode, "animation", uses the aforementioned crop methodology to make MANY barcodes—as many barcodes as there are pixels across in the original frames (their width). So, for a 1920 pixel wide video, the animation mode will create 1920 separate barcodes which are as wide as the number of frames that were dumped (again, this can be variable, but defaults to 1920, so let's just keep that our measure for simplicity's sake). The first barcode will be comprised of column 1 of every dumped frame. The second barcode will be comprised of column 2 of every dumped frame. And so on up through the one-thousand nine-hundred and twentieth barcode. Imagine this as creating an effect where each column of pixels in the resulting piece is like an extremely narrow window, each one a view "panning" across a separate moment in time (frame) from the original movie (and while each of these views "pans across", the cumulative visual effect of seeing these played in sequence is not one of left-to-right movement, but a lot of shifting, morphing vertical up-and-down movement—it's really kinda neat).

Any clearer?

[fmw42]

Yes, much clearer.

My script then has the ability to create a barcode (or barcodes, I'll get to that in a second) in one of a couple different ways. Its original mode, "resize" (the one that generates barcodes that look like the ones on the Tumblr I linked), creates a single barcode by resizing each dumped frame horizontally to 1 pixel wide (averaging it horizontally) and then appends those "slices" together to make a barcode. That takes a while, because it HAS to read and process every frame to make each slice, but that's fine. It only makes ONE barcode. I'm not talking about that mode.

In this case, use -scale to resize down to 1 column. It is much faster than -resize.

In your last case, you can use tile cropping to separate every column of the image and save that to a multilayer miff file. Each column will be a layer (frame or page, however you want to call it). Then you need to append from all the miff images all the frame [0] into your barcode. Then repeat for the next frame [1] to another barcode image, etc.

The issue may be how IM handles reading a layer from a miff file. If it has to read the whole image to get a single layer, then this will not work, since the second step requires you to open every miff file, which will still have the same sizes as all your frames.

However, if IM reads just one layer without opening the whole miff file, then this could work. But you would have to list each resulting image in the command line

convert frame1.miff[0] frame1.miff[1] frame1.miff[2] ... frame1.miff[N] +append barcode0.png

[killmoms]

Is there a way to get tile cropping to output to many different files, rather than just one, as I surmised in my next previous post? If that was the case, I could:

1) Read in a single frame
2) Blast out each column to its own "bucket"
3) Close that frame
4) Read in the next frame
5) Blast out each column to the proper, already-extant "buckets"
6) Close that frame
7) And so on until I'd gone through every frame.

Again, it seems to me the slowdown is in reading whole images over and over. If there was a way to "read once, split to the proper targets", that'd solve the issue. Even if I then had to run each of those "buckets" through an "append" process (say, if they were multi-layer MIFFs), that'd still only be reading the bulk of the data from disk twice—a big improvement over reading it 1920 times (for example).

[fmw42]

Is there a way to get tile cropping to output to many different files, rather than just one? If that was the case, I could:

Yes that is a good idea, you can write each column to a different image (but not a different directory, too bad). Then you have to access the correct one column (of all the original images)

Code: Select all

convert imageX -crop 1x +repage +adjoin imageX_%d.miff

The %d will result in imageX_0.miff imageX_1.miff ... imageX_N.miff. If you leave the _%d off, then it will become hypens automatically rather than underscores.

Code: Select all

convert image0_0.miff image1_0.miff image2_0.miff ... imageN_0.miff +append barcode0.png

The latter should be able to be done with a wildcard

Code: Select all

convert image*_0.miff +append  barcode0.png

[snibgo]

You were conversing as I was writing. My thoughts are slightly different to Fred's; I make no claim about which might be fastest.

Cropping each of 1920 frames into vertical strips is easy. Appending 1920 images together is easy. Doing both operations in one command isn't easy.

Suppose ffmpeg has created frame-1.png, frame-2.png, ... frame-1920.png

convert frame-1.png -crop 1x +adjoin /tempcrops/frame-1/f-%06d.miff
convert frame-2.png -crop 1x +adjoin /tempcrops/frame-2/f-%06d.miff
:
convert frame-1920.png -crop 1x +adjoin /tempcrops/frame-1920/f-%06d.miff

Now we have 1920 directories each with 1920 small files. The frame files can now be deleted.

convert @f1.txt +append out-1.png
convert @f2.txt +append out-2.png
:
convert @f1920.txt +append out-1920.png

Where f1.txt is:

Code: Select all

/tempcrops/frame-1/f-000000.miff
/tempcrops/frame-2/f-000000.miff
:
/tempcrops/frame-1920/f-000000.miff

f2.txt is:

Code: Select all

/tempcrops/frame-1/f-000001.miff
/tempcrops/frame-2/f-000001.miff
:
/tempcrops/frame-1920/f-000001.miff

f1920.txt is:

Code: Select all

/tempcrops/frame-1/f-001919.miff
/tempcrops/frame-2/f-001919.miff
:
/tempcrops/frame-1920/f-001919.miff

Now out-1.png, out-2.png etc can be assembled into a video.


Thus each frame file is read only once. Each column is written once and read once. I avoid having 4,000,000 files in one directory.

[fmw42]

Very similar ideas.

snibgo writes each set of columns from one image to its own directory rather than putting every column of every image into one directory. That is probably wise. He then creates text files to contain the correct columns to append. In either case, the extracted frame image itself can be deleted after the cropping is done.

snibgo, you don't show how you created the text files. How do you write all column 1 to a file and do that for every other column? Do you write a loop over each directory once for each column to create multiple text files after all the cropping is done?

[snibgo]

The text files (or the command-scripts, which are equivalent) can be created at any time, by a script or program with two loops. In pseudo-code:

Code: Select all

for (ColNum=0; ColNum<1920; ColNum++) {
  open_file ("f" + str(ColNum+1) + ".txt");
  for (FrameNum=1; FrameNum<=1920; FrameNum++) {
    write_file ("/tempcrops/frame-" + str(FrameNum) + "/f-" + str(ColNum, zeropad=6) + ".miff")
  }
  close_file ();
}

Of course, we don't need 1920 text files. We might create the first one, use it, re-write it for the second column, etc. I doubt this would make a significant different to the time.

If multi-image .miff or .mpc has fast random-access, that might be faster. We would reduce the number of file opens/closes by a factor of nearly 2. (Creating the column images would need 1920 file opens/closes instead of 1920*1920. But there would be no saving for the appending stage.)