Extract a region of an huge jpeg

[anthony]

Yes that seems to tweek some old memory cells. That may be it. Though as I said my own jpegtran program seems to be missing that feature.

My 'yum' package management system say two packages has jpegtran program in them, but the second package is part of a huge dependancy set, and I don't really want to install it just to check out its version.

[alex88]

My friend tried for me (i'm on mac), seems working, 1 second to output a desidered portion of an image
But i need more testing

PS: now i think someone shold implement this in imagemagick to add features

[anthony]

I doubt it will be added to IM, as it is very JPEG specific.

Also remember the crop would be limited to multiples of 8. so while you can jpeg crop the specific area quick some slight extra cropping is needed in IM before actual processing.


Can you send me all the details. The JPEGtrans program used (preferably a link) and the command line, with final cleanup. I would like to at least report the details in IM Examples, Massive Image Handling, even if I can't actually run it myself.

Of course I'll be linking back to this discussion too.

[alex88]

Sure, give me some minutes, i need to install windows and then i'll post examples too.

Just a thing, what you mean with final cleanup?

[Drarakel]

Also remember the crop would be limited to multiples of 8.

With lossless JPEG cropping, the size is not limited - but the starting point must be at a block boundary. Or, to put it in other words, the upper left point must be at a multiple of 8 or 16 (even bigger blocks are possible - but not widely used), and the lower right point can be anywhere. Without further knowledge about the input files, I would use a starting point/offset at multiples of 16. If you want to be more precise, first check the sampling-factor of your JPEG in order to know the block sizes. If your starting point is not on a block boundary, it will be moved to the 'next' (up and/or left) block, so the size will also be increased (at least with jpegtran).
See also the jpegtran documentation for that.

An example with jpegtran would be:

Code: Select all

jpegtran -crop 100x100+0+0 -copy none image.jpg output.jpg

(That of course works without problems, as the offset 0,0 is always on a block boundary.)

I don't have that much experience with jpegtran alone (I normally use jpegcrop for lossless cropping - a GUI that also uses the IJG library). But from some small tests, I would say, that it's a bit quicker when compared to the stream/convert solution. But: Contrary to stream/convert, it consumes a large part of the memory - and the option for restricting the memory usage doesn't seem to work (at least with my Windows binaries of the IJG 8c library).
You'll have to test it with your very large images.. (I sometimes use it with large images, but normally not larger than 20000x20000.)

[alex88]

So, these are my tests:

Downloaded this, the one you can find in the "Lossless crop 'n' drop (cut & paste)" part of this page.

Test #1:

Image: Lighthouse image from windows 7 pictures
Size: 1024x768
Command: .\jpegtran.exe -crop 100x100+0+0 .\Lighthouse.jpg output.jpg
Time took: 76 ms


Test #2:

Image: http://goes.gsfc.nasa.gov/pub/goes/0809 ... poster.jpg
Size: 12000x12000
Command: .\jpegtran.exe -crop 100x100+0+0 .\080913.ike.poster.jpg output.jpg
Time took: 1251 ms

Test #3:

Image: http://rvvs89.ucc.asn.au/stuff/huge/huge4.jpg
Size: 30000x30000
Command: .\jpegtran.exe -crop 100x100+0+0 .\huge4.jpg output.jpg
Failed: Insufficient memory (case 4) (even with maxmemory set to 500000)

Kinda fail... :/ Btw you're true, it takes about 500mb when processing the 12kx12k px image, like it decodes the whole image.

[Drarakel]

Image: http://rvvs89.ucc.asn.au/stuff/huge/huge4.jpg
Size: 30000x30000
Command: .\jpegtran.exe -crop 100x100+0+0 .\huge4.jpg output.jpg
Failed: Insufficient memory (case 4) (even with maxmemory set to 500000)

Well, with your huge4.jpg, ImageMagick (even stream) throws that error, too - at least on my system:
identify -ping huge4.jpg
Magick: Insufficient memory (case 4) `huge4.jpg' @ error/jpeg.c/EmitMessage/235.

Downloaded this, the one you can find in the "Lossless crop 'n' drop (cut & paste)" part of this page.

Yeah, I'm also using that version.

By the way: huge4.jpg has 2x2 subsampling (block size 16x16), so you would have to use the multiples of 16 - if it would work with that image.
And, I think, it's progressive. Now I don't know the internals of progressive JPGs, but it's possible that the complete file has to be read with that type before a certain block can be accessed.
Perhaps test jpegtran with a normal (non-progressive) 30000x30000 JPG.

[alex88]

By the way: huge4.jpg has 2x2 subsampling (block size 16x16), so you would have to use the multiples of 16 - if it would work with that image.
And, I think, it's progressive. Now I don't know the internals of progressive JPGs, but it's possible that the complete file has to be read with that type before a certain block can be accessed.
Perhaps test jpegtran with a normal (non-progressive) 30000x3000 JPG.

You mean using .\jpegtran.exe -crop 1600x1600+0+0 .\huge4.jpg output.jpg? Same thing to me.

Even with this image (sorry for megaupload but a friend passed to me, normal 30000x30000 image made with photoshop and saved as jpeg) i get memory error.

[Drarakel]

You mean using .\jpegtran.exe -crop 1600x1600+0+0 .\huge4.jpg output.jpg?

No, no - that thing with the multiples was only a general hint regarding the starting point/offset. But with "+0+0", that's of course no problem here.

Even with this image (sorry for megaupload but a friend passed to me, normal 30000x30000 image made with photoshop and saved as jpeg) i get memory error.

Hmm...
But at least that image works again with my ImageMagick installation - and the stream/convert solution is relatively fast: 10 seconds (with my rather old system).

[alex88]

But at least that image works again with my ImageMagick installation - and the stream/convert solution is relatively fast: 10 seconds (with my rather old system).

10 seconds? mmhh,, can you please post the command used? So i try that also, and the version of imagemagick used.

EDIT: nm i've tried with

Code: Select all

stream -map rgb -storage-type char -extract 100x100+0+0 image.jpg - | convert -depth 8 -size 100x100 rgb:- output.jpg

and worked in 6 seconds. But i still don't get why my java program does that in 300ms in the top left part of the image.

Probably for a whole image process (i mean subdivide the image in parts) it's better to divide the image in block and then subdivide the remaining blocks, but i think that still isn't "skipping" to a starting pixel and start reading from there, it still takes too much time.

[Drarakel]

it still takes too much time.

But way less time than with a normal "convert -crop" - on my system, this would take not 10 seconds, but 10 minutes! (As convert really reads the full image, and that's too big for my RAM, so sloow swapping starts.)

[alex88]

it still takes too much time.

But way less time than with a normal "convert -crop" - on my system, this would take not 10 seconds, but 10 minutes! (As convert really reads the full image, and that's too big for my RAM, so sloow swapping starts.)

Well sure, but i think there is a faster way, i think i should start learning jpeg decoding for skipping to a pixel and read only the necessary.

[anthony]

With lossless JPEG cropping, the size is not limited - but the starting point must be at a block boundary. Or, to put it in other words, the upper left point must be at a multiple of 8 or 16 (even bigger blocks are possible - but not widely used), and the lower right point can be anywhere.

Okay I gather then that if the offset of the crop is moved, the image size will be increase just enough to include all the pixels requested. As size is now a problme then the final cleanup of the region should be a straight forward trim of the top and left pixels to get the actual size wanted. A SouthEast crop in IM should do that task nicely!

Code: Select all

  jpegtran -crop 100x100+123+425 -copy none huge.jpeg  crop.jpg
  convert crop.jpg -gravity SouthEast -crop 100x100+0+0 +repage crop_fixed.png

It then should not matter if it is 8 or 16 (or other) sampling. Though right and bottom edge crops may need to be adjusted manually.

The final region output is PNG to avoid data loss as IM will decode the JPEG data. Note I can not test the above at this time as I do not have this version of jpegtran. So can someone verify.

Are the same arguments used for jpegcrop?
If so that may be the better suggestion, so as to avoid confusion with the multiple versions of jpegtran.

[anthony]

The image... http://rvvs89.ucc.asn.au/stuff/huge/huge4.jpg doesn't work with Firefox either

It reports... The Image {...} cannot be displayed, because it contains errors.

Perhaps it is again because it is too big! But it is hard to say for ceratin.

[anthony]

But i think there is a faster way, i think i should start learning jpeg decoding for skipping to a pixel and read only the necessary.

Hmmm.. skipping will be the way to go for handling HUGH images. However it does mean the image must then be disk based and can not be comming from a network stream or pipeline. That is the tradeoff. It may be why the previous commands still reads the whole file, even if it is not storing it all into memory.

Still it would be a great addition.

Question is the original JPEG Club people still active. Those pages and utilities are actually quite old. If they are active perhaps they would like to help.