Page 1 of 1
Convert does not finish
Posted: 2017-07-19T00:33:03-07:00
by groovejunkk
Hello Guys,
I'm quite new to computer science at all so please excuse any mistakes you might see.
I wanted to build a script to unify images in bash, so I did the following:
Code: Select all
#!/bin/bash
for pic in /some/dir/*.jpg; do
echo $pic
convert $pic -resize 2000x2000^ $pic
echo "resize done"
convert -verbose -units PixelsPerInch $pic -resample 300 $pic
done
When I run the script it starts the second convert on the first image but never finishes it.
This is what my output is, due to the verbose flag
Code: Select all
/some/dir/448528.jpg JPEG 2000x2000 2000x2000+0+0 8-bit sRGB 3.208MB 0.080u 0:00.080
I wrote the second convert to bring my images up to 300 dpi from 96 dpi.
I run ImageMagick 6.9.7-4 Q16 x86_64 20170114 on debian stable.
Any help is of course greatly appreciated.
Re: Convert does not finish
Posted: 2017-07-19T04:27:08-07:00
by snibgo
groovejunkk wrote:... but never finishes it.
I suppose that means it was doing work for a long time, and you killed it.
You claim the input was 96 dpi, but have you checked this? Resampling from 96 dpi to 300 dpi will increase the number of pixels by a factor approx 3 in each dimension, so from 2000x2000 to 6000x6000. That shouldn't be a problem, but if the input was 1 dpi the output would be 600000x600000 which would be a problem.
Re: Convert does not finish
Posted: 2017-07-19T04:53:14-07:00
by groovejunkk
Yes I have checked the dpi for the input pictures. My input pictures are approximately 1200x2400px at 96dpi or some at 72dpi which i just found out.
Never finishing in my case means I left it running for a good 45 minutes with no result. This seemed to slow on a quadcore cpu with 16gb of ram in my opinion.
I just ran identify -verbose on one of the pictures (already resized to 2000x2000) but the data seems alright to me
Code: Select all
Image: 448528.jpg
Format: JPEG (Joint Photographic Experts Group JFIF format)
Mime type: image/jpeg
Class: DirectClass
Geometry: 2000x2000+0+0
Resolution: 72x72
Print size: 27.7778x27.7778
Units: PixelsPerInch
Type: TrueColor
Endianess: Undefined
Colorspace: sRGB
Depth: 8-bit
Channel depth:
red: 8-bit
green: 8-bit
blue: 8-bit
Channel statistics:
Pixels: 4000000
Red:
min: 161 (0.631373)
max: 255 (1)
mean: 218.122 (0.855379)
standard deviation: 10.5832 (0.0415026)
kurtosis: 0.0607825
skewness: -0.0176774
entropy: 0.835328
Green:
min: 151 (0.592157)
max: 255 (1)
mean: 208.093 (0.81605)
standard deviation: 11.0974 (0.0435192)
kurtosis: 0.132938
skewness: 0.0657159
entropy: 0.823453
Blue:
min: 152 (0.596078)
max: 255 (1)
mean: 210.699 (0.826271)
standard deviation: 11.152 (0.0437331)
kurtosis: 0.164899
skewness: -0.0369316
entropy: 0.824477
Image statistics:
Overall:
min: 151 (0.592157)
max: 255 (1)
mean: 212.305 (0.832567)
standard deviation: 10.9472 (0.0429301)
kurtosis: 1.00205
skewness: -0.0164577
entropy: 0.827753
Rendering intent: Perceptual
Gamma: 0.454545
Chromaticity:
red primary: (0.64,0.33)
green primary: (0.3,0.6)
blue primary: (0.15,0.06)
white point: (0.3127,0.329)
Background color: white
Border color: srgb(223,223,223)
Matte color: grey74
Transparent color: black
Interlace: None
Intensity: Undefined
Compose: Over
Page geometry: 2000x2000+0+0
Dispose: Undefined
Iterations: 0
Compression: JPEG
Quality: 99
Orientation: TopLeft
Properties:
date:create: 2017-07-19T09:29:03+02:00
date:modify: 2017-07-19T09:29:03+02:00
exif:BitsPerSample: 8, 8, 8
exif:ColorSpace: 65535
exif:Compression: 1
exif:Copyright: Copyright (C) reserved
exif:DateTime: 2014:12:08 15:19:49
exif:ExifImageLength: 1060
exif:ExifImageWidth: 1060
exif:ExifOffset: 344
exif:ImageLength: 1060
exif:ImageWidth: 1060
exif:Model:
exif:Orientation: 1
exif:PhotometricInterpretation: 2
exif:PlanarConfiguration: 1
exif:ResolutionUnit: 2
exif:SamplesPerPixel: 3
exif:Software: Adobe Photoshop CC (Macintosh)
exif:thumbnail:Compression: 6
exif:thumbnail:JPEGInterchangeFormat: 482
exif:thumbnail:JPEGInterchangeFormatLength: 1392
exif:thumbnail:ResolutionUnit: 2
exif:thumbnail:XResolution: 72/1
exif:thumbnail:YResolution: 72/1
exif:XResolution: 72/1
exif:YResolution: 72/1
icc:copyright: Copyright (C) 2007 by Color Solutions, All Rights Reserved. License details can be found on: http://www.eci.org/eci/en/eciRGB.php
icc:description: eciRGB v2
icc:manufacturer: eciRGB v2
icc:model: eciRGB v2
jpeg:colorspace: 2
jpeg:sampling-factor: 1x1,1x1,1x1
signature: 3d8fcd1bd2bf9b913bccddad2b6472d5c1ff94ee39613f0313d87fd4e6bafd77
Profiles:
Profile-8bim: 4098 bytes
Profile-exif: 1880 bytes
Profile-icc: 1992 bytes
Profile-iptc: 50 bytes
City[1,90]: 0x00000000: 254700 -%
City[1,90]: 0x00000000: 254700 -%
unknown[2,0]:
Copyright String[2,116]: Copyright (C) reserved
Profile-xmp: 5054 bytes
Artifacts:
filename: 448528.jpg
verbose: true
Tainted: False
Filesize: 3.208MB
Number pixels: 4M
Pixels per second: 40MB
User time: 0.110u
Elapsed time: 0:01.099
Version: ImageMagick 6.9.7-4 Q16 x86_64 20170114 http://www.imagemagick.org
Re: Convert does not finish
Posted: 2017-07-19T05:09:52-07:00
by snibgo
I would expect a handful of seconds.
What is it doing in that time? Do you have temporary files in /tmp? What does "convert -list resource" say? On my machine, this is:
Code: Select all
Resource limits:
Width: 214.7MP
Height: 214.7MP
Area: 8.4474GP
Memory: 3.9336GiB
Map: 7.8672GiB
Disk: unlimited
File: 1536
Thread: 8
Throttle: 0
Time: unlimited
Re: Convert does not finish
Posted: 2017-07-19T23:48:52-07:00
by groovejunkk
Code: Select all
Resource limits:
Width: 20KP
Height: 20KP
Area: 512MP
Memory: 6GiB
Map: 512MiB
Disk: 10GiB
File: 49152
Thread: 4
Throttle: 0
Time: unlimited
I suppose the width and height parameters are a little small but still extensive enough for 6000x6000.
The map and area parameters seem a little far off to me. As I understood the map determines the size of the pixel cache in memory, so a value like this shouldn't be a problem with pictures between 3 and 4 MB. Still I am uncertain about the area value.
I found that in the policy.xml there is
Code: Select all
<policy domain="resource" name="area" value="512MB"/>
But in -identify resource it says MP instead of MB.
When the command runs I get a temp file named magick-126892JjPX_qof01W, but I'm not sure how to investigate the file further.
Re: Convert does not finish
Posted: 2017-07-20T03:03:35-07:00
by groovejunkk
I just had a quick look at the process in htop and it shows 0% CPU and 0% memory load for the convert process.
Re: Convert does not finish
Posted: 2017-07-20T04:05:27-07:00
by snibgo
groovejunkk wrote:... a temp file named magick-126892JjPX_qof01W
That's probably a pixel cache, on disk because IM can't get enough real memory. Your limits look okay, so something else is limiting your process.
How large is it, in bytes? If your IM is Q16 integer, divide bytes by 8 to get pixels.
Re: Convert does not finish
Posted: 2017-07-20T05:14:14-07:00
by groovejunkk
Code: Select all
du -b 448528.jpg
23808547 448528.jpg
So the first Image has 23808547 bytes
Re: Convert does not finish
Posted: 2017-07-20T06:02:06-07:00
by snibgo
Divide by 8 for the number of pixels, and that's approx 1200*2400 pixels. So IM is asking for memory for this pixel cache, and your OS is declining the request, so IM is using disk.
What OS sysadmin tools should you use to diagnose and fix this? Sorry, I don't know.
Re: Convert does not finish
Posted: 2017-07-20T08:45:22-07:00
by groovejunkk
But shouldn't disk space also be somewhat reasonably fast? I've got a pretty fast SSD in my machine..
Anyway: Thank you for your time and help I think I know some people to ask when it comes to debian sysadmin tools
Re: Convert does not finish
Posted: 2017-07-20T09:21:07-07:00
by snibgo
Yes, I think your process isn't just slow, but stopped. I don't know why, only that IM is asking for resources it isn't getting. My last serious work on Unix was 22 years ago, so I don't know where to begin looking. Sorry.
Re: Convert does not finish
Posted: 2017-07-20T10:03:41-07:00
by fmw42
Both these commands work fine for me on IM 6.9.9.0 Q16 Mac OSX.
Code: Select all
for pic in /Users/fred/desktop/test/*.jpg; do
echo $pic
convert $pic -resize 2000x2000^ $pic
echo "resize done"
convert -verbose $pic -units PixelsPerInch -resample 300 $pic
done
Code: Select all
for pic in /Users/fred/desktop/test/*.jpg; do
echo $pic
convert $pic -resize 2000x2000^ -verbose -units PixelsPerInch -resample 300 $pic
echo "done"
done
Except that the -resample did nothing. See below regarding your input files having no units or density specified.
Have you tried just copying and pasting the code into a terminal window rather than calling it as a script?
What dimensions are your input images?
Do you really want to resample the images or do you just want to have the resized results assigned 300 dpi. If the latter then just do
Code: Select all
convert $pic -resize 2000x2000^ -units PixelsPerInch -density 300 $pic
Another possible issue is that your input images have no units define and possibly no density or density=1. That may cause an issue on your version of IM. You can check by identify -verbose input.jpg. So the proper way to do your command would be to make sure you have assigned a density and units to your input images or the results of your resized images. Nominal density is 72. But if your input files have no units, then the density that it might have is useless or possibly it gets assigned a density of 1.
Code: Select all
for pic in /Users/fred/desktop/test/*.jpg; do
echo $pic
convert $pic -resize 2000x2000^ -units PixelsPerInch -density 72 $pic
echo "resize done"
convert -verbose $pic -units PixelsPerInch -resample 300 $pic
done
Now the -resample makes the output images much larger and cases the script to run much slower. Output sizes are >10,000 due to going from 72 dpi to 300 dpi. That is why I asked if you really wanted the resample and not just setting the output density.
Re: Convert does not finish
Posted: 2017-08-04T08:23:42-07:00
by groovejunkk
After some intense reading about images i learned, that I indeed, did not want to resample the Images.
Anyway I want to thank you for your kind help and have to say I appreciate this community and will try to become part of it.