Page 1 of 1
Broken results on running a batch convert process
Posted: 2014-12-05T07:21:13-07:00
by gabw
Hi.
Basically I've to convert tifs to pngs. The png's are getting clipped and resized, so this is my command:
Code: Select all
/usr/bin/convert
'src.tif'
-limit memory 0
-limit map 0
-limit file 0
-alpha transparent
-clip
-alpha opaque
-resize 800x600
'end.png'
2>&1
This works perfectly on running it with a loop with 1 file or 10 files or even 100 files - every result was perfect.
So I've runned this batch-process on 14000 files overnight and the first 100 results looked well so I left it running.
The result of the ~170th file was broken, and nearly every image I found after the "first good files" was also broken... (files at the end of the post)
To be sure I've runned this process a second time and the result was the same.
The output of the convert command is fine (only some notices like 'wrong data type 7 for "RichTIFFIPTC"; tag ignored')
Sometimes (at most at the end of the process at file 12000/13000) this error shows up:
Code: Select all
convert: unable to write blob `/tmp/magick-xLs2M3Ga': No space left on device @ error/blob.c/BlobToFile/290.
convert: unable to extend cache `<input_file>': No space left on device @ error/cache.c/OpenPixelCache/4099.
convert: unable to write blob `/tmp/magick-xLs2M3Ga': No space left on device @ error/blob.c/BlobToFile/290.
convert: unable to extend cache `<input_file>': No space left on device @ error/cache.c/OpenPixelCache/4099.
convert: unable to write blob `/tmp/magick-xLs2M3Ga': No space left on device @ error/blob.c/BlobToFile/290.
The batch process works for each file and waits till the convert command exits, no multithreading/multiprocessing.
Input File:
Broken Image example:
http://gab3.de/share/new/Orangina.tif
Re: Broken results on running a batch convert process
Posted: 2014-12-05T07:35:56-07:00
by snibgo
gabw wrote:`/tmp/magick-xLs2M3Ga': No space left on device
If you run out of space in tmp, things will stop working. When a process fails, it may leave temporary files there.
If the current version of IM finishes normally but leaves temporary files, please report this in the bugs forum.
Re: Broken results on running a batch convert process
Posted: 2014-12-05T10:36:53-07:00
by fmw42
What version of IM are you using? If old, then upgrade. This kind of result has been seen before and may be a bug in an older version of IM that has been fixed.
Re: Broken results on running a batch convert process
Posted: 2014-12-08T01:08:09-07:00
by gabw
Hi, thanks for the responses.
My server provider told me just now that on '/tmp' the space is limited, and i should just use my user-specific tmp-folder (that would be '/usr/home/user_imgdb/.tmp'
How can I specify a custom tmp path?
My Version: ImageMagick 6.7.7-10 2014-03-08 Q16
Re: Broken results on running a batch convert process
Posted: 2014-12-08T07:37:32-07:00
by snibgo
Re: Broken results on running a batch convert process
Posted: 2014-12-08T09:42:49-07:00
by gabw
That should do the job. Thank you very much.
Re: Broken results on running a batch convert process
Posted: 2015-01-12T03:05:46-07:00
by gabw
Hi,
now i've tried it with this enviroment variable but the results doesn't change anyway. I watched the new specified tmp path and it seems IM uses this folder cause it creates files like: magick-ADJRUWA8, magick-YQqVkdFP
So what else could it be?
Re: Broken results on running a batch convert process
Posted: 2015-01-12T03:22:04-07:00
by snibgo
What error message(s) do you get?
I suppose you are looping through the images. What is your code?
Your version 6.7.7 is very old. You might try a more recent version.
Re: Broken results on running a batch convert process
Posted: 2015-01-12T05:02:08-07:00
by gabw
snibgo wrote:What error message(s) do you get?
I suppose you are looping through the images. What is your code?
Your version 6.7.7 is very old. You might try a more recent version.
My code looks like this (PHP 5.5):
https://gist.github.com/CodeBrauer/6200c5ca6a5befe0208e
I will try to update to the newest version.
Re: Broken results on running a batch convert process
Posted: 2015-01-12T06:24:15-07:00
by snibgo
What errors are shown in log.txt?
Re: Broken results on running a batch convert process
Posted: 2015-01-12T06:48:59-07:00
by gabw
So I've runned the script for 256 files.
So first of all while running the script I get this 'errors':
Code: Select all
sh: line 1: 25065 Killed /usr/bin/convert 'public_html/source_files/XXXX123/XXXX123/XXXX123.tif' -limit memory 0 -limit map 0 -limit file 0 -alpha transparent -clip -alpha opaque -resize 800x600 'public_html/converted/XXXX123/XXXX123/XXXX123_web.png' 2>&1
sh: line 1: 25702 Killed /usr/bin/convert 'public_html/source_files/XXXX123/XXXX123/XXXX123.tif' -limit memory 0 -limit map 0 -limit file 0 -alpha transparent -clip -alpha opaque -resize 800x600 'public_html/converted/XXXX123/XXXX123/XXXX123_web.png' 2>&1
sh: line 1: 26868 Killed /usr/bin/convert 'public_html/source_files/XXXX123/XXXX123/XXXX123.tif' -limit memory 0 -limit map 0 -limit file 0 -alpha transparent -clip -alpha opaque -resize 800x600 'public_html/converted/XXXX123/XXXX123/XXXX123_web.png' 2>&1
sh: line 1: 27680 Killed /usr/bin/convert 'public_html/source_files/XXXX123/XXXX123/XXXX123.tif' -limit memory 0 -limit map 0 -limit file 0 -alpha transparent -clip -alpha opaque -resize 800x600 'public_html/converted/XXXX123/XXXX123/XXXX123_web.png' 2>&1
The log shows this:
https://gist.github.com/CodeBrauer/3043962064be8e78448d
Sorry - I had to censor the filenames...
FYI: Here some host-system information:
* Debian 7.8 wheezy
* Intel Xeon E3-1245
* 32GB RAM
* 3TB HDD
Re: Broken results on running a batch convert process
Posted: 2015-01-12T07:04:34-07:00
by snibgo
log.txt shows no significant errors from IM.
Re: Broken results on running a batch convert process
Posted: 2015-01-12T07:11:25-07:00
by gabw
snibgo wrote:log.txt shows no significant errors from IM.
So you think the server/hostsystem or my files are broken?
Running the convert command in terminal with no loop delivers a perfect result.
Re: Broken results on running a batch convert process
Posted: 2015-01-12T09:19:51-07:00
by gabw
Now I transferred my whole files to another server with less power and the latest stable for Ubuntu 14.04 LTS Server (Version: ImageMagick 6.7.7-10 2014-03-06)
And i got the same errors. CPU and RAM was loaded about 35-50%, nothing special.
Here an example:
-> With loop:
-> runned in terminal
