Broken results on running a batch convert process

[gabw]

Hi.

Basically I've to convert tifs to pngs. The png's are getting clipped and resized, so this is my command:

Code: Select all

/usr/bin/convert 
'src.tif'
  -limit memory 0
  -limit map 0
  -limit file 0
  -alpha transparent
  -clip
  -alpha opaque
  -resize 800x600
  'end.png'
2>&1

This works perfectly on running it with a loop with 1 file or 10 files or even 100 files - every result was perfect.

So I've runned this batch-process on 14000 files overnight and the first 100 results looked well so I left it running.

The result of the ~170th file was broken, and nearly every image I found after the "first good files" was also broken... (files at the end of the post)

To be sure I've runned this process a second time and the result was the same.

The output of the convert command is fine (only some notices like 'wrong data type 7 for "RichTIFFIPTC"; tag ignored')

Sometimes (at most at the end of the process at file 12000/13000) this error shows up:

Code: Select all

convert: unable to write blob `/tmp/magick-xLs2M3Ga': No space left on device @ error/blob.c/BlobToFile/290.
convert: unable to extend cache `<input_file>': No space left on device @ error/cache.c/OpenPixelCache/4099.
convert: unable to write blob `/tmp/magick-xLs2M3Ga': No space left on device @ error/blob.c/BlobToFile/290.
convert: unable to extend cache `<input_file>': No space left on device @ error/cache.c/OpenPixelCache/4099.
convert: unable to write blob `/tmp/magick-xLs2M3Ga': No space left on device @ error/blob.c/BlobToFile/290.

The batch process works for each file and waits till the convert command exits, no multithreading/multiprocessing.

Input File:
Broken Image example: http://gab3.de/share/new/Orangina.tif

[snibgo]

`/tmp/magick-xLs2M3Ga': No space left on device

If you run out of space in tmp, things will stop working. When a process fails, it may leave temporary files there.

If the current version of IM finishes normally but leaves temporary files, please report this in the bugs forum.

[fmw42]

What version of IM are you using? If old, then upgrade. This kind of result has been seen before and may be a bug in an older version of IM that has been fixed.

[gabw]

Hi, thanks for the responses.

My server provider told me just now that on '/tmp' the space is limited, and i should just use my user-specific tmp-folder (that would be '/usr/home/user_imgdb/.tmp'

How can I specify a custom tmp path?

My Version: ImageMagick 6.7.7-10 2014-03-08 Q16

[snibgo]

Set MAGICK_TEMPORARY_PATH. See http://www.imagemagick.org/script/resou ... nvironment

[gabw]

Set MAGICK_TEMPORARY_PATH. See http://www.imagemagick.org/script/resou ... nvironment

That should do the job. Thank you very much.

[gabw]

Hi,

now i've tried it with this enviroment variable but the results doesn't change anyway. I watched the new specified tmp path and it seems IM uses this folder cause it creates files like: magick-ADJRUWA8, magick-YQqVkdFP

So what else could it be?

[snibgo]

What error message(s) do you get?

I suppose you are looping through the images. What is your code?

Your version 6.7.7 is very old. You might try a more recent version.

[gabw]

What error message(s) do you get?

I suppose you are looping through the images. What is your code?

Your version 6.7.7 is very old. You might try a more recent version.

My code looks like this (PHP 5.5):
https://gist.github.com/CodeBrauer/6200c5ca6a5befe0208e

I will try to update to the newest version.

[snibgo]

What errors are shown in log.txt?

[gabw]

So I've runned the script for 256 files.

So first of all while running the script I get this 'errors':

Code: Select all

sh: line 1: 25065 Killed                  /usr/bin/convert 'public_html/source_files/XXXX123/XXXX123/XXXX123.tif' -limit memory 0 -limit map 0 -limit file 0 -alpha transparent -clip -alpha opaque -resize 800x600 'public_html/converted/XXXX123/XXXX123/XXXX123_web.png' 2>&1
 
sh: line 1: 25702 Killed                  /usr/bin/convert 'public_html/source_files/XXXX123/XXXX123/XXXX123.tif' -limit memory 0 -limit map 0 -limit file 0 -alpha transparent -clip -alpha opaque -resize 800x600 'public_html/converted/XXXX123/XXXX123/XXXX123_web.png' 2>&1
 
sh: line 1: 26868 Killed                  /usr/bin/convert 'public_html/source_files/XXXX123/XXXX123/XXXX123.tif' -limit memory 0 -limit map 0 -limit file 0 -alpha transparent -clip -alpha opaque -resize 800x600 'public_html/converted/XXXX123/XXXX123/XXXX123_web.png' 2>&1

sh: line 1: 27680 Killed                  /usr/bin/convert 'public_html/source_files/XXXX123/XXXX123/XXXX123.tif' -limit memory 0 -limit map 0 -limit file 0 -alpha transparent -clip -alpha opaque -resize 800x600 'public_html/converted/XXXX123/XXXX123/XXXX123_web.png' 2>&1

The log shows this: https://gist.github.com/CodeBrauer/3043962064be8e78448d

Sorry - I had to censor the filenames...

FYI: Here some host-system information:
* Debian 7.8 wheezy
* Intel Xeon E3-1245
* 32GB RAM
* 3TB HDD

[snibgo]

log.txt shows no significant errors from IM.

[gabw]

log.txt shows no significant errors from IM.

So you think the server/hostsystem or my files are broken?

Running the convert command in terminal with no loop delivers a perfect result.

[gabw]

Now I transferred my whole files to another server with less power and the latest stable for Ubuntu 14.04 LTS Server (Version: ImageMagick 6.7.7-10 2014-03-06)

And i got the same errors. CPU and RAM was loaded about 35-50%, nothing special.

Here an example:
-> With loop:

-> runned in terminal