Errors on batch processing

[mgo]

I am getting once every few days (let's say every 30 batch runs) a lot of errors from ImageMagick.
The errors cannot be reproduced right after, so I suspect batch processing nature of the problem.
The processed files are correct, and the errors cannot be reproduced while processing a single file.

I will post the errors, perhaps you will have an idea what happens.
Batch run 1:

Code: Select all

composite: improper image header `/home/events/2019/03/15/14652/872287/dsc-1-13_baae18_940x627_border.png' @ error/png.c/ReadPNGImage/4291.
composite:  `/home/events/2019/03/15/14652/872287/dsc-1-13_baae18_940x627_border.png' @ error/composite.c/CompositeImageCommand/1627.
composite: IDAT: CRC error `/home/events/2019/03/15/14652/872288/dsc-1-14_2835e3_940x627_border.png' @ error/png.c/MagickPNGErrorHandler/1714.
composite:  `/home/events/2019/03/15/14652/872288/dsc-1-14_2835e3_940x627_border.png' @ error/composite.c/CompositeImageCommand/1627.
composite: IDAT: CRC error `/home/events/2019/03/15/14652/872292/dsc-1-18_7ca1e0_940x627_border.png' @ error/png.c/MagickPNGErrorHandler/1714.
composite:  `/home/events/2019/03/15/14652/872292/dsc-1-18_7ca1e0_940x627_border.png' @ error/composite.c/CompositeImageCommand/1627.
convert: Premature end of JPEG file `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf.jpg' @ warning/jpeg.c/JPEGWarningHandler/396.
convert: Corrupt JPEG data: premature end of data segment `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf.jpg' @ warning/jpeg.c/JPEGWarningHandler/396.
composite: IDAT: CRC error `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf_940x627_border.png' @ error/png.c/MagickPNGErrorHandler/1714.
composite:  `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf_940x627_border.png' @ error/composite.c/CompositeImageCommand/1627.
convert: Premature end of JPEG file `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf.jpg' @ warning/jpeg.c/JPEGWarningHandler/396.
convert: Corrupt JPEG data: premature end of data segment `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf.jpg' @ warning/jpeg.c/JPEGWarningHandler/396.
convert: Premature end of JPEG file `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf.jpg' @ warning/jpeg.c/JPEGWarningHandler/396.
convert: Corrupt JPEG data: premature end of data segment `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf.jpg' @ warning/jpeg.c/JPEGWarningHandler/396.
convert: Premature end of JPEG file `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf.jpg' @ warning/jpeg.c/JPEGWarningHandler/396.
convert: Corrupt JPEG data: premature end of data segment `/home/events/2019/03/15/14652/872293/dsc-1-19_e22eaf.jpg' @ warning/jpeg.c/JPEGWarningHandler/396.
composite: IDAT: CRC error `/home/events/2019/03/15/14652/872294/dsc-1-20_4266c2_940x627_border.png' @ error/png.c/MagickPNGErrorHandler/1714.
composite:  `/home/events/2019/03/15/14652/872294/dsc-1-20_4266c2_940x627_border.png' @ error/composite.c/CompositeImageCommand/1627.
Not a JPEG file: starts with 0x00 0x00
convert: IDAT: CRC error `/home/events/2019/03/15/14652/872295/dsc-1-21_317185_940x627_border.png' @ error/png.c/MagickPNGErrorHandler/1714.
convert: no images defined `/home/events/2019/03/15/14652/872295/dsc-1-21_317185_940x627_border.jpg' @ error/convert.c/ConvertImageCommand/3300.
Aborted
Not a JPEG file: starts with 0x00 0x00
composite: IDAT: CRC error `/home/events/2019/03/15/14652/872297/dsc-1-23_1e454f_940x627_border.png' @ error/png.c/MagickPNGErrorHandler/1714.
composite:  `/home/events/2019/03/15/14652/872297/dsc-1-23_1e454f_940x627_border.png' @ error/composite.c/CompositeImageCommand/1627.
composite: IDAT: CRC error `/home/events/2019/03/15/14652/872299/dsc-1-25_f28d7c_940x627_border.png' @ error/png.c/MagickPNGErrorHandler/1714.
composite:  `/home/events/2019/03/15/14652/872299/dsc-1-25_f28d7c_940x627_border.png' @ error/composite.c/CompositeImageCommand/1627.
composite: IDAT: CRC error `/home/events/2019/03/15/14652/872300/dsc-1-26_a3fe1f_940x627_border.png' @ error/png.c/MagickPNGErrorHandler/1714.
composite:  `/home/events/2019/03/15/14652/872300/dsc-1-26_a3fe1f_940x627_border.png' @ error/composite.c/CompositeImageCommand/1627.

The commands are nothing special:
composite with -gravity SouthEast creating the _border.png file (why "improper image header" since the file is created by IM?)
and then convert, which scales/crops.

[snibgo]

The error messages are consistent with running an IM command on an input image file before it has been fully written. Perhaps your batch process detects a file and starts processing it before a previous process has finished writing it.

[mgo]

Dear Snibgo. I will investigate this possibility. It is likely. Thank you!

[mgo]

Hmm. Does convert/composite end before file bas heen fully written?

[snibgo]

It depends on your computer and operating system.

On my computer (HP laptop, running Windows 8.1) when I run a series of commands in one window, each command won't start until the previous command has fully written and closed any files.

But I might run a series of commands in one window, and another series of commands in another window. The first window might have a command that creates a file, then takes time writing it (eg a large PNG file). The second window might have a command that waits until it sees a PNG file has been created, and then tries to read and process it. But that can happen before the PNG file has been totally written, so the file appears to be corrupt.

A simple workaround is to create an extra command at the end of the first series that writes another file, eg "FINISHED.TXT". The second series of commands doesn't start until it sees that FINISHED.TXT exists.

[mgo]

I am processing the images on a Linux server, from a PHP process. Calling exec('convert and then exec('composite.
I wonder if process ends before the file is fully written, but I guess it's possible. Depends on write time, right?

[snibgo]

I don't use PHP so can't advise. The question is whether exec works asynchronously, ie whether one can start before the previous one has finished.

[mgo]

Dear Snibgo,

Thanks for helping out.

Exec works synchronously. For some reason convert process ends with success status before file is fully written.