Performance issues after 100+ convert commands

[tricon]

Hello everybody,

I have a question regarding the performance of ImageMagick.
The system (a VPS from Amazon Web Services) I am running (the latest) ImageMagick on is: 1 Xeon E5-2650 CPU, 615MB Ram

I have created a batch file, which runs this command:
FOR /l %%x IN (0, 1, 255) DO FOR /l %%y IN (0, 1, 255) DO convert xc:rgb(0,%%x,%%y) -profile AdobeRGB1998.icc -profile CoatedFOGRA39.icc -colorspace CMYK -precision 3 -format "%%[fx:round(100*c)],%%[fx:round(100*y)],%%[fx:round(100*m)],%%[fx:round(100*k)]\n" info: >> cmyk0.txt
This gives me appropriate color conversion from rgb to cmyk (the discussion about the formula is in this post: viewtopic.php?f=1&t=25600)

The problem I ran into is that after running the command about 100-140 times (so upon reaching rgb(0,0,150) approximately) the speed of it all breaks down to *horrendously slow*.
The benchmark for this command gave me a result of 38sec to run 256 times.
So expanding the calculation would mean that converting 256*256 colors (rgb(0.0,0) -> rgb(0,255,255)) means 38sec * 256 = 2.7h
The server has been up and running the command for over a day now and imagemagick only reached rgb(0,150,50).

Things I have tried so far: setting the variable MAGICK_THREAD_LIMIT=1 (even though IM already only used 1 thread since it's a single core VPS).
Running IM in the powershell instead of cmd.
Halting the script and resuming at that point.

Does anybody have an idea for me, what to do? If the speed of IM was constant at 38sec / 256 converts, the whole thing (16.7m colors) should be done in less than 2 days.
With the current speed I am looking at a months-long process

Thanks for any ideas
Tricon

Edit: I should add this: sometimes, as I am monitoring all servers running the 1 batch file each, one of the scripts speeds up to initial speed, but only for about 30-40 converts. After that it breaks down to super-slow again. Which script/server that is, is always random. Sometimes the speed increase happens, when the script switches to the next G value of the rgb code (so from rgb(0,13,255) to rgb(0,14,0)) - but as I have observed, this seems also very random).

[snibgo]

... a VPS from Amazon Web Services ... after running the command about 100-140 times ... *horrendously slow*.

IM has nothing that would slow it down like that. I suspect your server is throttling the process.

You are starting convert, opening two profiles, and appending to a text file 16 million times. I wouldn't do that.

I would use an image size 256x256, not 1x1. Then operate on that image 256 times instead of 16 m times. True, work then has to be done. But the basic job would be something like:

Code: Select all

%IM%convert ^
  ( -size 1x256 gradient: -rotate 90 -scale "256x256^!" ) ^
  ( -size 1x256 gradient: -scale "256x256^!" ) ^
  ( -size 256x256 xc:rgb(3,3,3) ) ^
  -combine ^
  base.png

del c.txt

FOR /l %%z IN (0, 1, 255) DO @%IM%convert base.png -channel b -evaluate set %%z +channel -profile AdobeRGB1998.icc -profile CoatedFOGRA39.icc -colorspace CMYK txt:>>c.txt

This processes all 16 m pixels in 7 minutes (on my Windows 8.1 laptop). The created file, c.txt, is nearly 1 GB in size.

EDIT: Oops, I didn't copy-paste the start of the script. Sorry.

[tricon]

Thank you for the very quick answer snibgo,

could you explain to me what the command does, step by step? I tried to interpret it while reading the cli options on the IM website but I don't quite get my head around it 100%.
I mean only the parts "-channel b -evaluate set %%z +channel".

Thank you again

Edit: I have not seen your edit before writing this

[snibgo]

Yes, I doesn't make sense without the edit. Sorry about that.

The script creates an image where red varies from 0 to 255 in the x-direction, and green varies from 0 to 255 in the y-direction. The blue channel doesn't matter. I have set all blue values to rgb(3,3,3) for no good reason.

In the loop, we read the image, and set the blue channel to 0, then 1, up to 255. So in total we have had all 256*256*256 combinations.

I mean only the parts "-channel b -evaluate set %%z +channel".

-channel b: apply following operations to the blue channel only

-evaluate set %%z: set the value to %%z. This variable is set by Windows. It is the loop variable, that changes from 0 to 1 ... to 255.

+channel: apply following settings to all channels.

[tricon]

Oh, indeed, with your explenation of the edit it makes a lot more sense.
I figured you're iterating through the var set by windows but I just had not understood what exactly the -evaluate and -channel parts do, but now it makes a lot of sense.

I'll have to experiment around a bit with the output, since i need the cmyk values in the text file to be arranged in a very certain order -> it will be inserted in a database later on.
Depending on my success I might post here again with a question or a result

Thanks a TON for your support again.

[snibgo]

In the initial creation of base.png, you might want to rotate the green channel by 180 so the top is 0 and the bottom is 255. And you might want to adjust the order of the channels, so red is the last one to change, or whatever. You can just swap those 3 lines around, if you ensure that the "-channel B" changes to suit.

[tricon]

As I am currently working on it, I have a question about the -format option.
According to the documentation, I can output a specific pixel's color channel by using, for example

Code: Select all

-format "%[pixel:p{x,y}.c]

to get that pixel's cyan value, m for magenta and so on.
But I am wondering, why my output gives 3x (for rgb; 4x for cmyk) the same value for each color channel.
For example, if i read the base.png (without all the profile conversions) and do

Code: Select all

convert base.png -precision 3 -format "%[pixel:p{123,0}.g}] info: > c.txt

I get srgb(123,123,123), so 3x the green value, instead of just once.

Why is that? And do you know of a method to limit the output to just the specified color channel? I cannot seem to find anything useful.

Thank you
Tricon

[snibgo]

"pixel:" returns a colour. "fx:" returns a number.

An example colour is "srgb(123,123,123)". An example number is "123" or "0.81858".

So you might want:

Code: Select all

convert base.png -precision 3 -format "%[fx:p{123,0}.g}] info: > c.txt

or

Code: Select all

convert base.png -precision 3 -format "%[fx:100*p{123,0}.g}] info: > c.txt

[tricon]

Oh!

Now I understand the logic behind [fx:] and [pixel:] - thanks!