Feature Request: Command line pipes other than stdin/stdout

[dmhost]

It would be really cool if the command line tools of imagemagick supported more than just the default pipes (using the "-" argument) much like command line tools such as openssl and gpg

This would be super handy to pass in many input images files using numbered pipes aka file descriptors such as 3, 4, 5 etc.

For example something like this (using a suggested notation):

Code: Select all

convert - -fd3 -fd4 -fd5 -fd6 -append output.png

I am requesting that such a command would append input from file descriptor 0, 3, 4, 5 and 6 and write it to the file output.png

What do you think? Right now to do something like this (especially with more complicated commands) we need to use temporary files.

Thanks!

[magick]

Easy enough to do. From you we need a simple test environment we can use to verify our implementation. Which command sequence can we use to read an image from file descriptor 3, for example?

[dmhost]

Wow that was a fast reply!

Here is an example using a suggested notation of -fdN where N is a number greater than 2 corresponding to the pipe.

The code demonstrating theoretical custom file descriptors/pipes is written in PHP (simply due to my ignorance of shell scripting although I'm sure somebody with a little experience could help out). Let me know if you would like any extra info.

Code: Select all

<?php

// To run this script type: php SCRIPTNAME.php

// this script should output image-output.png which in theory is the following 5 images appended using convet -append
// in a real life situation these blobs would be coming from some other source such as another process's output descriptos or maybe even a database
$inputImageBlogA = file_get_contents('image-a.png');
$inputImageBlogB = file_get_contents('image-b.png');
$inputImageBlogC = file_get_contents('image-c.png');
$inputImageBlogD = file_get_contents('image-d.png');
$inputImageBlogE = file_get_contents('image-e.png');

// set up the descriptors
$descriptors = array(
  0 => array("pipe", "r"), // this is standard input
  1 => array("pipe", "w"), // this is standard output
  2 => array("pipe", "w"), // this is standard error
  3 => array("pipe", "r"), // this our custom input pipe to feed images
  4 => array("pipe", "r"), // this our custom input pipe to feed images
  5 => array("pipe", "r"), // this our custom input pipe to feed images
  6 => array("pipe", "r")  // this our custom input pipe to feed images
);

// build the command line
$commandLine = 'convert - -fd3 -fd4 -fd5 -fd6 -append image-output.png';
$processHandle = proc_open($commandLine, $descriptors, $pipes);

if(is_resource($processHandle)) {
  // push image E to pipe 6
  fwrite($pipes[6], $inputImageBlogE);
  fclose($pipes[6]);

  // push image D to pipe 5
  fwrite($pipes[5], $inputImageBlogD);
  fclose($pipes[5]);

  // push image C to pipe 4
  fwrite($pipes[4], $inputImageBlogC);
  fclose($pipes[4]);

  // push image B to pipe 3
  fwrite($pipes[3], $inputImageBlogB);
  fclose($pipes[3]);

  // push image A to pipe 0 (standard input)
  fwrite($pipes[0], $inputImageBlogA);
  fclose($pipes[0]);

  // read standardOut
  $standardOut = '';
  while(!feof($pipes[1])) {
    $standardOut .= fgets($pipes[1], 1024);
  }
  fclose($pipes[1]);

  // read standardError
  $standardError = '';
  while(!feof($pipes[2]))    {
    $standardError .= fgets($pipes[2], 1024);
  }
  fclose($pipes[2]);

  // close the process
  $returnCode = proc_close($processHandle);
} else {
  throw new Exception("Cannot run process");
}

echo "RETURNED CODE: $returnCode\n";
echo "STANDARD OUT: $standardOut\n";
echo "STANDARD ERR: $standardError\n";

?>

[magick]

Give us a few days to apply a patch to the ImageMagick Subversion trunk. Your command will end up looking like this:

• convert - fd:3 fd:4 fd:5 fd:6 -append image-output.png

[anthony]

NOTE: You can do that already -- using BASH!!!!!

And as PHP does not provide a nice way of running your convert command without the use of a shell (arrrggghhh...) you may as well use it.

Code: Select all

   convert  /dev/fd3  /dev/fd/4  /dev/fd/5  -append  image-output.png

To find out if your 'PHP shell' is bash run

Code: Select all

<?php
  header('Content-Type: text/plain');
  system("exec 2>&1; echo shell is $SHELL");
?>

So if you are using BASH to generate the extra file descriptors than you
already have it.

Of course this does not help if you are avoiding the use of a SHELL
and in a perl script, or C programming. Also as this is probably
more to do with secure programing, it is probably a good idea.


However I would expand that idea to OUTPUT as well.

Code: Select all

   convert  rose:   -write fd:5   -swirl 90   fd:6

But it may need some method of specifing the OUTPUT format

For example

Code: Select all

   convert  rose:   -write jpeg:fd:5   -swirl 90   png:fd:6

WARNING WARNING WARNING.

While the large UNIX buffers will prevent a lot of problems with I/O deadlocks handling multiple streams of data can cause problems, unless each stream come from or goes to a completely separate parallel process. You need to be certain that IM can read all the data for each image, in the order given.

Your own script for example relies on the UNIX buffer being large enough to hold all the image data, as you write to the pipelines in the WRONG order!

[anthony]

NOTE some image file formats allow multiple images

MIFF: for example is especially good for this as you can just concatenate the MIFF image files together to pass it to IM. this makes it perfect to have a shell loop or sub-process that just generates each image and sends it down the pipeline one-after-the-other, for the main command to process.

For example in shell

for i in image1.gif image2.jpg image3.png
do
convert "$i" miff:-
done |
convert miff:- +append all_images_appened.png

Another example of this is in IM Examples, laying images
http://www.imagemagick.org/Usage/layers/#example

I also use this in the IM shell script "segment-image"
where I am splitting up an image into multiple sub-images
which still needs some extra processing to finish off the job.
http://www.imagemagick.org/Usage/scripts/segment_image

[magick]

We considered the /dev/fd devices, however, ImageMagick is cross-platform and this method does not work under Windows. Instead we use the solution stated earlier, for example, fd:3.

[anthony]

I agree... the /dev/fd devices are really for use by BASH and bash scripts, which makes it useless for more direct execution without a shell, from things like perl and C.

Though PHP does not have a simple direct execution of a command, without having that command parsed by a shell. A situation I thought silly considering that PHP is used in such a security conscious environment!

No the fd:# special format is a good general solution, and one I would also like to see used for output as well, though as mentioned the output format will still need to be settable.

[magick]

You can fd:# for output as well:

• convert fd:3 gif:fd:4

[anthony]

Great

[dmhost]

Wow I am happy to see this discussion evolve into what it is!
What a great idea to add this with outputs as well: convert fd:3 gif:fd:4

Best!

[dmhost]

Aside: anthony did you know with in windows with php's proc_open you can bypass the shell by setting a flag on the $other_options parameter?

[anthony]

I took a look at the manual and could not see that flag, However it looks like a great function for starting a background process with multiple file descriptor channels. Something that is hard to do in a shell without resorting to multiple named pipes.

However the command remands 'shell parsed', in that command and its arguments are still given as a single string, which requires some shell to divide into the argument array of the command. And that is a security hazard.

There is a 'bypass shell' but that is for windows only.

See Perl system() function, for an example. if multiple arguments are given it calls the command direct!

[Trevor]

Hi

I know this is somewhat of an old thread but I thought it would be appropriate to post my question on it.
I am calling ImageMagick from a node js environment as a child-process and trying to pass 2+ buffers as arguments into the convert process.
How can I know what, if any fd is assigned to a buffer?
Please see my original question http://stackoverflow.com/questions/3365 ... ld-process
My code below uses 1 buffer through stdin but I want all the input "files" which in fact are not files rather buffers to be passed as buffers without the need to save them as files. Over there I was suggested to use the fd:3 fd:4 but I don't know in practice how to get the real fd numbers, I wouldn't have thought that a "buffer" has an fd as it's not a file.
The code I currently have is below, I would really appreciate an example of how to get it to work without using any input files.
Thanks in Advance

Trevor

Code: Select all

var arrow1JpgBase64, arrow2JpgBase64, arrowBuffer1, arrowBuffer2, magicCommands, imagic;

spawn = require('child_process').spawn;
exec = require('child_process').exec;

arrow1JpgBase64 = "/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAUDBAQEAwUEBAQFBQUGBwwIBwcHBw8LCwkMEQ8SEhEPERETFhwXExQaFRERGCEYGh0dHx8fExciJCIeJBweHx7/2wBDAQUFBQcGBw4ICA4eFBEUHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh7/wAARCAAFAAkDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD1/wCFngjxJba78QbWT4reLTLH4pZ2mjt9PLTCSxs5VL+bbSYZUkWPCFUxGu1EHFdz/wAIb4j/AOiseM//AAF0n/5BoooA/9k=";
arrow2JpgBase64 = "/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAUDBAQEAwUEBAQFBQUGBwwIBwcHBw8LCwkMEQ8SEhEPERETFhwXExQaFRERGCEYGh0dHx8fExciJCIeJBweHx7/2wBDAQUFBQcGBw4ICA4eFBEUHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh7/wAARCAAFAAkDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD1/wCOvgjxJc/Dt7NPit4tMt1q+k20Ty2+nqsLyajbIkoMFtFJuRmDjbIpyo5xmu5/4Q3xH/0Vjxn/AOAuk/8AyDRRQB//2Q==";
arrowBuffer1 = new Buffer(arrow1JpgBase64, 'base64');
arrowBuffer2 = new Buffer(arrow2JpgBase64, 'base64');
magicCommands = ["jpg:",
                 "in_2.jpg",
                 "+append",
                 "out.jpg"];

imagic = spawn("convert", magicCommands);
imagic.stdin.write(arrowBuffer1);
imagic.stdin.end();

imagic.on('exit', function (code) {
    if (code === 0) {
        exec("open out.jpg");
    } else {
        console.log("error code: " + code);
    }
}); // end of on exit

[anthony]

all streams have a file number.
stdin is 0 stdout is 1 stderr is 2
After that streams are opened with later numbers.
you can convert file descriptors (Suchas what "stdin" really means) to there fileno

Shells use file numbers more directly for example

exec 3<some image
convert fd:3 show:
exec 3<&-

which is equivalent to the shell doing the open and close of the file, and giving the already opened file streams to the command.

As the file are opened by the shell, the shell itself could read or write extra information to those streams in addition to the convert command itself. For example dealing with multi-image file streams, or network streams that is also sending and receiving embedded images in the stream.

The original request is to get IM to not close such streams itself when it reaches the end of an image it is reading.
Of course that only works for images that have a well defined end, without relying on a actual EOF condition.
Not that in reading an concatenated multi-image streams, will need things to tell IM how many images to read before stopping!