ImageMagick Orchestrator

[Seth]

I have begun seeking a means to manage or configure ImageMagick though some kind of orchestrator rather than through bash and was wondering does such a thing exist already?

My current use case is the for a repository for emotes:
https://gitlab.com/Elypia/elypia-emotes

I am using CI and ImageMagick installed in a Docker container to execute some scripts to build export all emotes, in all colors and sizes, and some montages.

However, - the build is still awfully slow and the biggest room for improvement I feel is being smarter about what to export. Why export every emote every time if I can cache the old ones, and store a hash of each emote so if the hash has changed we know we need to re-export it, else skip? Seems like simple enough logic but it adds more code and further "complication" to what's meant to be a simple emotes repository. (If the configuration changes then invalidate the cache)

The solution? A separate smaller project which acts as a ImageMagick orchestrator, it does everything required of it and manages the work including caching and such itself but and just required a simple configuration file such as YAML to tell it the state it should be, and to produce optimized commands to execute to achieve what is required.

An orchestrator would make it simpler to reuse anything here by just specifying colors/hue-shifts, sizes, and globs, amongst other variables.

Does something for this exist already? Should I bother making it? Is it possible it such a tool may exist built in at some point instead?

[Seth]

I mocked up an example configuration for myself, however I have noticed that it's very specific for my use case, so I'm unsure if it's be easy to make this more generic in order to give it a wider use case. - It might be best for me to just make a mini one for my use case specifically but posted the suggestion here regardless.

https://gist.github.com/SethX3/dea0adad ... 7bcddfa89d

So `magick-exporter check` could verify if it's happy with the file, and any tests/checks that was defined, and `magick-exporter build` could actually export all images as defined by the file.

[snibgo]

That sounds like the standard tool "make". Give it the dependencies of what files depend on what files, and the commands that build files from other files, "make" will execute just the required commands, according to timestamps of files.

I use it to (re-)build my web pages, with the added feature that a script automatically finds the dependencies between files.

[cancerberosgx]

I have begun seeking a means to manage or configure ImageMagick though some kind of orchestrator rather than through bash and was wondering does such a thing exist already?

I also think shell scripts are not the right language for complex things. In my JavaScript IM library https://github.com/cancerberoSgx/magica I give support for template language which ends up generating a script (but instead programming the logic with bash you do it with your favorite language and generate a bit dump script).

I'm not familiar with templates in other languages, but in JS is pretty simple to use https://handlebarsjs.com/ or https://ejs.co/ - the first is logic less and the second is not so much flexible at the cost of complex templates. I think for this case something like the later makes more since since basically allows to embed any kind of language expression. Also in my case, since my API is asynchronous it allows me to have asynchronous templates. Examples:

Spherical distortion map. The interesting part is the first line, where I'm actually calling convert asynchrounsly form the template's code to obtain the input image size and then start building the sript commands using those values. See it alive in the online playground : https://cancerberosgx.github.io/demos/m ... EucG5nIn0=

Code: Select all

<% 
  var size = await inputFiles[0].size()
%>

convert -size <%=size.width+'x'+size.height%> xc:  -channel R \
  -fx 'yy=(j+.5)/h-.5; (i/w-.5)/(sqrt(1-4*yy^2))+.5' \
  -separate  +channel  sphere_lut.miff

convert -size <%=size.width+'x'+size.height%> xc:black -fill white \
  -draw 'circle <%=size.width/2%>,<%=size.height/2%> <%=size.width/2%>,0'    sphere_mask.miff

convert sphere_mask.miff \
  ( +clone -blur 0x90 -shade <%=size.width/2%>x<%=size.height/4%> -contrast-stretch 0% \
      +sigmoidal-contrast 6x50% -fill grey50 -colorize 10%  ) \
  -composite sphere_overlay.miff

convert <%=inputFiles[0].name %> -resize <%=size.width+'x'+size.height%>! sphere_lut.miff -fx 'p{ v*w, j }' \
  sphere_overlay.miff -compose HardLight  -composite \
  sphere_mask.miff -alpha off -compose CopyOpacity -composite \
  sphere_lena.png

Creating a gif (flux anim). Demo: https://cancerberosgx.github.io/demos/m ... 0uZ2lmIn0=

Code: Select all

<% 
function sequence(start, step, end){
  var a = []
  for(var i = start; i<=end; i+=step){
    a.push(i)
  }
  return a
} 
var size = '100x100'
var delay = 12
var list = sequence(0, 20, 359)
%>
# first generate a random noise image. We use miff format which is faster
convert -size <%= size %> xc: +noise Random random.miff
# copy it just to see it in the browser
convert random.miff random.gif 

# Now generate sequence of variations of it with function Sinusoid
<% list.forEach(i=>{ %>
  convert random.miff -channel G -function Sinusoid 1,<%=i%> \
    -virtual-pixel tile -blur 0x8 -auto-level \
    -separate flux_<%=i%>.miff
<%})%>
# And append them all in an anim gif:
convert -set delay <%= delay %> <%= list.map(i=>`flux_${i}.miff`).join(' ') %> flux_anim.gif

# a threshold variation of previous:
convert flux_anim.gif -threshold 70% flux_thres_anim.gif

# Another "filaments" variation:
convert flux_anim.gif  \
  -sigmoidal-contrast 30x50% -solarize 50% -auto-level \
  -set delay <%= delay %> filaments_anim.gif

# generate another list this time "ripples"
<% list.forEach(i=>{ %>
  convert random.miff -channel G \
    -function Sinusoid 1,<%=i%> \
    -virtual-pixel tile -blur 0x8 -auto-level \
    -function Sinusoid 2.5,<%=i*5%> \
    -separate +channel flux_ripples_<%=i%>.miff
<%})%>
convert -set delay <%= delay %> <%= list.map(i=>`flux_ripples_${i}.miff`).join(' ') %> flux_ripples_anim.gif

I wonder if others are using similar tools based on templates since it seems the logical technology to write and automate complex tasks without having to learn yet another programming language like script magick thing. If so which host languages / template languages are you using ? IMO, there are lots an lots of cool examples but written in bash and .bat and this somehow blocks new people to understand / learn from them. (although I'm not sure how the kind of approach I had with js templates would really help here, but at least is capable of generate the whole final script to examine)

[Seth]

I have begun seeking a means to manage or configure ImageMagick though some kind of orchestrator rather than through bash and was wondering does such a thing exist already?

I also think shell scripts are not the right language for complex things. In my JavaScript IM library https://github.com/cancerberoSgx/magica I give support for template language which ends up generating a script (but instead programming the logic with bash you do it with your favorite language and generate a bit dump script).

I'm not familiar with templates in other languages, but in JS is pretty simple to use https://handlebarsjs.com/ or https://ejs.co/ - the first is logic less and the second is not so much flexible at the cost of complex templates. I think for this case something like the later makes more since since basically allows to embed any kind of language expression. Also in my case, since my API is asynchronous it allows me to have asynchronous templates. Examples:

Spherical distortion map. The interesting part is the first line, where I'm actually calling convert asynchrounsly form the template's code to obtain the input image size and then start building the sript commands using those values. See it alive in the online playground : https://cancerberosgx.github.io/demos/m ... EucG5nIn0=

Code: Select all

<% 
  var size = await inputFiles[0].size()
%>

convert -size <%=size.width+'x'+size.height%> xc:  -channel R \
  -fx 'yy=(j+.5)/h-.5; (i/w-.5)/(sqrt(1-4*yy^2))+.5' \
  -separate  +channel  sphere_lut.miff

convert -size <%=size.width+'x'+size.height%> xc:black -fill white \
  -draw 'circle <%=size.width/2%>,<%=size.height/2%> <%=size.width/2%>,0'    sphere_mask.miff

convert sphere_mask.miff \
  ( +clone -blur 0x90 -shade <%=size.width/2%>x<%=size.height/4%> -contrast-stretch 0% \
      +sigmoidal-contrast 6x50% -fill grey50 -colorize 10%  ) \
  -composite sphere_overlay.miff

convert <%=inputFiles[0].name %> -resize <%=size.width+'x'+size.height%>! sphere_lut.miff -fx 'p{ v*w, j }' \
  sphere_overlay.miff -compose HardLight  -composite \
  sphere_mask.miff -alpha off -compose CopyOpacity -composite \
  sphere_lena.png

Creating a gif (flux anim). Demo: https://cancerberosgx.github.io/demos/m ... 0uZ2lmIn0=

Code: Select all

<% 
function sequence(start, step, end){
  var a = []
  for(var i = start; i<=end; i+=step){
    a.push(i)
  }
  return a
} 
var size = '100x100'
var delay = 12
var list = sequence(0, 20, 359)
%>
# first generate a random noise image. We use miff format which is faster
convert -size <%= size %> xc: +noise Random random.miff
# copy it just to see it in the browser
convert random.miff random.gif 

# Now generate sequence of variations of it with function Sinusoid
<% list.forEach(i=>{ %>
  convert random.miff -channel G -function Sinusoid 1,<%=i%> \
    -virtual-pixel tile -blur 0x8 -auto-level \
    -separate flux_<%=i%>.miff
<%})%>
# And append them all in an anim gif:
convert -set delay <%= delay %> <%= list.map(i=>`flux_${i}.miff`).join(' ') %> flux_anim.gif

# a threshold variation of previous:
convert flux_anim.gif -threshold 70% flux_thres_anim.gif

# Another "filaments" variation:
convert flux_anim.gif  \
  -sigmoidal-contrast 30x50% -solarize 50% -auto-level \
  -set delay <%= delay %> filaments_anim.gif

# generate another list this time "ripples"
<% list.forEach(i=>{ %>
  convert random.miff -channel G \
    -function Sinusoid 1,<%=i%> \
    -virtual-pixel tile -blur 0x8 -auto-level \
    -function Sinusoid 2.5,<%=i*5%> \
    -separate +channel flux_ripples_<%=i%>.miff
<%})%>
convert -set delay <%= delay %> <%= list.map(i=>`flux_ripples_${i}.miff`).join(' ') %> flux_ripples_anim.gif

I wonder if others are using similar tools based on templates since it seems the logical technology to write and automate complex tasks without having to learn yet another programming language like script magick thing. If so which host languages / template languages are you using ? IMO, there are lots an lots of cool examples but written in bash and .bat and this somehow blocks new people to understand / learn from them. (although I'm not sure how the kind of approach I had with js templates would really help here, but at least is capable of generate the whole final script to examine)

Thanks for responding, I was actually hoping to avoid logical language at all. Perhaps I'm getting too use to declarative configuration but I was hoping to take on that approach. Rather than specifying actions, specify the result and have the tool perform the actions on your behalf. Functionality may be a bit limited but so long as it can cover some generic use cases I'd be happy with the result.

Currently my configuration is just pure bash - I hadn't actually considered a templating approach however also don't consider it quite simple enough as I'd like it to be an abstraction that hides almost all code/commands away from the images. It may be less powerful, and expose less of the API, but that can that be improved as requirements expand or if someone else bothers to use it.

Just after making a small project which will be easy to make and manage, easy to configure through probably YAML, and easy to deploy in a Docker container for use in CI. Not to mention a templating thing won't be enough to provide out of the box logic for caching images already exported to avoid exporting them again and such, which I'm pretty sure with this my build times could drop quite a lot.