PDF to PNG - Depth vs Channel depth and Colorspace vs Type

[BenjaminKim]

Please note - for all below examples, I'm using Version: ImageMagick 7.0.8-40 Q16 x64 2019-04-13

When I convert testpdf_grays.pdf to png:

i.) If I use the following command, the resulting PNG is saved as "Grayscale".

magick convert -density 72 "testpdf_grays.pdf" -background white -flatten testpdf_grays.png

I get the warning: convert: profile 'icc': 'RGB ': RGB color space not permitted on grayscale PNG `testpdf_grays.png' @ warning/png.c/MagickPNGWarningHandler/1747


Ii.) If I use the following command, the resulting PNG is "Colorspace: sRGB" and Type:"Grayscale"

magick convert -density 72 "testpdf_grays.pdf" -background white -flatten PNG24:testpdf_grays24.png

iii.) If I use the following command to create a PNG that's "Colorspace: sRGB" and "Type: GrayscaleAlpha"

magick convert -density 72 "testpdf_grays.pdf" -background white -flatten PNG32:testpdf_grays32.png

The overall "depth" settings in PNG are:
Depth: 8-bit
Channel depth:
Red: 8-bit
Green: 8-bit
Blue: 8-bit
Alpha: 1-bit

However, PDF is:
Depth: 16/8-bit
Channel depth:
Red: 8-bit
Green: 8-bit
Blue: 8-bit
Alpha: 8-bit


Iv.) If I use the following command to create a PNG that's "Colorspace sRGB and "Type: GrayscaleAlpha":

magick convert -density 72 "testpdf_grays.pdf" -background white -flatten PNG64:testpdf_grays64.png

The overall "depth" settings in PNG are:
Depth: 16-bit
Channel depth:
Red: 16-bit
Green: 16-bit
Blue: 16-bit
Alpha: 1-bit

However, PDF is:
Depth: 16/8-bit
Channel depth:
Red: 8-bit
Green: 8-bit
Blue: 8-bit
Alpha: 8-bit

Furthermore - if I do an RGB sampling of the "Top Secret" gray text on the image, it is "203,203,203" in this one image (testpdf_grays64.png), but "202,202,202" in all other versions mentioned above (pdf and pngs).

Question 1: Why is the RGB value changing on the PNG64 iteration?

Question 2: Is it possible to convert the "testpdf_grays.pdf" PDF to a PNG that has bit depths that match the PDF? That is, as follows:
Depth: 16/8-bit
Channel depth:
Red: 8-bit
Green: 8-bit
Blue: 8-bit
Alpha: 8-bit

Question 3: Is the command I'm using to generate the png32 PNG "good enough" to match original PDF?
magick convert -density 72 "testpdf_grays.pdf" -background white -flatten PNG32:testpdf_grays32.png? The resulting PNG has depths:
Depth: 8-bit
Channel depth:
Red: 8-bit
Green: 8-bit
Blue: 8-bit
Alpha: 1-bit

Ultimately, I need to ensure that what's "converted" to PNG is as accurate as possible, in regards to "color", and most importantly, "brightness" / "lightness", as the original PDF. The sample PDF appears "gray", however, it's sRGB, and need to account for PDFs that actually have color (for example, testpdf_colors.pdf). And perhaps PNG isn't the best destination format for such a task - if anyone has recommendations re better destination format, then I'm all ears.


All files mentioned above can be found here - along with identify -verbose output.

https://www.dropbox.com/sh/b8ntq5eydy3s ... KlYYa?dl=0

(some of the png files don't look correct when rendering via browser window - not sure why that is. If one downloads a file, and then views, it looks alright. Not too concerned about this.)

[edited to include version]

[fmw42]

Please always provide your IM version and platform. Also in Imagemagick 7, use magick, not magick convert.

[snibgo]

Don't worry about the warning "RGB color space not permitted on grayscale PNG".

if I do an RGB sampling of the "Top Secret" gray text on the image, it is "203,203,203" in this one image (testpdf_grays64.png),

Your image has 16 bits/channel/pixel, so values are from 0 to 65535. Your sampling software rounds this to 8 bits, ie out of 0 to 255. Rounding could account from a difference between 202 and 203 (out of 255).

However, PDF is:
Depth: 16/8-bit
etc

I suppose you mean "When IM reads the PDF the depths are ...".

Is it possible to convert the "testpdf_grays.pdf" PDF to a PNG that has bit depths that match the PDF?

You are flattening the image, which makes all pixels opaque, so IM optimises the file storage by storing alpha in 1 bit.

Is the command I'm using to generate the png32 PNG "good enough" to match original PDF?

What is "good enough"?

Ultimately, I need to ensure that what's "converted" to PNG is as accurate as possible, in regards to "color", and most importantly, "brightness" / "lightness", as the original PDF.

The PDF stores the image as vectors, which have no fading between foreground black text and the background. The raster image made by IM does have this fading, called "anti-aliasing". So the raster image is inherently inaccurate.

Aside from that, the PNG64 colours are accurate.

[BenjaminKim]

if I do an RGB sampling of the "Top Secret" gray text on the image, it is "203,203,203" in this one image (testpdf_grays64.png),

Your image has 16 bits/channel/pixel, so values are from 0 to 65535. Your sampling software rounds this to 8 bits, ie out of 0 to 255. Rounding could account from a difference between 202 and 203 (out of 255).

Ultimately, I need to ensure that what's "converted" to PNG is as accurate as possible, in regards to "color", and most importantly, "brightness" / "lightness", as the original PDF.

The PDF stores the image as vectors, which have no fading between foreground black text and the background. The raster image made by IM does have this fading, called "anti-aliasing". So the raster image is inherently inaccurate.

Aside from that, the PNG64 colours are accurate.
[/quote]

So, PNG64 is my best bet for accurate colors, though when I tried it, the color sampling from the most important part of the document was off.

I raise these questions because my ultimate goal is to poll lots and lots of PDFs, and check them to see if they have any of that "Top Secret" stamp on them. That document is just one example - but it demonstrates what I need to be able to do. I need to identify if anything in a PDF has objects or text that is lighter than a certain threshold. For instance - for this example, the RGB is 202,202,202 and it will constitute a problem because it is "too" light"

Anything more than 67% bright will be too bright. However, though, if I'm analyzing a document that has (rgb 170,170,170 ) and the rendered PNG64 is showing it as "171,171,171", then it would get flagged, even though it wasn't supposed to.

Ideally, Iike to try and do this by analyzing the PDF along - and not having to out put a file. In such a case, I can avoid having to convert. Of course, for problem docs (the problem I'm polling a lot of PDF for) - I may want to end up generating a copy of the image that meets criteria, as flag the document in the report.

The problem I'm having now is trying to analyze the RGBA data. If I can do that, then I think I got it.


Thanks a million for your feedback and assistance.

[snibgo]

... and check them to see if they have any of that "Top Secret" stamp on them.

Ah, well. The "Top Secret" stamp isn't plain text in the PDF file, and pdftotext can't see it.

testpdf_colors has a similar stamp "SAMPLE". Do you need to distinguish this stamp?

The stamps have opacity around 50%. So does anti-aliasing around the small text, so we can turn off anti-aliasing.

Code: Select all

magick +antialias testpdf_colors.pdf -alpha extract -solarize 50% -evaluate Multiply 2 -threshold 90% testpdf_stamp.png

The mean value of testpdf_stamp.png will give the proportion of pixels that had an opacity around 50%. This is around zero when there is no stamp, and 0.03796 for this "SAMPLE" image.

Or you could rotate that, blur and threshold to remove the small text, and OCR the result. That will tell you if the stamp is "Top Secret" or "SAMPLE".

[BenjaminKim]

Thank you so much - this is extremely helpful. I'm still working with your feedback, and ensuring I fully understand before asking any further questions.

Re the Top Secret vs Sample - I had initially been focusing on this from a "gray" angle (the "Top Secret", even though I realize it is technically sRGB), and then realized I had to account for color as well. The source PDFs could be of anything, and I've compiled a relatively large library of the various possible objects and combinations of object possible in a PDF. I tried to break it down to the core question(s) I had when posting and then try and expand on any insight gained.

The sample you've so graciously provided is incredibly helpful. I had actually explored "solarize" a bit, but got away from it - mostly just because of how overwhelmed I was with everything the first couple of weeks. I'm slowly, but surely, coming along, though - all thanks to such helpful folks such as yourself. I hate asking for help, but I realized after a while that I was just going in circles. So, again, thank you.

[BenjaminKim]

This works absolutely brilliantly. Thank you! I'm going to kick the tires on it some more.

I've also been trying to figure out something fmw42 mentioned in another thread, when I first realized I need to account for color.

https://www.imagemagick.org/discourse-s ... 30#p165686

"You can look for white in the product of B * (negate of S) in HSB. White will be bright and low saturation (or high values in the negate of S) image. Thus you can threshold that product image.."

I'm assuming this means take channel B in HSB and x (times it) by (negate of channel S). Am I misreading? I've been looking all over to figure out how to get the product of two channels - I'm assuming this means multiply the values together (in a math sense, not an imaging sense) - and the result will help identify lighter colors.

I'm interested in " product of B * (negate of S) in HSB" because the source PDFs may not have an alpha layer (or maybe all PDFs have alpha layers?).

The latest post from snibgo is outstanding, so far - just need to run it against more tests and tweak - and in particular, see if I can get a PDF without an alpha channel.

Once I saw "look for white in the product of B * (negate of S) in HSB", I was hooked, and been trying to figure it out ever since. I've found examples where maths is done against certain channels - but not sure how to get the product of two channels B and (negate of S).

[snibgo]

To make values between 45 and 55% into white and the others to black, I suggested ...

-solarize 50% -evaluate Multiply 2 -threshold 90%

... but in v7.0.8-10 onwards, this can be done in a single operation:

Code: Select all

-range-threshold 45,45,55,55%

My version of IM is older, so I can't test that.


To read the stamped text:

Code: Select all

magick +antialias testpdf_colors.pdf -alpha extract -solarize 50% -evaluate Multiply 2 -threshold 90% -background Black -rotate 45 -blur 0x5 -threshold 25% -resize 25% t.png
tesseract t.png x

The file x.txt then contains:

Code: Select all

SAMPLE

I'm assuming this means take channel B in HSB and x (times it) by (negate of channel S). Am I misreading? I've been looking all over to figure out how to get the product of two channels - I'm assuming this means multiply the values together (in a math sense, not an imaging sense) - and the result will help identify lighter colors.

"-compose Multiply -composite" multiplies two images, pixel by pixel. So (untested, Windows syntax):

Code: Select all

colorspace HSB -channel GB separate \
( -clone 0 -negate ) \
-delete 1 \
-compose Multiply -composite

[snibgo]

Testing my commands on your testpdf_grays.pdf results in the text file:

Code: Select all

TOP SECRET

My command relies on the stamp being the only feature in the PDF that is written with alpha between 45% and 55%, ie semi-transparent. If your PDFs may contain stamps that are opaque, you will need something else to isolate the stamp.

This is a common task in image processing: find some visual characteristic that isolates the feature we want.

[BenjaminKim]

This is a common task in image processing: find some visual characteristic that isolates the feature we want.

Yes, indeed. It is really fascinating. I never fully understood just how much could be done with imagemagick - it's incredible - opened a whole new world. Once I'm done with this latest adventure, I'm going to write up a screenshot batch file. And then work on a way to also convert to animated GIFs from screenshots, as needed (unrelated to current challenge).

Thank you again for the assistance. You definitely got me on the right track - and look forward to really putting it through the paces over the weekend against test documents, and then tweaking as needed. I'll post whatever I have once it's in final form.