Imagemagick won't follow HTTP Location headers

Post any defects you find in the released or beta versions of the ImageMagick software here. Include the ImageMagick version, OS, and any command-line required to reproduce the problem. Got a patch for a bug? Post it here.
Post Reply
bjn
Posts: 13
Joined: 2010-09-06T05:46:08-07:00
Authentication code: 8675308

Imagemagick won't follow HTTP Location headers

Post by bjn »

Edit: old subject was "Imagemagick fails when trying to get images from HTTPS URLs", which was incorrect -- cause of bug is now clearer.

I found recently that I can specify input images as URLs, and that works very well. But I found today that if I use an HTTPS URL Imagemagick fails.

Example (useless but shows the issue):

Code: Select all

$ convert -version
Version: ImageMagick 6.8.0-10 2013-03-03 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2013 ImageMagick Studio LLC
Features:  OpenCL

$ convert "https://graph.facebook.com/4/picture?width=177&height=113" -resize 50x50 out.jpg
convert: unable to open image `/var/tmp/magick-71628FTyx9aB5tjab': No such file or directory @ error/blob.c/OpenBlob/2638.
convert: unable to open file `/var/tmp/magick-71628FTyx9aB5tjab': No such file or directory @ error/constitute.c/ReadImage/589.
convert: no images defined `out.jpg' @ error/convert.c/ConvertImageCommand/3078.
$ convert "http://graph.facebook.com/4/picture/width=177&height=113" -resize 50x50 out.jpg
$ (success)
Last edited by bjn on 2013-09-19T14:11:01-07:00, edited 1 time in total.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Imagemagick fails when trying to get images from HTTPS U

Post by snibgo »

I've used IM to get https files before, though not often.

I note that wget succeeds for your http example, and fails for the https. I think that if wget fails, I would expect IM to also fail.
snibgo's IM pages: im.snibgo.com
bjn
Posts: 13
Joined: 2010-09-06T05:46:08-07:00
Authentication code: 8675308

Re: Imagemagick fails when trying to get images from HTTPS U

Post by bjn »

wget works for me.

Code: Select all

$ wget "https://graph.facebook.com/4/picture?width=177&height=113" -O test.jpg
--2013-09-18 22:42:01--  https://graph.facebook.com/4/picture?width=177&height=113
Resolving graph.facebook.com... 31.13.74.49, 2a03:2880:f00b:301:face:b00c::1
Connecting to graph.facebook.com|31.13.74.49|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-ash1/c0.0.160.160/p160x160/202896_4_1782288297_n.jpg [following]
--2013-09-18 22:42:02--  https://fbcdn-profile-a.akamaihd.net/hprofile-ak-ash1/c0.0.160.160/p160x160/202896_4_1782288297_n.jpg
Resolving fbcdn-profile-a.akamaihd.net... 204.93.47.49, 204.93.47.43, 204.93.47.25, ...
Connecting to fbcdn-profile-a.akamaihd.net|204.93.47.49|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4977 (4.9K) [image/jpeg]
Saving to: ‘test.jpg’

100%[============================================================================================================================================================================================================================================================================>] 4,977       --.-K/s   in 0s

2013-09-18 22:42:02 (791 MB/s) - ‘test.jpg’ saved [4977/4977]

$ identify test.jpg
test.jpg JPEG 160x160 160x160+0+0 8-bit sRGB 4.98KB 0.000u 0:00.009
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Imagemagick fails when trying to get images from HTTPS U

Post by snibgo »

Interesting. On wget 1.5.3, on Windows 7, I get:

Code: Select all

D:\web\im>wget "https://graph.facebook.com/4/picture?width=177&height=113" -O test.jpg
--15:13:17--  ftp://https:21/%2Fgraph.facebook.com/4/picture?width=177&height=11
3
           => `test.jpg'
Connecting to https:21...
https: Host not found
unlink: No such file or directory

No matches on pattern `picture?width=177&height=113'.


D:\web\im>wget "http://graph.facebook.com/4/picture?width=177&height=113" -O test.jpg
--15:17:46--  http://graph.facebook.com:80/4/picture?width=177&height=113
           => `test.jpg'
Connecting to graph.facebook.com:80... connected!
HTTP request sent, awaiting response... 302 Found
Location: http://profile.ak.fbcdn.net/hprofile-ak-ash1/c0.0.160.160/p160x160/202896_4_1782288297_n.jpg [following]
--15:17:47--  http://profile.ak.fbcdn.net:80/hprofile-ak-ash1/c0.0.160.160/p160x
160/202896_4_1782288297_n.jpg
           => `test.jpg'
Connecting to profile.ak.fbcdn.net:80... connected!
HTTP request sent, awaiting response... 200 OK
Length: 4,977 [image/jpeg]

    0K -> ....                                                   [100%]

15:17:47 (4.75 MB/s) - `test.jpg' saved [4977/4977]
Perhaps my wget is faulty.
snibgo's IM pages: im.snibgo.com
bjn
Posts: 13
Joined: 2010-09-06T05:46:08-07:00
Authentication code: 8675308

Re: Imagemagick fails when trying to get images from HTTPS U

Post by bjn »

My wget is "GNU Wget 1.14", which says it's built with "+https" and "+ssl/openssl". Maybe yours wasn't built with that.

Try that URL in a browser, or with curl or something. Do you get a picture?

The point, anyway, is that Imagemagick (the version I have at least) succeeds on HTTP but fails on HTTPS, with a confusing error message.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Imagemagick fails when trying to get images from HTTPS U

Post by snibgo »

Oh, I see what you mean, and agree IM seems to have a bug.

As a simpler example, where the http and https versions are available from a web browser:

Code: Select all

convert http://duckduckgo.com/assets/logo_homepage.normal.v102.png d.png
convert https://duckduckgo.com/assets/logo_homepage.normal.v102.png d.png
The first works; the second doesn't (tested with v6.8.6-9 on Windows 7), with the error:

Code: Select all

convert.exe: unable to open image `//duckduckgo.com/assets/logo_homepage.normal.v102.png': No such file or directory @ error/blob.c/OpenBlob/2638.
convert.exe: unable to open file `//duckduckgo.com/assets/logo_homepage.normal.v102.png' @ error/png.c/ReadPNGImage/3691.
IM mostly assumes a prefix with a colon is an image type, and I suspect is is special-casing "http:". If so, it should also special-case "https:" and probably "ftp:".

A developer would need to comment.
snibgo's IM pages: im.snibgo.com
User avatar
magick
Site Admin
Posts: 11064
Joined: 2003-05-31T11:32:55-07:00

Re: Imagemagick fails when trying to get images from HTTPS U

Post by magick »

The command
works for us.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Imagemagick fails when trying to get images from HTTPS U

Post by snibgo »

Is that on Windows? If so, that would suggest I have a Windows configuration issue.
snibgo's IM pages: im.snibgo.com
User avatar
dlemstra
Posts: 1570
Joined: 2013-05-04T15:28:54-07:00
Authentication code: 6789
Contact:

Re: Imagemagick fails when trying to get images from HTTPS U

Post by dlemstra »

It looks like a windows specific issue, I am investigating it.

EDIT:
LibXml is being used to download a file, but that does not support https. I will try to see if I can add a patch that uses internal windows methods to download a file.
.NET + ImageMagick = Magick.NET https://github.com/dlemstra/Magick.NET, @MagickNET, Donate
bjn
Posts: 13
Joined: 2010-09-06T05:46:08-07:00
Authentication code: 8675308

Re: Imagemagick fails when trying to get images from HTTPS U

Post by bjn »

I'm on Mac OS X, not Windows.
bjn
Posts: 13
Joined: 2010-09-06T05:46:08-07:00
Authentication code: 8675308

Re: Imagemagick fails when trying to get images from HTTPS U

Post by bjn »

On my Linux machine Imagemagick will load the HTTPS duckduckgo file but not the one from Facebook.

Code: Select all

<bjn@sponvoid:/tmp>$ convert https://duckduckgo.com/assets/logo_homepage.normal.v102.png d.png
<bjn@sponvoid:/tmp>$ convert "https://graph.facebook.com/4/picture?width=177&height=113" out.jpg
convert: unable to open image `/tmp/magick-dMA39w4a':  @ error/blob.c/OpenBlob/2587.
convert: unable to open file `/tmp/magick-dMA39w4a':  @ error/constitute.c/ReadImage/571.
convert: missing an image filename `out.jpg' @ error/convert.c/ConvertImageCommand/3011.
<bjn@sponvoid:/tmp>$
Looking at curl's output, which may provide clues:

Code: Select all

<bjn@sponvoid:/tmp>$ curl -v https://duckduckgo.com/assets/logo_homepage.normal.v102.png >/dev/null
* About to connect() to duckduckgo.com port 443 (#0)
*   Trying 50.18.192.250...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server key exchange (12):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using ECDHE-RSA-RC4-SHA
* Server certificate:
*        subject: serialNumber=evVwiDo-ql6zOlrQXqCb6FI6kq2HyL1z; OU=GT00431633; OU=See www.rapidssl.com/resources/cps (c)12; OU=Domain Control Validated - RapidSSL(R); CN=*.duckduckgo.com
*        start date: 2012-10-28 19:42:01 GMT
*        expire date: 2013-11-30 19:34:29 GMT
*        subjectAltName: duckduckgo.com matched
*        issuer: C=US; O=GeoTrust, Inc.; CN=RapidSSL CA
*        SSL certificate verify ok.
> GET /assets/logo_homepage.normal.v102.png HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: duckduckgo.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Thu, 19 Sep 2013 21:04:25 GMT
< Content-Type: image/png
< Content-Length: 21612
< Last-Modified: Wed, 11 Sep 2013 20:53:14 GMT
< Connection: keep-alive
< ETag: "5230d83a-546c"
< Expires: Fri, 19 Sep 2014 21:04:25 GMT
< Cache-Control: max-age=31536000
< Cache-Control: public
< Accept-Ranges: bytes
<
{ [data not shown]
100 21612  100 21612    0     0   141k      0 --:--:-- --:--:-- --:--:--  178k
* Connection #0 to host duckduckgo.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]
<bjn@sponvoid:/tmp>$
That has returned the image right away. Now the Facebook image:

Code: Select all

<bjn@sponvoid:/tmp>$ curl -v "https://graph.facebook.com/4/picture?width=177&height=113" >/dev/null
* About to connect() to graph.facebook.com port 443 (#0)
*   Trying 31.13.76.8...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server key exchange (12):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using ECDHE-RSA-AES128-SHA
* Server certificate:
*        subject: C=US; ST=California; L=Palo Alto; O=Facebook, Inc.; CN=*.facebook.com
*        start date: 2013-04-11 00:00:00 GMT
*        expire date: 2016-03-05 23:59:59 GMT
*        subjectAltName: graph.facebook.com matched
*        issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)10; CN=VeriSign Class 3 Secure Server CA - G3
*        SSL certificate verify ok.
> GET /4/picture?width=177&height=113 HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: graph.facebook.com
> Accept: */*
>
< HTTP/1.1 302 Found
< Access-Control-Allow-Origin: *
< Cache-Control: private, no-cache, no-store, must-revalidate
< Content-Type: image/jpeg
< Expires: Sat, 01 Jan 2000 00:00:00 GMT
< Location: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-ash1/c0.0.160.160/p160x160/202896_4_1782288297_n.jpg
< Pragma: no-cache
< X-FB-Rev: 941825
< X-FB-Debug: VLHMe2cQho+bfSPH/h/k8ZzpI8JqbeEWec/J1jKusOQ=
< Date: Thu, 19 Sep 2013 21:04:35 GMT
< Connection: keep-alive
< Content-Length: 0
<
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host graph.facebook.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]
<bjn@sponvoid:/tmp>$
Note that Facebook has returned a 302 redirect code and a Location header. If I switch on following Location headers:

Code: Select all

<bjn@sponvoid:/tmp>$ curl -vL "https://graph.facebook.com/4/picture?width=177&height=113" >/dev/null                                            0|14:04
* About to connect() to graph.facebook.com port 443 (#0)
*   Trying 31.13.76.8...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server key exchange (12):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using ECDHE-RSA-AES128-SHA
* Server certificate:
*        subject: C=US; ST=California; L=Palo Alto; O=Facebook, Inc.; CN=*.facebook.com
*        start date: 2013-04-11 00:00:00 GMT
*        expire date: 2016-03-05 23:59:59 GMT
*        subjectAltName: graph.facebook.com matched
*        issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)10; CN=VeriSign Class 3 Secure Server CA - G3
*        SSL certificate verify ok.
> GET /4/picture?width=177&height=113 HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: graph.facebook.com
> Accept: */*
>
< HTTP/1.1 302 Found
< Access-Control-Allow-Origin: *
< Cache-Control: private, no-cache, no-store, must-revalidate
< Content-Type: image/jpeg
< Expires: Sat, 01 Jan 2000 00:00:00 GMT
< Location: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-ash1/c0.0.160.160/p160x160/202896_4_1782288297_n.jpg
< Pragma: no-cache
< X-FB-Rev: 941825
< X-FB-Debug: RcZphz9BBJZWNyp6e8JTtcIfAwcDAEcmzNB2QECKmaw=
< Date: Thu, 19 Sep 2013 21:04:49 GMT
< Connection: keep-alive
< Content-Length: 0
<
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host graph.facebook.com left intact
* Issue another request to this URL: 'https://fbcdn-profile-a.akamaihd.net/hprofile-ak-ash1/c0.0.160.160/p160x160/202896_4_1782288297_n.jpg'
* About to connect() to fbcdn-profile-a.akamaihd.net port 443 (#1)
*   Trying 23.59.189.211... connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using AES256-SHA
* Server certificate:
*        subject: C=US; ST=MA; L=Cambridge; O=Akamai Technologies, Inc.; CN=a248.e.akamai.net
*        start date: 2013-07-24 17:49:01 GMT
*        expire date: 2013-12-31 18:47:42 GMT
*        subjectAltName: fbcdn-profile-a.akamaihd.net matched
*        issuer: C=US; O=GTE Corporation; OU=GTE CyberTrust Solutions, Inc.; CN=GTE CyberTrust Global Root
*        SSL certificate verify ok.
> GET /hprofile-ak-ash1/c0.0.160.160/p160x160/202896_4_1782288297_n.jpg HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: fbcdn-profile-a.akamaihd.net
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: image/jpeg
< Last-Modified: Fri, 05 Jul 2013 19:49:34 GMT
< Content-Length: 4977
< Access-Control-Allow-Origin: *
< Cache-Control: max-age=1209600
< Expires: Thu, 03 Oct 2013 21:04:49 GMT
< Date: Thu, 19 Sep 2013 21:04:49 GMT
< Connection: keep-alive
<
{ [data not shown]
100  4977  100  4977    0     0  41093      0 --:--:-- --:--:-- --:--:-- 41093
* Connection #1 to host fbcdn-profile-a.akamaihd.net left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]
* Closing connection #1
* SSLv3, TLS alert, Client hello (1):
} [data not shown]
<bjn@sponvoid:/tmp>$
Back to the Mac, I confirm that it can grab the HTTPS duckduckgo image just fine.

So the bug isn't to do with HTTPS, it's to do with handling redirects and Location headers.

Updated the thread subject to reflect.
User avatar
dlemstra
Posts: 1570
Joined: 2013-05-04T15:28:54-07:00
Authentication code: 6789
Contact:

Re: Imagemagick won't follow HTTP Location headers

Post by dlemstra »

You should add the -L option in your delegates.xml:

<delegate decode="https" command=""curl" -s -k -L -o "%o" "https:%M""/>

I will ask magick if we can change this in the next release. I will add support for opening https files on windows, you can expect that in the next release.
.NET + ImageMagick = Magick.NET https://github.com/dlemstra/Magick.NET, @MagickNET, Donate
Post Reply