Legacy ImageMagick Discussions Archive

Use https://github.com/ImageMagick/ImageMagick/discussions instead.
https://imagemagick.com/discourse-server/

Imagemagick won't follow HTTP Location headers

https://imagemagick.com/discourse-server/viewtopic.php?t=24121

Page 1 of 1

Imagemagick won't follow HTTP Location headers

Posted: 2013-09-18T21:33:55-07:00
by bjn
Edit: old subject was "Imagemagick fails when trying to get images from HTTPS URLs", which was incorrect -- cause of bug is now clearer.

I found recently that I can specify input images as URLs, and that works very well. But I found today that if I use an HTTPS URL Imagemagick fails.

Example (useless but shows the issue):

Code: Select all

$ convert -version
Version: ImageMagick 6.8.0-10 2013-03-03 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2013 ImageMagick Studio LLC
Features:  OpenCL

$ convert "https://graph.facebook.com/4/picture?width=177&height=113" -resize 50x50 out.jpg
convert: unable to open image `/var/tmp/magick-71628FTyx9aB5tjab': No such file or directory @ error/blob.c/OpenBlob/2638.
convert: unable to open file `/var/tmp/magick-71628FTyx9aB5tjab': No such file or directory @ error/constitute.c/ReadImage/589.
convert: no images defined `out.jpg' @ error/convert.c/ConvertImageCommand/3078.
$ convert "http://graph.facebook.com/4/picture/width=177&height=113" -resize 50x50 out.jpg
$ (success)

Re: Imagemagick fails when trying to get images from HTTPS U

Posted: 2013-09-18T22:39:29-07:00
by snibgo
I've used IM to get https files before, though not often.

I note that wget succeeds for your http example, and fails for the https. I think that if wget fails, I would expect IM to also fail.

Re: Imagemagick fails when trying to get images from HTTPS U

Posted: 2013-09-18T22:43:21-07:00
by bjn
wget works for me.

Code: Select all

$ wget "https://graph.facebook.com/4/picture?width=177&height=113" -O test.jpg
--2013-09-18 22:42:01--  https://graph.facebook.com/4/picture?width=177&height=113
Resolving graph.facebook.com... 31.13.74.49, 2a03:2880:f00b:301:face:b00c::1
Connecting to graph.facebook.com|31.13.74.49|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-ash1/c0.0.160.160/p160x160/202896_4_1782288297_n.jpg [following]
--2013-09-18 22:42:02--  https://fbcdn-profile-a.akamaihd.net/hprofile-ak-ash1/c0.0.160.160/p160x160/202896_4_1782288297_n.jpg
Resolving fbcdn-profile-a.akamaihd.net... 204.93.47.49, 204.93.47.43, 204.93.47.25, ...
Connecting to fbcdn-profile-a.akamaihd.net|204.93.47.49|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4977 (4.9K) [image/jpeg]
Saving to: ‘test.jpg’

100%[============================================================================================================================================================================================================================================================================>] 4,977       --.-K/s   in 0s

2013-09-18 22:42:02 (791 MB/s) - ‘test.jpg’ saved [4977/4977]

$ identify test.jpg
test.jpg JPEG 160x160 160x160+0+0 8-bit sRGB 4.98KB 0.000u 0:00.009

Re: Imagemagick fails when trying to get images from HTTPS U

Posted: 2013-09-19T07:19:22-07:00
by snibgo
Interesting. On wget 1.5.3, on Windows 7, I get:

Code: Select all

D:\web\im>wget "https://graph.facebook.com/4/picture?width=177&height=113" -O test.jpg
--15:13:17--  ftp://https:21/%2Fgraph.facebook.com/4/picture?width=177&height=11
3
           => `test.jpg'
Connecting to https:21...
https: Host not found
unlink: No such file or directory

No matches on pattern `picture?width=177&height=113'.


D:\web\im>wget "http://graph.facebook.com/4/picture?width=177&height=113" -O test.jpg
--15:17:46--  http://graph.facebook.com:80/4/picture?width=177&height=113
           => `test.jpg'
Connecting to graph.facebook.com:80... connected!
HTTP request sent, awaiting response... 302 Found
Location: http://profile.ak.fbcdn.net/hprofile-ak-ash1/c0.0.160.160/p160x160/202896_4_1782288297_n.jpg [following]
--15:17:47--  http://profile.ak.fbcdn.net:80/hprofile-ak-ash1/c0.0.160.160/p160x
160/202896_4_1782288297_n.jpg
           => `test.jpg'
Connecting to profile.ak.fbcdn.net:80... connected!
HTTP request sent, awaiting response... 200 OK
Length: 4,977 [image/jpeg]

    0K -> ....                                                   [100%]

15:17:47 (4.75 MB/s) - `test.jpg' saved [4977/4977]
Perhaps my wget is faulty.

Re: Imagemagick fails when trying to get images from HTTPS U

Posted: 2013-09-19T11:05:53-07:00
by bjn
My wget is "GNU Wget 1.14", which says it's built with "+https" and "+ssl/openssl". Maybe yours wasn't built with that.

Try that URL in a browser, or with curl or something. Do you get a picture?

The point, anyway, is that Imagemagick (the version I have at least) succeeds on HTTP but fails on HTTPS, with a confusing error message.

Re: Imagemagick fails when trying to get images from HTTPS U

Posted: 2013-09-19T11:41:00-07:00
by snibgo
Oh, I see what you mean, and agree IM seems to have a bug.

As a simpler example, where the http and https versions are available from a web browser:

Code: Select all

convert http://duckduckgo.com/assets/logo_homepage.normal.v102.png d.png
convert https://duckduckgo.com/assets/logo_homepage.normal.v102.png d.png
The first works; the second doesn't (tested with v6.8.6-9 on Windows 7), with the error:

Code: Select all

convert.exe: unable to open image `//duckduckgo.com/assets/logo_homepage.normal.v102.png': No such file or directory @ error/blob.c/OpenBlob/2638.
convert.exe: unable to open file `//duckduckgo.com/assets/logo_homepage.normal.v102.png' @ error/png.c/ReadPNGImage/3691.
IM mostly assumes a prefix with a colon is an image type, and I suspect is is special-casing "http:". If so, it should also special-case "https:" and probably "ftp:".

A developer would need to comment.

Re: Imagemagick fails when trying to get images from HTTPS U

Posted: 2013-09-19T11:45:30-07:00
by magick
The command
works for us.

Re: Imagemagick fails when trying to get images from HTTPS U

Posted: 2013-09-19T12:29:42-07:00
by snibgo
Is that on Windows? If so, that would suggest I have a Windows configuration issue.

Re: Imagemagick fails when trying to get images from HTTPS U

Posted: 2013-09-19T12:41:34-07:00
by dlemstra
It looks like a windows specific issue, I am investigating it.

EDIT:
LibXml is being used to download a file, but that does not support https. I will try to see if I can add a patch that uses internal windows methods to download a file.

Re: Imagemagick fails when trying to get images from HTTPS U

Posted: 2013-09-19T14:00:37-07:00
by bjn
I'm on Mac OS X, not Windows.

Re: Imagemagick fails when trying to get images from HTTPS U

Posted: 2013-09-19T14:07:27-07:00
by bjn
On my Linux machine Imagemagick will load the HTTPS duckduckgo file but not the one from Facebook.

Code: Select all

<bjn@sponvoid:/tmp>$ convert https://duckduckgo.com/assets/logo_homepage.normal.v102.png d.png
<bjn@sponvoid:/tmp>$ convert "https://graph.facebook.com/4/picture?width=177&height=113" out.jpg
convert: unable to open image `/tmp/magick-dMA39w4a':  @ error/blob.c/OpenBlob/2587.
convert: unable to open file `/tmp/magick-dMA39w4a':  @ error/constitute.c/ReadImage/571.
convert: missing an image filename `out.jpg' @ error/convert.c/ConvertImageCommand/3011.
<bjn@sponvoid:/tmp>$
Looking at curl's output, which may provide clues:

Code: Select all

<bjn@sponvoid:/tmp>$ curl -v https://duckduckgo.com/assets/logo_homepage.normal.v102.png >/dev/null
* About to connect() to duckduckgo.com port 443 (#0)
*   Trying 50.18.192.250...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server key exchange (12):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using ECDHE-RSA-RC4-SHA
* Server certificate:
*        subject: serialNumber=evVwiDo-ql6zOlrQXqCb6FI6kq2HyL1z; OU=GT00431633; OU=See www.rapidssl.com/resources/cps (c)12; OU=Domain Control Validated - RapidSSL(R); CN=*.duckduckgo.com
*        start date: 2012-10-28 19:42:01 GMT
*        expire date: 2013-11-30 19:34:29 GMT
*        subjectAltName: duckduckgo.com matched
*        issuer: C=US; O=GeoTrust, Inc.; CN=RapidSSL CA
*        SSL certificate verify ok.
> GET /assets/logo_homepage.normal.v102.png HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: duckduckgo.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Thu, 19 Sep 2013 21:04:25 GMT
< Content-Type: image/png
< Content-Length: 21612
< Last-Modified: Wed, 11 Sep 2013 20:53:14 GMT
< Connection: keep-alive
< ETag: "5230d83a-546c"
< Expires: Fri, 19 Sep 2014 21:04:25 GMT
< Cache-Control: max-age=31536000
< Cache-Control: public
< Accept-Ranges: bytes
<
{ [data not shown]
100 21612  100 21612    0     0   141k      0 --:--:-- --:--:-- --:--:--  178k
* Connection #0 to host duckduckgo.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]
<bjn@sponvoid:/tmp>$
That has returned the image right away. Now the Facebook image:

Code: Select all

<bjn@sponvoid:/tmp>$ curl -v "https://graph.facebook.com/4/picture?width=177&height=113" >/dev/null
* About to connect() to graph.facebook.com port 443 (#0)
*   Trying 31.13.76.8...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server key exchange (12):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using ECDHE-RSA-AES128-SHA
* Server certificate:
*        subject: C=US; ST=California; L=Palo Alto; O=Facebook, Inc.; CN=*.facebook.com
*        start date: 2013-04-11 00:00:00 GMT
*        expire date: 2016-03-05 23:59:59 GMT
*        subjectAltName: graph.facebook.com matched
*        issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)10; CN=VeriSign Class 3 Secure Server CA - G3
*        SSL certificate verify ok.
> GET /4/picture?width=177&height=113 HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: graph.facebook.com
> Accept: */*
>
< HTTP/1.1 302 Found
< Access-Control-Allow-Origin: *
< Cache-Control: private, no-cache, no-store, must-revalidate
< Content-Type: image/jpeg
< Expires: Sat, 01 Jan 2000 00:00:00 GMT
< Location: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-ash1/c0.0.160.160/p160x160/202896_4_1782288297_n.jpg
< Pragma: no-cache
< X-FB-Rev: 941825
< X-FB-Debug: VLHMe2cQho+bfSPH/h/k8ZzpI8JqbeEWec/J1jKusOQ=
< Date: Thu, 19 Sep 2013 21:04:35 GMT
< Connection: keep-alive
< Content-Length: 0
<
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host graph.facebook.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]
<bjn@sponvoid:/tmp>$
Note that Facebook has returned a 302 redirect code and a Location header. If I switch on following Location headers:

Code: Select all

<bjn@sponvoid:/tmp>$ curl -vL "https://graph.facebook.com/4/picture?width=177&height=113" >/dev/null                                            0|14:04
* About to connect() to graph.facebook.com port 443 (#0)
*   Trying 31.13.76.8...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server key exchange (12):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using ECDHE-RSA-AES128-SHA
* Server certificate:
*        subject: C=US; ST=California; L=Palo Alto; O=Facebook, Inc.; CN=*.facebook.com
*        start date: 2013-04-11 00:00:00 GMT
*        expire date: 2016-03-05 23:59:59 GMT
*        subjectAltName: graph.facebook.com matched
*        issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)10; CN=VeriSign Class 3 Secure Server CA - G3
*        SSL certificate verify ok.
> GET /4/picture?width=177&height=113 HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: graph.facebook.com
> Accept: */*
>
< HTTP/1.1 302 Found
< Access-Control-Allow-Origin: *
< Cache-Control: private, no-cache, no-store, must-revalidate
< Content-Type: image/jpeg
< Expires: Sat, 01 Jan 2000 00:00:00 GMT
< Location: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-ash1/c0.0.160.160/p160x160/202896_4_1782288297_n.jpg
< Pragma: no-cache
< X-FB-Rev: 941825
< X-FB-Debug: RcZphz9BBJZWNyp6e8JTtcIfAwcDAEcmzNB2QECKmaw=
< Date: Thu, 19 Sep 2013 21:04:49 GMT
< Connection: keep-alive
< Content-Length: 0
<
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host graph.facebook.com left intact
* Issue another request to this URL: 'https://fbcdn-profile-a.akamaihd.net/hprofile-ak-ash1/c0.0.160.160/p160x160/202896_4_1782288297_n.jpg'
* About to connect() to fbcdn-profile-a.akamaihd.net port 443 (#1)
*   Trying 23.59.189.211... connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using AES256-SHA
* Server certificate:
*        subject: C=US; ST=MA; L=Cambridge; O=Akamai Technologies, Inc.; CN=a248.e.akamai.net
*        start date: 2013-07-24 17:49:01 GMT
*        expire date: 2013-12-31 18:47:42 GMT
*        subjectAltName: fbcdn-profile-a.akamaihd.net matched
*        issuer: C=US; O=GTE Corporation; OU=GTE CyberTrust Solutions, Inc.; CN=GTE CyberTrust Global Root
*        SSL certificate verify ok.
> GET /hprofile-ak-ash1/c0.0.160.160/p160x160/202896_4_1782288297_n.jpg HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: fbcdn-profile-a.akamaihd.net
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: image/jpeg
< Last-Modified: Fri, 05 Jul 2013 19:49:34 GMT
< Content-Length: 4977
< Access-Control-Allow-Origin: *
< Cache-Control: max-age=1209600
< Expires: Thu, 03 Oct 2013 21:04:49 GMT
< Date: Thu, 19 Sep 2013 21:04:49 GMT
< Connection: keep-alive
<
{ [data not shown]
100  4977  100  4977    0     0  41093      0 --:--:-- --:--:-- --:--:-- 41093
* Connection #1 to host fbcdn-profile-a.akamaihd.net left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]
* Closing connection #1
* SSLv3, TLS alert, Client hello (1):
} [data not shown]
<bjn@sponvoid:/tmp>$
Back to the Mac, I confirm that it can grab the HTTPS duckduckgo image just fine.

So the bug isn't to do with HTTPS, it's to do with handling redirects and Location headers.

Updated the thread subject to reflect.

Re: Imagemagick won't follow HTTP Location headers

Posted: 2013-09-21T03:36:09-07:00
by dlemstra
You should add the -L option in your delegates.xml:

<delegate decode="https" command=""curl" -s -k -L -o "%o" "https:%M""/>

I will ask magick if we can change this in the next release. I will add support for opening https files on windows, you can expect that in the next release.
All times are UTC-07:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited