Post any defects you find in the released or beta versions of the ImageMagick software here. Include the ImageMagick version, OS, and any command-line required to reproduce the problem. Got a patch for a bug? Post it here.
Your process is probably not hanging, instead its probably just taking a real long time to complete. Add -monitor to your command line to monitor the progress of the image rotation operation.
You would probably would have better luck with the Q8 version of ImageMagick if you are not already using it. It uses 50% less memory than the Q16 version.
We tried your command under Fedora Core 5 on a 1GB machine with ImageMagick 6.2.6-5 Q16 and it created a 2811x2811 rotated image in 17 seconds of wall clock time. It appears our 1GB Windows XP machine is rejecting our demand for pixel cache memory where Linux is not for the same image and mogrify command.
Not a bug, but there is room to improve the algorithm. It appears for 45 degrees we are over allocating the initial rotated image size before it is cropped to the final size. We could speed up the rotation a bit by attending to this enhancement. We'll add it to the list of things to do.
We'd certainly like to hear from you if you come up with a solution to the speed problem. We designed ImageMagick to be general purpose so we had to trade-off certain optimizations to ensure support for the greatest number of image formats, colorspaces, storage classes, etc.