It isn't a margin.
The pointsize of a font actually has nothing to do with the look or the size of the drawn text in the font. You can have a 100 point font and have text that is barely readable!!!! It would not be a very good or useful font, but it is a perfectally legal font.
The pointsize of a font defined the distande from one line of that font to the next line of the font on a page. The draw size, within that can can be anything at all!!!!
The size of the text in that space is purely up to the manufacturer of the font.
See Resolution, Pointsize, and Actual Font Size
http://imagemagick.org/Usage/text/#pointsize
If you want to control the font more you will have to somehow read the other attibutes of the font. Specifically the fonts accent and decend dimentions (draw area above and below the baseline), and adjust the point size until you get want you want.
You can read these attibutes in a ImageMagcik API like PerlMagick.
However even then the size of the drawn object or shape does not have to follow the reported attributes!!!! It could exceed those dimentions or be very small within that drawing area. Think for example of a 'period' character.
The only way to find out the real drawn area is to draw the font.
Examples of doing this is are in IM Examples...
http://imagemagick.org/Usage/text/#font_info
Basically how well a font 'behaves' itself depends on how good the font maker is. Special Shape or 'dingbat' type fonts for example are notoriously bad as the maker usually just wants the shape and does not take the time to get the attributes correct.