Mysterious “Parameter is not Valid” error

Well, just has a really weird “Parameter is not valid” ArgumentException exception being throw when trying to create a new TextBox.
e.g. Dim box As New TextBox()
Nothing weird about that you might think! The error was coming from down below, somewhere in the Font.GetHeight method.
Turned out that I was trying to be helpful in another part [...]