donmarkey speaks the truth. A drill chuck doesn't hold an end mill nearly as solid as a collet, and carbide endmills used this way never last long, they break. (I know that you never mentioned carbide, I'm just sayin').
You could try it, and we may all be surprised at your good results, but, sadly, history is not on your side.
I do, however, use my hand drill as a sort of a crude lathe. I place anything cylindrical in the chuck and clamp the drill in my bench vise, and have at it with a file. As I said, crude but effective.
I have on occasion clamped a lathe bit in my milling machine vise, and used the spindle of the mill like the headstock of a lathe. It works OK for simple OD turning, but no substitute for a nice (thread cutting) lathe. Man, I gotta get one of them!
Manitou