I refined the drawing and added some shading which helped a lot.
Now this actually looks more like the photo. I probably will not shade the rest of the body, maybe the hands when I get all the way down there, but the portrait definitely needed some help and this worked.