Well, sight radius does matter a bit, but no where near as much as skill.
When it comes down to (practical) facts, yes, you 'should' be more accurate with the 4.5 XDm vs the 3.8". Whether someone shoots better with one or the other isn't tangible data unless there's a controllable factor.
And blah blah blah math blah.
I personally chose an XDm 4.5, which really does it for me. Depends what you want it for. Competition I'd say 4.5 since weight and length aren't crazy important. A sidearm/duty/OC, a 4.5 or 3.8 would both be fine. For conceal... I'd choose neither!