After fiddling some time on this, my take is, this is not a vcl-styles bug at all. This indeed is related with the behavior in the article mentioned in a comment to the question by mghie.
The specific behavior is that, the size of a maximized window is larger than the work area of the monitor that the window is maximized on. Supposedly, the window manager hides the overhang borders. Apparently, it doesn't quite do so with customized frames. Note that MSDN's own custom window frame example seems to suffer from the same problem (refer to the post titled "Bug when window is Maximized " in community content). VCL's application is different than the MSDN example in that it's not based on DWM, but I still think it is the same issue.
The overhang borders have the size of the system sizing border (SM_C[X|Y]SIZEFRAME), but this is irrelevant to the workaround below as it disregards the OS suggested size/position and uses the work area.
Unfortunately I don't think this workaround is usable at all. For one, the mentioned behavior is not documented, for two, the workaround is not perfect; there's still an odd pixel out. If you snap the window exactly on the work area, the window manager decides to offset the window to where it thinks the window (with hidden frames) should be placed. (The VCL could probably be modified to do what the window manager does, and take into account overhang and do not draw them or something similar, but it would be more work and it still would be to workaround undocumented behavior..)
TForm1 = class(TForm)
// overriding styles is not necessary since TFormStyleHook.WMGetMinMaxInfo
// first calls the default window procedure
procedure WMGetMinMaxInfo(var Message: TWMGetMinMaxInfo);
procedure TForm1.WMGetMinMaxInfo(var Message: TWMGetMinMaxInfo);
// always arrives with MinMaxInfo.ptMaxPosition = (-SM_CXFRAME, -SM_CYFRAME)
// and MinMaxInfo.ptMaxSize = (PrimaryMonitor.Width (?) + 2 * SM_CXFRAME, ... )
// should test for OS, styles etc. before running the below
R := Monitor.WorkareaRect;
InflateRect(R, -1, -1); // odd pixel
OffsetRect(R, -Monitor.Left, -Monitor.Top);
Message.MinMaxInfo.ptMaxPosition := R.TopLeft;
Message.MinMaxInfo.ptMaxSize := Point(R.Width, R.Height);