Where do you suppose that the page (regardless of what is on it) is going to come from, when the browser requests it? Yep, the web server. You can't get around web based unless you ditch the browser.
Now, that doesn't mean that the Arduino necessarily needs to wear an ethernet shield. You could, instead, connect the Arduino to the server using the USB cable. Then, the server can tell the Arduino what to do, via the serial port.
And, no, the server can not tell your computer to tell the Arduino what to do.
There are plenty of options for building GUIs that do not require a browser to do the rendering.