A digital input cannot be left floating (not connected to anything). If you are not activating the switch, it must be pulled down to LOW, or up to HIGH through a resistor. There are pullup resistors in the CPU that can be activated through the pinMode(pin, INPUT_PULLUP) command, or you can add a external resistor to the circuit.
By adding a small capacitor across the resistor, we eliminate switch bounce, as the capacitor charges during the press of the button, then slowly (from the microcontroller's point of view) discharges, enabling the microcontroller to see a single activation of the switch.
In the diagram above, if +5v is connected to the right pin, Gnd to the center, and the Signal is connected to a digital input, you have built a debounced switch in pulldown configuration, meaning the arduino will see a LOW unless the switch is pushed. If Gnd is connected to the right pin, +5v connected to the center, and the signal pin is connected to a digital input, the Arduino will read a HIGH unless the switch is pushed. The capacitor ensures only one activation is read per push.