Why does a floating digital input read random values, and what pull-up resistor value should I use?

A CMOS input is the gate of a pair of MOSFETs: effectively a capacitor of a few picofarads with a leakage current of well under a microamp. With the button open, nothing defines the voltage on that capacitor. Stray charge, capacitive coupling from nearby signals and mains hum picked up by the wire (your hand is a good antenna) move it up and down across the logic threshold.

A floating input is also bad for the chip's supply current. When the pin sits near mid-supply both input transistors conduct partially, which is why low-power designs never leave inputs undefined.

The resistor value is bounded on both sides. The low limit is the current wasted while the button is held: at 3.3 V, 1 kΩ draws 3.3 mA, 10 kΩ draws 0.33 mA and 100 kΩ draws 33 µA. The high limit comes from leakage and noise. The pin's leakage current flows through the resistor, so with a worst-case 1 µA, a 1 MΩ pull-up drops 1 V and leaves only 2.3 V on a 3.3 V system, right at a typical minimum high level of 0.7 × VDD. 100 kΩ drops just 0.1 V.

So anything from about 4.7 kΩ to 100 kΩ works. 10 kΩ is the usual default; battery-powered designs lean towards 100 kΩ or higher.

The internal pull-up is normally fine for a button on the same board. It is typically a few tens of kilohms, with a wide tolerance given in the datasheet, and it is enabled in software, for example pinMode(pin, INPUT_PULLUP) in Arduino code. It saves a part and cannot be left off the layout by accident.

Use an external resistor when the wire to the switch is long or runs near motors and relays. A weak pull-up with a long wire is easy to disturb, and the time constant grows: 50 kΩ with 1 nF of cable capacitance is 50 µs, against 4.7 µs for 4.7 kΩ. A lower value, plus a small capacitor at the pin, gives much better noise immunity.

On pull-up against pull-down: both define the level equally well, but wiring the button to ground with a pull-up is the convention. Internal pull-ups are available on almost every microcontroller while internal pull-downs are not universal, and the wire going out to the switch carries ground instead of the supply rail, so a pinched wire shorting to a metal case does no harm.

The only cost is inverted logic: pressed reads 0. Handle that once in code, for example pressed = !digitalRead(pin), and use the positive name everywhere else.

Once the level is defined you will meet the next problem, contact bounce. Mechanical contacts open and close several times over a few milliseconds when pressed, and the microcontroller is fast enough to count each one. Either ignore further changes for 10 to 20 ms after the first edge in software, or add 100 nF from the pin to ground; with a 10 kΩ pull-up that gives a 1 ms time constant on release.

The same pull-up reasoning applies to open-drain lines such as I2C, reset and interrupt outputs: those drivers can only pull low, so without a resistor nothing ever produces the high level.