Why height matters
A horizontal antenna's signal goes up and down as well as out. The downward part reflects off the ground and combines with the direct signal. At some angles the two add, and at others they cancel. The result depends on the antenna's height in wavelengths:
First (lowest) lobe: sin θ = λ ÷ 4h
Here h is the height, λ the wavelength and θ the elevation angle. A dipole half a wavelength up has its main lobe at 30°. At one wavelength the main lobe drops to 14.5°, which is much better for DX, and further lobes appear above it. Below about a quarter-wavelength the signal goes mostly straight up.
DX, Europe or local?
Signals that leave at a low angle travel further on each hop off the ionosphere. For long-distance DX, angles of 5° to 15° work best. For contacts across Europe from the UK, 15° to 30° is ideal, and for contacts within the UK on 80m and 40m, near-vertical incidence skywave (NVIS) at 60° to 90° fills in the gap that a high antenna skips over. That's why a low dipole on 80m works so well for UK nets, and why DXers put their 20m beams up at 15 to 20 m.
R is the Earth's radius and H the height of the reflecting layer: about 300 km for the F2 layer, which carries most HF DX, and 110 km for Sporadic E.
Limits of the model
The pattern assumes a horizontal antenna over flat, perfectly conducting ground, viewed broadside to the antenna. Real ground absorbs some of the reflected signal, which mostly reduces the very lowest angles and fills in the nulls slightly. Sloping ground, buildings and trees change things further: a hilltop or sloping ground in the direction of the DX lowers the effective angle. Use it to compare heights rather than for exact dB figures.
Questions
How high should a 40m dipole be?
For UK and European contacts, 8 to 12 m works well. For DX, as high as possible: 20 m (half a wavelength) puts the main lobe at about 30°, and higher is better still.
What is the take-off angle of a dipole at half a wavelength?
About 30°, with the signal falling to nothing straight up and at the horizon.
What height is best for NVIS?
Between about 0.1 and 0.25 wavelengths, which is 4 to 10 m on 80m. The signal then goes mostly straight up for contacts out to about 400 km.
Does the pattern apply to Yagis too?
Yes. A horizontal Yagi's vertical pattern is shaped by the same ground reflection, so the main lobe angle depends on its height in the same way.