There's lots of evidence right under our noses that some fairly large things have come this way in the past, but you do have to know what to look for. Although it's estimated that over 100 million meteorites come in through the atmosphere every single day, the vast majority are pretty tiny and burn up long before they can hit the ground - we see these ones as shooting stars in the night sky. Some, however, are just to big to burn away completely, and the results are spectacular.
The Moon's Cratered Surface
Just one look at the Moon through a telescope - or even a pair of binoculars - will tell you a great deal about the kind of 'asteroid weather' we've had in this part of the solar system in the last few billion years. Some of the craters on the Moon are still quite fresh, appearing much brighter than some of the older ones. It's true that craters will erode a great deal more slowly on the Moon, of course, because there's no weathering there. Most of the craters on Earth have long since been lost because of various types of erosion.
There is a very old Chinese record of an observation of a strange bloom coming off the moon way back in the 13th Century. The position on the Moon's disk (just around one side of it) was also recorded, and sure enough, when one of the first space probes sent around the Moon was taking pictures, there was a very fresh and quite large crater there. This is reckoned to be the first live viewing of a meteorite impact on a heavenly body.
Meteor Crater, Arizona, USA
About half a billion years ago, it's reckoned, there was a collision between two large bodies in the Asteroid Belt - maybe a comet was involved, or maybe a large asteroid broke up all by itself. One of the resulting chunks of rock was pushed into a new trajectory that would take it out of the Asteroid Belt and into the orbits of the inner planets.
Now scroll forward a few hundred million years...
About 50,000 years ago, this asteroid turned up again, although no one at the time would have (a) known what it was, or (b) had time to say more than 'Wow!' as it hit Earth's atmosphere, screamed down through it, and landed in what is now Arizona. This particular rock was a nickel-iron meteorite, and from the size of the crater it left, astronomers have calculated it to have been about 50m across, weighing several hundred thousand tonnes. It was so large that it is thought to have lost virtually none of its mass or velocity as it came through the upper atmosphere at something like 30,000 to 40,000 miles per hour.
When an asteroid hits Earth, all the energy that was once tied up in its velocity is turned into heat, sound and a propensity to move rather a lot of the Earth in front of it out of the way, leaving a dish-shaped crater in the surface. It's usual to think of the energy that is expended in this way by comparing it to an equivalent mass of TNT (or dynamite). Three basic factors go into calculating this energy - the speed and angle at which the meteorite hits, the mass of the meteorite, and its composition.
The numbers for this particular meteorite are quite impressive. It impacted the surface with the equivalent of about 20 million tonnes of TNT, and at the moment of impact, it raised the local pressure to about 20 million pounds per square inch, and raised the temperature enough to simply vaporise much of the underlying rock and much of the meteorite itself. The parts of the surface that didn't melt (about 175 million tonnes of limestone) were thrown for up to two kilometres across the surrounding landscape. Some small particles of graphite trapped in the limstone matrix were instantly transformed by the temperature and pressure into diamonds. The resulting crater is over 215m deep (which would comfortably house a 60-storey skyscraper) and about 1.2km across - nearly a mile. You could, theoretically, host 20 football games simultaneously on the floor of the crater, and seat over two million people around the sloping walls. It's big.
During the last century, many scientific tests were carried out on the material left behind my the impact, and two important new minerals were discovered: coesite and stishovite, which are both forms of silicon dioxide that are only created under phenomenal temperature and pressure. These two minerals have been found at many asteroid impact sites since, and are used as common pointers to likely meteoritic crater candidates. Similarly, the element iridium is commonly found in quite high concentrations in meteoritic iron, and is fortunately quite rare on Earth itself, so if you find high levels of iridium in your large hole in the ground, that's also a good pointer to a meteoritic origin.
The Tunguska Incident
A remote region in northern Siberia in Russia was rocked in 1908 by a very large explosion whose origins and cause were unknown at the time. After much scratching of heads, the only suggestion that made any sense was that a meteorite had collided with Earth's orbit and exploded somewhere low in the atmosphere, flattening and burning all the trees in the region and vapourising quite a lot of reindeer. No fragements were ever found, and it looks like it never actally got to the surface - there wasn't even a crater, but the surrounding devastation was obvious.
The L'Aigle Incident
On April 26, 1803, the residents of L'Aigle, a small town in France, were treated to a spectacular and probably very frightening display of meteorites that had not been recorded before or since. According to all reports, several large rocks came in through the atmosphere at a very oblique angle and broke up into thousands of smaller meteorites, with a thunderous noise and a great deal of fiery exploding. The incident left hundreds of rocks, some of them up to half a metre across, strewn across the landscape.
Other Craters on Earth
Now that geologists and astronomers knew what to look for, they began to find impact sites all over the place. These range in size from a few yards across, to just under a mile for the Arizona Meteor Crater described above. Some are quite round, indicating a vertical passage through the atmosphere, and some are elongated, indicating that the offending rock came through the atmosphere at an angle and ploughed more of a furrow than a crater. Some desert areas of Australia and America have a fairly liberal scattering of small black rocks that are the remains of meteor cores that have mostly burnt up upon entry.
All of our Dinosaurs are Missing...
When archaeologists first discovered dinosaur fossils, the thought immediately following the obvious 'Gosh - I wonder what they were like' was probably 'Hmmm - I wonder where they all went.' It seemed pretty obvious that they all died out over time, and as more was discovered about these fossils - their age, their locations, and so on - it became clearer that they had all died out at pretty much the same time (about 60 million years ago). What could have done that? A plague? A giant flood? Disappearing food supplies, for whatever reason?
One of the most striking theories1 put forward recently is that all of these things did happen, but that they were all caused by one thing: a meteor strike. A large enough chunk of rock hitting the Earth would throw millions of tonnes of rock and dust into the atmosphere, blocking out the Sun for years and killing off a great deal of flora, together with the fauna that fed on it (and in turn the fauna which fed on that). Such an impact would also cause catastrophic damage to the Earth's crust, with earthquakes and volcanic action that would destroy much of the surface with fire, lava, floods and tsunamis - generally making it fairly hard to survive, especially if you were big and lumbering.
This seems like quite a good explanation, but initially it was not readily accepted, partly because it seemed rather too out-of-the-ordinary, but mostly because there was no good evidence: no huge gaping crater, no layers of deposited debris - nothing that would directly scream out 'meteor!' Part of this acceptance problem was that astronomers and geologists didn't really know what a meteor strike of this size would look like today, after tens of millions of years erosion. In fact it was only when technology got good enough for us (a) to be able to search the Earth's surface from the good vantage point of space, and (b) to know what elements and other tell-tale physical signs to look for, that anyone could point to a place on Earth and put it forward as a good candidate.
After studying the other readily-visible craters and searching around Earth on a large scale, there was only one location that could possibly be suitable as the crater: a region to the east of Central America. Because this was largely under the Caribbean Sea, it wasn't until a modern underwater surface scan was performed that geologists saw the tell-tale concentric mountain rings created by the impact shock-wave. There was also quite a concentration of iridium in the surrounding area, which helped the theory. This crater - now called Chicxulub - was then positively identified as the footprint of the dinosaur-killer. Of course, we can't be sure that this is definitely what happened, but all the evidence seems to point that way.