The subject line is a comment that surface surprisingly often in the literature of folks who want to deny that climate is changing, or if it isn't, that any of it is due to human activity, or if it is, then it will be good for us. Whatever. The science of interest here is the notion of climate change, how fast, how much, how often, and the like. The human side of concern ... we'll get to a bit of that as well.
As we look through the period where we have thermometers (the last 100 or so years) recording temperatures, we see that there are indeed changes from year to year (though a year is too short really to call climate) and decade to decade (a better period to average over). But this includes the period of significant human activity (the now 6+ billion of us), so doesn't necessarily tell us a lot about what the climate does without human effects.
For longer range, say the last 600-2000 years, we have climate proxies that tell us about climate without being thermometers. Tree rings and ice cores are two such sources of proxies. When we look here, we see that climate does change decade to decade, century to century, and even millennium to millenium. The changes are small, tenths of a degree for the global average (and the longer a period you average over, the smaller the changes). But they're observable and present.
Even longer term, we still have ice cores (to 800,000 years ago), and then also start looking at marine sediments (to about 100,000,000 years ago). We see here that climate still changes, even on 10,000 to 10,000,000 year time scales. A very large change, 5 C or so global average, is associated with the northern hemisphere ice age cycle. About as much is associated with the start of the Antarctic ice cap about 35 million years ago.
The exact causes for the changes are a subject of study with some good answers and some not as confident. They include carbon dioxide (greenhouse gas) changes, orbital variations (the earth's orbit isn't exactly constant), continental drift, and land surface changes.
So what do we need to consider as humans with human interests regarding climate? As scientists, we want to understand everything, through all of time. But as citizens thinking about policy, we can look at a few things. One thing is, although there have been large changes to climate before, humans weren't around for it, certainly not 6+ billion of us, many living in large cities near the ocean. Another is, these large climate changes were also associated with large extinctions. We might not want to cause climate change sufficient to drive a large extinction. During the glacial to interglacial warming 10-18,000 years ago, global human population was likely only a few million, and were largely nomadic. So when climate got bad in one area, they simply moved (or died). Today, with 6+ billion people occupying the earth in stationary cities, it'd be difficult to move out of the way if climate got bad where you were.
Perhaps the strongest indicator of our time scale of concern is human life. If change were to be small over 70 years, we might not be concerned as little would be different between our birth and death. More quickly, we might want to ensure that there was reasonable stability between the time our kids were born and the time they reached adulthood, 20ish years. Given the two, 'a few decades' becomes our time range. Cities also point us towards the few decade time scale -- it is on this scale that urban infrastructures are built and rebuilt. If changes are slow enough that 'all' we have to do it rebuild the city further inland over the next 50 years, for instance, then relatively simple practices could get us there (but this would require some consideration on how to carry it out, as people who live now in the area marked not to be rebuilt might object!).
So, while I'm very interested as a scientist in, for instance, the 100,000 year cycles of the ice ages (my first scientific paper was on it), it's just too slow to be a concern to me as a citizen. As a citizen, I'd as soon that scientists did understand the 100,000 year cycle -- the more that is understood, the more likely that good estimates can be made of the futures. But the estimates that matter are those for the next few decades to maybe a century or two.