Indiana Temperature data

Saturday is when I look at State Temperature data, and after looking at Illinois last week, I thought Indiana might be a reasonable next choice, moving East after the sojourn out West that started the series. Having found, as with Missouri, little correlation of temperature with time, this week’s hypothesis is that the Central States are less sensitive to changes in temperature over time, than the rest of the country. So let’s see if it is true. (For those new to this I try and pose a hypothesis at the start of the post, and then, writing the post as I run the calculations, find out if it is true. Some are, some aren’t). (And for those who have been reading these for a while there was a significant surprise this week – see if you spot it – grin - it proves I write these as I do them).

It is a little irritating to note that the temperature values I am using are not actually raw data, but as Dr. Hansen noted recently, the USHCN data is already adjusted for urban heat island effects.

We use the unadjusted version of GHCN. However, note that a subset of GHCN, the United States Historical Climatology Network (USHCN), has been adjusted via a homogenization intended to remove urban warming and other artifacts [Karl et al., 1990; Peterson and Vose, 1997].

It is an interesting note that this current study shows that the effect of population size on the data remains present, in many states, and we will now see if it holds true in Indiana. But at some stage I suppose I will (once I get sufficient states under my belt to be credible) have to see about the raw data issue.


So starting off for Indiana with the USHCN Network for that state, there are 36 stations so there will be a slight pause while I create the spreadsheet and import the data. Oh, and our first surprise of the day. The web site for the data has been reconfigured, so that the new page (this is going for Angola, which is the second USHCN site in Indiana) carries a number of options, rather than just the one set of adjusted data that I had been using until now:


So the question is whether we want the fully adjusted data, the raw data, or the time of observation adjusted data. Well I should run a series of tests to see what the difference between the sets turns out to be – but 36 stations is a lot of data to load, so I will probably do that at a later date for a smaller state, and for now use the data set that is merely adjusted for time of observation bias (The TOBS data set). So let’s get those into the spreadsheet.

And to remind you how I am doing that, you select the middle file option to download:


Click on that brings up this screen:


Select the average mean temperature (TOBS) and click the submit button at the bottom.

This brings up the download page:


Click on the blue line, and this downloads the file, which I then save as Angola Ann Temps on my Desktop, since this makes it easier to then find, and download the data into the spreadsheet. (See Missouri file for how I do that. The states I have covered already are in a list at the very bottom of the right-hand column on the site).

Well, having done that I discover, using Angola for the review, that the RAW and TOBS data sets have years missing, and that it is only the fully adjusted file that has the data for every year. Tsk! Well, had I more time today I would do a little adjusting of my own, but since I don’t, I will set that to one side for today, and go back to using the adjusted mean values for today. (Too many other things to do). Well at least I caught this early).

But just to give an example of the differences, here are the RAW, TOBS and adjusted data for the first 12 years of each data set. (There is then a gap of several years in the RAW and TOBS files – though not the ADJ file).


You will notice that the RAW and TOBS files have several temperatures in the 50’s, while the ADJ file does not. Hmm! The average RAW data temperature is 48.7 deg; the average TOBS temperature is 49.4 degrees: the average ADJ temperature is 46.3 deg. Well how does that compare with more recent temperatures – let’s look at the most recent 12 years:


You can see that the ADJ values (the ones that are used by GISS) are much closer to the RAW and TOBS values, with the average RAW temperature being 48.3 deg F, the TOBS value being 48.8 def F, and the ADJ data being 48.7 deg F. Now if I were tempted to be suspicious it would seem a little odd that the adjustment to get the used data lowers the historic temperatures while leaving current values more alone. But to substantiate that I would need to look at a lot more data than I have time to examine today, but it does cast some slight suspicion on the data as I move forward, for now using the Adjusted data.

So now, we move back to getting the data, and that means re-acquiring Anderson as an Adjusted Mean temperature. And so we get the data from the USHCN site, now carefully marking it as adjusted, and go to get the GISS data. There are four GISS stations in Indiana, according to Chiefio and these are in Evansville, Indianapolis, Fort Wayne and South Bend.

There is no data for 1895 or 1896 for Evansville and Fort Wayne, and South Bend data does not start until 1944. In the past I have interpolated data, or used adjacent sites to generate those numbers. But this time I will just average the value for the GISS stations based on the number of stations in that one year. Doing that clearly shows that the number is small enough, that each has a significant impact. Consider the difference between the GISS data and the USHCN data:


The change when the fourth GISS station is added is clear. Looking at the two segments of the curve, however, it is also clear that there is an increasing difference between the GISS stations and the USHCN stations, in both. This would indicate (and remember that the cities GISS chose are some of the largest in the state) that they are warming more than the average for the state.

For the state as a whole (now averaging all stations, for each year) the influence of the lack of one station becomes much less. But, as predicted by the hypothesis at the beginning of the post, there has not been a significant (i.e. R-squared greater than 0.05) increase in temperature over the 110 years.


The state is fairly flat, and longer North-South so that there is enough scatter for a good correlation with Latitude, and as with every other state, that is evident from the data.


And since there aren’t any strong geological features East-West I wasn’t looking for a strong correlation with longitude, and there isn’t one:


However, even though the state elevation only fluctuates over a couple of hundred meters, there is a strong correlation with elevation.


The other two graphs that I usually plot are the standard deviation of the data over time (to see if Anthony Watts concern about station deterioration is true – if so SD should perhaps increase over time.


And it does appear as though, within the past decade or two, that quality is beginning to suffer.

And so the question is, with the data for population being “factored out” of the data by the USHCN, is there a population trend. Bear in mind that the states where this was evident were states where there were a lot of very small communities. In Indiana the communities with stations are increasingly larger.

Hmm! As with Illinois last week it appears that there is no longer a correlation with population. Maybe the USHCN correction is a good one? (Perhaps, since the tabulation is new, I should go back and see if the earlier state data is still the same?)


Well, more interesting results. The story seems to be getting curiouser and curiouser the more we look. And we aren’t even half-way done yet!

Illinois temperature data

Today I am going to return to looking at state temperatures over the past 110 years. It is something that I started doing on Saturdays a while ago, since there has been some debate, in the blog sphere, about the validity of certain choices for stations used in the Goddard Institute for Space Studies (GISS) network for the United States. In the non-too-distant past they have truncated the number of stations that they use to determine temperature, and I became curious as to why they chose the stations that they did. So, recognizing that it is a long-handed way of doing this, I put together a procedure for looking at individual state data, which now covers a number of states (See the list on the lower right side of the page).

And in that procedure I write the post as I find and evaluate the numbers, so that as I start I don’t actually know what I am going to find. The first step in the process is to download the USHCN data for the sites that they list in Illinois and add them to the spreadsheet. There are 36 sites, and the data is smoothly transferred.

A quick check, and Chiefio states that GISS uses three stations for Illinois, Chicago, Peoria and Moline. Now this immediately gives a problem. Because when you look for Chicago on the GISS site it gives you two alternatives – Midway and O’Hare. The problem is that the data for Midway is from 1880 to 1981, and that for O’Hare is 1958 to 2010. For those not familiar with the city, I seem to recall that Midway actually sits on the lake, while O’Hare is a bit further inland. But being curious I downloaded both sets of data – with the intent of correlating the concurrent years and seeing if there was a correction I could then apply to the O’Hare data to give a full suite of data over the period of interest (1895 to 2008).


Peoria is fine, a full set of data, but Moline is also a problem, since GISS doesn’t have a station there. Hmmm! Odd! Well there is one for Davenport Iowa, which is just across the river, so I’ll use this for now. But then I check on the stations nearest to Davenport and lo! There is a Moline station, but it doesn’t start until 1944. So I’ll patch it and the Davenport data from before that period together. But first a check to ensure that they are similar, over the common period. Looking at both the two Chicago stations and then the Davenport/Moline one. Well it turns out that even though they are close, the two temperatures don’t exactly follow each other directly in either case. I have plotted the differences (Midway – O’Hare, Davenport – Moline) over the common time periods.


There is a very slight trend line increase in difference with time for Chicago, but it is not great, and the average temperature difference in Chicago is that Midway is 1.26 deg F warmer, and Moline is 0.97 deg F cooler than Davenport. It is a bit of a kludge to put the two together, but with that adjustment to allow using a full set of data for the 3 GISS stations, the difference between the GISS stations and the USHCN stations is a fairly consistent (no significant statistical change over time) 1.13 degrees lower. Which is not surprising, really, given that the 3 GISS stations are all located in the Northern part of the state.


As for the state as a whole, looking at the temperature trend for the past 115 odd years we get:


Not a very strong increase in temperature with time, in fact barely significant, and the trend is moving back towards the historical average. (I am beginning to suspect that we might see this in the MidWest states, maybe I should pose it as a hypothesis in the next state I do?)

Well I said that the reason that the GISS averages were low was that they were in the Northern part of the state, which would mean a strong correlation of temperature with latitude. And, as usual, there is.


On the other hand there isn’t much of a correlation with Longitude – not a whole lot of change in the state, going from one side to the other.


And here is the first surprise of the evening, with relatively little elevation change, I didn’t think that there would be much correlation with elevation – Wrong!


So even though the change In height is small, it is consistent with what we saw for states with much greater elevation change out west.

The premise of the next plot was that conditions around stations have deteriorated with time, and that, as a result the standard deviation of the station data would increase with time. With one or two exceptions that hasn’t proved true, but this shows steady improvement – which may be due to the automation of the data gathering.


So now it is time to get the population data (since this is one of the larger disagreements with GISS). I use the information from a Google search, and the city-data values, where available, which in this case is for all the sites in this survey. And, whoops, the second surprise of the night.


This is about the first time that there has not been a good correlation with a log-plot, and perhaps it is because of the geography of the state, where the communities are scattered around the state, with the largest city in the North.
Well I could take out latitude, and see what that did, but since I am just getting back into this, I think, for now, we’ll leave it where it is.

Now which way to go next?

Some thoughts on statistics

Back when both my arms were fully functional (a time that hopefully will return soon) I would spend one of the two posts I wrote on Saturdays looking at what was being said on some of the Climate debate websites. I visit sites on both sides of the debate, being particularly interested in the evidence that is called forth, and the conclusions drawn from it. (The current list is given by the CC sites on the blogroll). These usually lead on to others, and so one can, to a degree, filter out new evidence from that which is being rehashed. However it can require a bit more due diligence, when both sides claim that the rational answer lies only with their side of the argument.

Consider, for example, the little debate this week between Real Climate and Climate Audit on the analysis of the data from the Antarctic that I have mentioned in an earlier couple of posts. The debate relates to how many Principal Components (PCs)* should be considered in deriving the predictive equation from which to generate the model results that decide whether the Antarctic is warming or cooling.

In the original paper Steig et al had used 3 PCs in order to generate the model and thus to conclude:

Here we show that significant warming extends well beyond the Antarctic Peninsula to cover most of West Antarctica, an area of warming much larger than previously reported. West Antarctic warming exceeds 0.1 °C per decade over the past 50 years, and is strongest in winter and spring. Although this is partly offset by autumn cooling in East Antarctica, the continent-wide average near-surface temperature trend is positive.

The paper had a number of flaws in it, based on the source data, but I initially assumed that the statistical analysis was carried out using reasonably widely accepted methods, a conclusion that I am increasingly unsure of.

In an earlier post at Climate Audit the number of PCs used was shown to have a very dramatic effect on the conclusions that could be drawn from the data with Antarctica getting either warmer or cooler depending on how many PCs were chosen (ranging up to 15). And so it seems that it is time to “bite the bullet” and talk a little about what proxies, PC’s and the terms that crop in these discussions mean. (And then to point to the misdirection of a couple of stories where they aren't).


I originally sought wisdom from review of the debate over the use of recognized statistical methods that was performed for the House Energy and Commerce Committee.

The Chairman of the House Committee on Energy and Commerce along with Chairman of the Subcommittee of Oversight and Investigations have been interested in discovering whether or not the criticisms of Mann et al. are valid and if so, what are the implications. To this end, Committee staff asked for advice as to the validity of the complaints of McIntyre and McKitrick [MM] and related implications. Dr. Wegman formed an ad hoc Committee (Drs. Edward J. Wegman – George Mason University, David W. Scott – Rice University, and Yasmin H. Said – The Johns Hopkins University). The Committee was organized with our own initiative as a pro bono committee.

It submitted a report to the Committee on Energy and Commerce.

As part of that report they explained PC analysis thus:

*Principal component analysis is a method often used for reducing multidimensional datasets to lower dimensions for analysis. In this context, dimensions refer to the number of distinct variables. The time series proxy data involved are transformed into their principal components, where the first principal component is intended to explain most of the variation present in the variables. Each subsequent principal component explains less and less of the variation. In the methodology of MBH98/99, the first principal component is used in the temperature reconstruction, and also has the highest explained variance. This method is intended for dimension reduction. In most datasets, the first principal component should be the least smooth (because of the higher variance).

(MBH98/99 refers to the Mann paper that produced the “hockey stick” curve.)

They discuss the effects of temperature change on tree rings, ice cores and coral (of which more anon). but it doesn’t really help those of us who would like this explained in layman’s language. So let me see if I can do that without getting too many folk offended.

When we want to find out the surface temperature of the Earth at some time in the past, and in places where there were neither thermometers, nor folk who kept records, we have to find some other measure that shows what the temperature was. We call these measures “proxies” in part because the behavior of the selected measure approximates the changes in the value we’re interested in. For example, from the list above, the structure, density and width of the ring that a tree grows in a year will vary with the local temperature. And so, by taking a core that recovers the section through a tree that has been around for a long time, we might get an estimate, from the size and structure of the individual tree rings, of the temperature when each ring was formed.

Unfortunately it is not that simple, for example for a Scots pine in Finland.

Earlywood width is controlled by precipitation in June and temperatures in mid winter (December/January) and March. Low mean temperature in April, adequate precipitation in May and a warm July results in wide rings. A long and warm vegetation period results in high latewood density, the strongest correlations occurring with July and August temperature and precipitation.

Tree Ring structure Source NOAA

Since the density and width of the ring can vary from more than one cause, the obvious two cited above are temperature and rainfall, we need to know how much each contributes to the change in the ring. This is not that easy, since the above quote shows that you can’t just use an average value for the year, but have to look at seasonal variables within the year, and this imposes additional variations in the result that we are using as our measure. Further there are some things that may change around the tree that we don’t know about. (from the Wegman report)

Each tree ring is composed of large thin walled cells called early wood and smaller more densely packed thick walled cells called late wood. The average width of a tree ring is a function of many variables including the tree species, tree age, stored carbohydrates in the tree, nutrients in the soil, and climatic factors including sunlight, precipitation, temperature, wind speed, humidity, and even carbon dioxide availability in the atmosphere. Obviously there are many confounding factors so the problem is to extract the temperature signal and to distinguish the temperature signal from the noise caused by the many confounding factors.

Given that complexity the first thing that we need to do is to create a model from data that we already have sufficient information so that we can estimate the impact of changing different values. And this is where our Principal Component comes in. (And for the purists again my apologies I am trying to keep this simple). We can take, for example, data from the last 100 years for a site where we know the variations in temperature, rainfall, relative sunlight etc, and plot the variation in our ring values against these various factors to see which ones best explain the changes in the tree value. The ones that correlate best are called the Principal Components. For example we might find that we explain 60% of the change in tree value by change in temperature, and then when we add rainfall that we have an equation that explains 80% of the variation. By adding further factors (say number of sunny days) we can improve our model so that it predicts with increasing accuracy (smaller variance) the actual values that we found from measuring the rings. So we build our predictive model, adding additional PCs (say soil nutrition) in order of their effectiveness in reducing the variation in the model from the measured data. At some point the improvement is too small to be considered significant, and variables that fall below that value (perhaps insect density) can be neglected.

Having got, or “calibrated” the model, we can then go back from the time and values that we know the PC values for, to look at times when we don’t. Knowing the inter-relationship between the factors and the values, it is possible to estimate what the temperatures were back when the tree rings were grown. (The process is also described by NOAA)

At least that is the basis of the procedure. The problem comes in the complexity of some of the proxies that have been used to assess what the temperatures were back then. To make calculations simpler the data that is used does not use the actual values, but rather looks at the variation in the proxy value from the averaged value over the time period. So that the variation in value from that mean is the measured value against which the models are calibrated. (The process is known as centering the model, since it is the variation from the central value of the data that is used in the subsequent analysis). One of the reasons for doing this is to cope with conditions that are below the average. (By using variation from the mean one can more easily note that a drop in temperature below the average can create a negative growth in the ring that year.) It is important that this be carried out correctly since, when it has not been, and the Wegman report notes that the “hockey stick” paper did not do so, then the results obtained can be flawed.

Our committee believes that the assessments that the decade of the 1990s was the hottest decade in a millennium and that 1998 was the hottest year in a millennium cannot be supported by the MBH98/99 analysis. As mentioned earlier in our background section, tree ring proxies are typically calibrated to remove low frequency variations. The cycle of Medieval Warm Period and Little Ice Age that was widely recognized in 1990 has disappeared from the MBH98/99 analyses, thus making possible the hottest decade/hottest year claim. However, the methodology of MBH98/99 suppresses this low frequency information. The paucity of data in the more remote past makes the hottest-in-a-millennium claims essentially unverifiable.

It also comments on something else that I find quite troubling, namely the very small circle that is formed of the “climate scientists” that publish in this area, and their inter-relationships, with most of the major authors in the field being tied to Dr. Mann – but that is an issue for another day. However, this "clique" does not include many with a strong grounding in statistics, and it is this weakness, and the errors generated in the predictions as a result, which make the papers that the groups generate (and which get significant international publicity) very vulnerable to criticism, which may well be justified.

I want to go back, however, to the comments on coral, since the growth of coral is one of the parameters that are used to assess Climate Change. The Wegman Report noted:

Reef-building corals are strongly affected by temperature and, as temperature drops, the rate of calcification drops with lower temperature potentially presaging the death of the colony. Coral growth rates vary over a year and can be sectioned and x-rayed to reveal high- and low-density bands. High density layers are produced during times of higher sea surface temperatures. Thus not unlike tree rings, data on corals also can be calibrated to estimate (sea) surface temperatures.

I draw attention to that since there is a current interest in coral data, with stories commenting on the “ravages” that will occur to coral with increasing temperature. And yet, even in those stories, dealing with coral damage one finds

Sea level rise, which is projected to occur this century as the world's glaciers melt, would not necessarily kill coral reefs, Tamelander said, since the reefs can grow as waters get higher.

"A healthy reef should be able to keep up," Tamelander said.

. Climate Audit points out the inadequate statistical analysis behind the headline, and then when the statistics are applied the conclusion is wrong.

But then these facts, and the impact of land sinking

"It doesn't matter who's causing global warming. Sea-level rise is something we can measure," said Rob Young, a geosciences professor at Western Carolina University. "You can't argue that sea level isn't rising."

And it has been rising faster in the mid-Atlantic because the land here is sinking. Understanding this phenomenon requires thinking of the Earth as an enormous balloon. Push down in one spot on the ball's surface and surrounding areas are raised up. Glaciers did this to Earth's surface during the last ice age: They pressed down on northern North America and areas to the south tilted up, like the other end of a seesaw. Today, thousands of years after the glaciers retreated, the seesaw is tipping back the other way, and the region from New York to North Carolina is falling about six inches per century.

And this is my gripe for the day – we see anecdotal headlines about how this disaster or that is coming, courtesy of global warming, but these warnings are rarely accompanied by an adequate statistical analysis that shows that the facts discussed are in fact real and correlated to global warming in the manner projected.

One can see this, for example, in the recent comment by Kofi Annan story on the level of death caused by global warming. This is rebutted by the statistics as noted in the Wall Street Journal (Hat tip Irv.) There is no mention of the hundreds of thousands of acres of the sub-Sahara that have been converted to agriculture in the last decade, thereby prolonging, instead of shortening life. Rather those stories are hidden in the headlines of starvation

Oxfam expects cereal production across five countries in the dry Sahel belt south of the Sahara -- Burkina Faso, Mali, Mauritania, Niger and Senegal -- will be a record 18.5 million tonnes this year, but the food on sale will be beyond the budget of many in these, some of the world's poorest countries.