Gentle Cough - The Post Dispatch and Cherry-picking data

It has been a little while since I wrote anything about the Global Warming situation. Not that there is not an ongoing series of messages about how we are going to be drowned by increased glacial melting, or that extreme events might become more prevalent, and that we need to take precautions in case they do. Of course there is not a lot of evidence that the rate of extreme event occurrences has been increasing, but the alarmists feel that there is some need to drive home the message that the world has to be concerned about Global Warming, even when the globe isn’t warming. And so this post, which first notes why I wrote the last sentence, and then comments on how the media message is changing so that, by cherry picking data, alarm can still be spread.

So first let us look at the Global Warming situation. It has received very little coverage in the United States, and barely rated a mention in the UK, but the recent release of a new plot of global temperatures by the Climate Research Unit (CRU) at the University of East Anglia (UEA) is worth putting up, purely as a matter of record.

Figure 1. Global average temperatures over the past 15 years (British Daily Mail ).

This Met Office release (on a Friday) has largely been ignored by a scientific community that only exists in its current form as long as the reality that this graph presents remains ignored.

There was an immediate controversy in the UK (but not here, where it remains largely unknown) and there was a follow up report the following Sunday. But, even while ignored, the lack of increase in global temperature over the past fifteen years is surely some indication that the models widely used to predict an exponentially increasing global temperature, are falsified.

So what can a good alarmist do? Well consider the headline in the St. Louis Post Dispatch on November 26th. “2012 so far the warmest year on record in parts of Missouri.” So let me talk about this for a minute.

Notice that this does not say that the entire state is at its warmest. Rather it reports that Jayson Gosselin of the National Weather Service has noted that this was the warmest year on record for St. Louis and Columbia.

The average temperature in St. Louis so far this year is 63.4 degrees, a full degree higher than the 62.4-degree average seen in the previous warmest year, 1921. In Columbia, the previous warmest year as of Nov. 24 was in 1938, when the average was 61 degrees. This year, the average is 61.7 degrees. In Kansas City, Mo., it has been the fourth warmest year on record so far, with an average temperature of 61.3 degrees, Gosselin said.

He goes on to be more specific about when the heat wave occurred (in case we missed it!)

Gosselin, who works in the Weather Service's office near St. Louis, said the "meteorological spring" _ March through May _ was far and away the warmest ever in St. Louis with an average temperature of 61.1 degrees. Second warmest was 1910, when the average was 57.5 for the spring months. Summer also was unusually warm. Average temperatures in March, May and July all set records in St. Louis, he said.

For those who forget, I took a look at the Missouri State Temperatures first back in February 2010, when I first became curious as to whether our state was showing the global warming that everyone was talking about.

I found the location of all the US Historical Climate Network sites for Missouri and determined their location (latitude and longitude) elevation and population. Now as it turns out that there are 26 stations in Missouri, and so I took the average temperature for each station each year (this was the “homogenized” temperature in that initial post) and was able to plot the average state temperature over time.


Figure 2. Average “USHCN homogenized” temperatures for the state of Missouri (USHCN)

And if you look at that plot the state temperature has barely risen (less than half a degree Fahrenheit in 115 years) since official temperatures have been recorded, and the hottest years were in the 1930’s in the dust bowl years.

But there was something missing from the data table and it turns out that three of the largest cities in the state, Columbia, Springfield, and St. Louis were not tabulated in this network, but are, instead, part of the Goddard Institute for Space Studies (GISS) network that Dr. James Hansen used for his work.

And, being further curious, I then combined the two sets of data and obtained a plot for temperature as a function of population.


Figure 3. Temperature as a function of population size around the station. This conclusion, that there is a log relationship is not new. To quote from that post:

Oke (1973) * found that the urban heat-island (in °C) increases according to the formula –

➢ Urban heat-island warming = 0.317 ln P, where P = population.

Thus a village with a population of 10 has a warm bias of 0.73°C. A village with 100 has a warm bias of 1.46°C and a town with a population of 1000 people has a warm bias of 2.2°C. A large city with a million people has a warm bias of 4.4°C.

It is interesting to note that his coefficient is 0.317 and the one I found is 0.396.

( * Oke, T.R. 1973. City size and the urban heat island. Atmospheric Environment 7: 769-779.)

But then I revisited the state later in time, after the USHCN started also providing the raw and Time of Observation Corrected data (TOBS). And I found a few more interesting facts.

Firstly I compared the difference between the GISS data for the three large cities with the state average temperatures for both the raw data, and the “homogenized” data.


Figure 4. Difference between the average temperature in the large cities, and that of the average temperature in the State. The blue line is for the homogenized data, the red is for the raw.

I then went on to compare the TOBS average to that of the largest cities and this is what I got:


Figure 5. Difference between the average temperature in the large cities of the state, and that of the average temperature in the state using the TOBS data.

A slight upward trend, but not that significant. As for the temperatures in Missouri, over the past 100 years, with the correction – really there is no trend, it has been relatively stable:


Figure 6. Average TOBS temperature for the state of Missouri over the recorded interval.

I did note that the highest temperatures were some decades ago.

Oh and the correlation with population held up with the TOBS data, the coefficient was 0.327, and the r^2 value was 0.14.

Now I finished the entire contiguous United States some time ago, and that temperature relationship to population held up quite well, as the individual state reports listed on the rhs side of the blog show.

So what do we learn from this? That alarmist rhetoric is continuing with an embarrassing lack (for those of us who are scientists) of balance in the reporting. Data now has to be carefully cherry-picked to still be able to convey the message that the world is warming. One wonders how long they will be able to get away with this before they are called out by more prominent folk?

State temperatures - Missouri corrected for observation time

The data that I have been correlating in regard to the temperatures of the different recording stations in the different states of the Union, now comes in three flavors, rather than the one that I started with. The three are the Raw data, data that has been adjusted for the time of day of observation (TOBS), and that which has been homogenized to account for the urban heat island effect, among other things.

However the little data that I have massaged from about 8 states suggests that those massaging the data who ignored town sizes below 10,000 folk were neglecting a sensitivity that the data was beginning to repetitively indicate. Thus it does not seem appropriate to continue with using the “homogenized” data of the set that was the only one available at the U.S. Historical Climate Network (USHCN) when I started . On the other hand, a strong case can be made that the adjustment that the USHCN has made to correct for the varying time of days at which the readings were taken is a legitimate one. If one goes to the USHCN website and selects the time of day corrected data (under the listing TOBS for time of observation) then one can, in the same way as I did last week create a table of information using this simple correction to the Raw data. And so, I will see, this week, whether making that change will affect the trends in the data.

Using the procedures that I have outlined, therefore, I have just done that, and will now continue to work back through the states I have already covered (marking the list with either RAW or TOBS) and beyond using this less manipulated data. For Missouri, you may recall that my original concern was for the difference between the GISS stations and the USHCN. Last week I posted the comparison of the Raw data and it showed that changing the data and the measuring stations both made a difference.

The blue line is for the homogenized data, the red is for the raw.

And if I now look at the difference using the TOBS data I get:


A slight upward trend, but not that significant. As for the temperatures in Missouri, over the past 100 years, with the correction – really there is no trend, it has been relatively stable:


I do note that the highest temperatures were some decades ago.

Actually a little surprise in looking at the changes in standard deviation. They had been reducing over time, with both the raw and homogenized data, but when the TOBS values are used, then there is a slight increase over time.


And for the four parameters that had started to appear to have an impact on relative temperature across the states. First there is latitude:


For which there is not much change. Then there is longitude:


And though the state is relatively flat though it rises toward the west, the change due to elevation:


Which gives a better correlation (and one more logically based) than longitude.

And finally that of population, which I have shown both with log and normal scales, but for tonight – because it emphasizes the effect of towns below 10,000 in size, I’ll use the normal one again.


And so, starting with this as the new baseline of a mid-western state, let’s head off West again, and see what the TOBS data will show.

Restarting the Temperature reviews - Missouri RAW data

On Saturdays I have started looking at the temperatures of different states, as they have changed over the past hundred years. In February I started with Missouri which, apart from being where the current center of the population of the United States resides, happens to be where I live. Having been an experimental researcher for my professional life, I decided to see what the data really said, in light of the claims of both sides in the climate change debate.

When I began I had some little idea of what I might find but in the main did not want to prejudge the issue. But thought it would be fun to make the odd hypothesis and then see if the data sustained it. So far I have covered about ten states, they are listed at the very bottom of the column to the right. However, as I was writing up the information for Indiana I found that the data being presented at the US Historical Climate Network (USHCN), had changed. The data that I had been using in the reviews to that point had been adjusted before being posted, as I noted:

However, note that a subset of GHCN, the United States Historical Climatology Network (USHCN), has been adjusted via a homogenization intended to remove urban warming and other artifacts [Karl et al., 1990; Peterson and Vose, 1997].

Since one of the things that I was finding was that there remained an urban size effect in the data, even after homogenization, and that I did not know what the “other artifacts” were, it would have been nice to have access to the raw data. Well that information is now available. I did not use it, at the time I wrote the post about Indiana, since there are significant gaps in the data table provided. However, for today’s post I have gone back to Missouri, to see if the conclusions change, when raw data is used instead of homogenized.

Basically what I have done, to make my life easier was to duplicate the Missouri data table, which I described forming back in the original post I then acquired the data for the RAW mean temperatures, as I described for the Indiana post, and just pasted them into the table over the original data. Doing it this way meant that the curves were already set up, and did not have to be created again.

Then, because I want to be able to plot information from the two sets of data to seen what the differences are, I copied the entire data sheet containing the RAW data, and pasted it as a second sheet in the Excel file containing the original homogenized data. So that is where we start our investigation for the day.

The first set of values that I had looked at in the original post was to see if there was a difference between the stations that GISS is using for Missouri, and those for the more numerous and distributed stations in the USHCN. There has been some criticism that the reduction in station numbers by GISS (recognized at their site) would bias the results that they are reporting. To see if this was the case I subtracted the average temperature for the USHCN stations from the average of the 3 GISS stations and plotted the result. That result showed that the difference between the two was slightly reducing over time, implying that the criticism, for the Missouri data, was, perhaps unfounded. This was the original plot:


I now shrink the size of the individual data points, and add the same set of averaged difference between the values per year from Sheet 2 – the RAW data. (Ed. note - I will use a red color for the raw data and blue/purple for the homogenized values). And this is what I get:


Now this second curve (with equation etc to the right) shows a much stronger correlation of temp diff with time, and that, starting in about 1970 that the temperature of the GISS stations rose much more rapidly than the USHCN stations, which is the suspicion that Chiefio had expressed. This is, however, only one state and we will see what happens as we progress through the rest.

Looking at the actual temperatures over the century the curves are slightly more significantly changed. In the old plot (with homogenized data) we found:


With the RAW data, however, the plot is a little different:


Instead of there being a steady (though minor) rise in temperature over time, there is a slight decline – not particularly statistically significant, but on the other hand significant in that the “homogenization” of the data takes a negative trend in the RAW data and adjusts it the other way.

There are, in fact sufficient differences between the two sets of curves that the initial conclusions are not as strongly supported by the raw data. That, at this stage, is merely just worth noting since, for this sort of topic there is a need for much more data before stronger conclusions can be drawn. And so we will work back through the states, noting how the differences in data change the conclusions. And perhaps, after a while, there will be enough information to start to draw stronger conclusions.

But for now I am going to finish this by posting the original plots derived from the “normalized” data with the raw data graphs set below them (with the addition of the longitude graph that I did not show the first time).

Moving on to the second hypothesis, which comes from GISS, and is that temperature is insensitive to adjacent population below a community size of 10,000 folk. This is a plot of row 129 plotted against row 9. I am going to show the plot twice. The first time I am using a log scale for the horizontal axis to cover the range from a population of 30 to that of over a million.


And now I am going to change the scale so that the horizontal scale is linear, and truncate it so that it only shows the data up to a population of 50,000.


Notice how the temperature is much more sensitive to population BELOW a population of 10,000 relative to the sensitivity above that size. Thus the assumption that GISS makes in classifying every town below 10,000 as rural without any sensitivity to population is clearly not correct. That was my original conclusion – here is the data plot when the raw data was used:


Not quite as impressive a correlation, but again the scatter in the data and the form of the regression is still evident, so the conclusion stands, albeit more weakly.

I commented earlier that interestingly this also possibly explains the decline in the temperature difference with time (although it would require inputting data from earlier years census to fully explore the topic). The assumption behind the first two hypotheses was that the larger towns had a greater sensitivity to urban heat, which is getting worse, but in reality, if the smaller towns were growing faster (and require less population change to have an impact on the measured temperature) then they would be gaining temperature, because of that growth, faster than the urban sites – hence the negative slope to the graph. Given that the slope is now changed this thought (while worth remembering as we go to larger data sets is now not substantiated by the raw data analysis)

Which brings me to my hypothesis that the scatter in the data would get larger with time, given the deterioration and urbanization around the weather stations. By using standard deviation to illustrate scatter, the plot, if I am right should have an upward slope, over time.


Hmm! Well it looks as though I got that wrong – it was heading the way I thought until the 1940’s and then it started to bend the other way. Apparently the change from glass thermometers to the automated Maximum/Minimum Temperature System (MMTS) started about then and the changing shape of the curve is perhaps indicative of the spread of the new system. And that conclusion hasn’t changed.


In all these graphs it should be borne in mind that Missouri has had a relatively stable climate over the past hundred and fifteen years or so.

There are also likely influences across the state due to changes in latitude and longitude. And since, with the data table assembled, generating additional plots is easy and relatively fast, we can take a look. Originally I said “ It turns out that Longitude doesn’t have that much effect, but the temperature values are much more sensitive to Latitude than anything else that we have discussed.” Based on these curves:


And


Now when the raw data is plotted, the scatter is reduced, but there are, consistently a couple of outlying data points that influence the correlation. So these are the raw data plots for Latitude and Longitude.




And I’ll add a final plot to the set that wasn’t in the original, but later proved important (and explained some of the longitudinal variation) that of elevation, though this is a relatively flat state.


So with some rather muddy results to start with, off we go on our trek again.

Temperature data for Kansas - does it help?

Well this is the third in what was not planned as a continuing saga. So far some significant climate assumptions haven’t held up so well on examination, so today I am moving from the data for Missouri into Kansas. And the first hypothesis we will look at is that the data and trends for Kansas are the same as for Missouri. As a subsidiary I will put the two sets of data together, and with a total of around 60 stations, see if this improves any of the statistics that we looked at earlier.

Going through the same process that I followed in putting together the data for Missouri, there are 31 stations in the USHCN data base, and with just a little bit of judicious editing I can use the same format that I have already developed to start putting the different plots together. There are in addition 3 stations in the GISS network (Wichita, Topeka and Concordia). The difference between Missouri and Kansas is that while the former has all the GISS stations in the larger metropolises the Kansas GISS data does include one rural station (Concordia – listed as rural in GISS).

Now there is a second modification to the procedures, since I don’t have a map of Kansas from which to get the census data. Instead I went to the Internet, and, for consistency, used the information from the series of city data that can be found on the Web. For example, if I seek “population Anthony Kansas” the top site is www.city-data.com/city/Anthony-Kansas.html. There are similar sites for all the places in both the USHCN and GISS stations, and, for consistency therefore I used the population numbers from this series of sites. (The data is given for 2008).

I got the information on which were the GISS sites in Kansas by using the list provided by Chiefio and as I noted these are in Wichita, Topeka and Concordia. I also added the station height information (from a Google search under “elevation Wichita Kansas” ), since we are moving closer to the Rockies.

So, if you remember there was no significant warming in Missouri over the last 114 years (the data sets are from 1895 on). My initial hypothesis is that this is also true for Kansas. For our purposes an r-squared value of less than 0.05 is considered not to be significant. (I explained this a little last week). And the data says:


And so Kansas is not showing the same trend as Missouri. Here there has been a significant warming over the past 114 years. So how about the other hypotheses that we looked at in that earlier post?

First of all is there a difference between the GISS stations and the overall average for the state. In Missouri that was a 1.19 degree F difference between the GISS stations and the rest. In Kansas this is, on average only a 0.27 deg F difference. Looking at the trend in the data over the years, we find:


And this is also not entirely expected, if one follows conventional UHI theory, since the two largest stations are in the GISS trio, and one might have thought that this would have led to an increase in the GISS temperatures over the rest of the state, which is largely rural.

However, if you remember from the Missouri data, there was a logarithmic relationship between temperature and population, which gave a greater temperature change as small towns grew, over larger city changes. Thus if Kansas, a largely rural state, was seeing a greater proportion of growth in its smaller communities then perhaps this would explain the change.


The significance is still rather low (since as Luis has pointed out, working with data from the rural states reduces the sample size for larger communities). But if we combine the two data sets, what does this do to the correlation?


Now, with more data, there is a significant correlation between population and temperature.

So let’s have a look at a couple of other parameters, there was a very strong (r-squared 0f 0.8) correlation of temperature with latitude, but not much of a correlation with longitude. How does that stand up for the Kansas data?


Hmm! The correlation isn’t nearly as good. So maybe there is another factor coming into play – how about longitude, which was not significant in Missouri?


I suppose this makes a bit of sense. The further west we are going the higher, since we are approaching the Rockies. So after inputting the elevation of the stations we can see if that gives us the same correlation:


And it does not!

Which means, I suppose, that we had better continue this investigation, and move the data acquisition another state West – which was not what I expected when I started writing this, but we’ll leave that investigation until next week.

And one last bit of curiosity, how do the standard deviations hold, over time, with the new state data?


Well we are still getting that improvement in quality with time, which, as I explained initially, may be due to the change from manually reading thermometers to the automated systems being introduced. We will have to see how this holds up as our search for meaning in the data continues.

P.S. As with all the information in this series, if you want a set of the data please let me know, through comments where you want it sent.

Being a Climate Scientist for a day

So who is right, Phil Jones or Anthony Watts? Basically they disagree over the influence of town size on measured temperature. So being an experimentalist I wondered what I could do to check and see who was right. And, since this is something that you can do at home, I’m going to explain exactly what I did, since I only used the data from Missouri, and there is data available for all the states, so that those who want to can repeat for their state, what I did. (And for that reason I will explain it in excruciating detail).

Now, if you are going to analyze data it is a good idea to define what the questions are that you are seeking an answer to before you start. So let me state 3 initial hypotheses. The first is from Jones et al in 2007, which refers back to a paper by Karl and James in 1990, which says, in part

If the Canadian stations behave similarly to stations in the United States, the decrease of the DTR (diurnal temperature range) may be exaggerated by about 0.1 dg C due to urbanization.

This correlates with the 2007 paper which says

Urban-related warming over China is shown to be about 0.1°C/ decade over the period 1951–2004, with true climatic warming accounting for 0.81°C over this period.

So the hypothesis is that the rate of warming does not significantly change, as a function of the size of the community around the weather station.

The second comes from the way in which the Goddard Institute for Space Sciences classifies site sizes, calling communities below 10,000 rural. So the hypothesis is that there is no change in temperature with population below a community size of 10,000.

And the third hypothesis comes from reading Anthony Watts, and it seems to me that if his finding about the deteriorating condition of weather stations holds true, then the scatter of the data should get worse with time. So my hypothesis is that the standard deviation of the results should increase with time.

OK so I have my hypotheses – where do I get my data (you also need to have a spread sheet program, I am going to use Microsoft Excel running on a Mac) and a state map (or equivalent source for town populations). As I said I am going to look at the data for Missouri, but before I get the data I need a table to put it into. So I open Excel and create the table I want. To do this I first type titles starting in square B3, and going sequentially down, inserting the titles Station; GISS; USHCN Code; Latitude; Longitude; Elevation; and Population. I then move to square A11, and type in Calendar Year. So we start with a table that looks like this:


I am going to be putting data in from 1895 to the year 2008, and so I put 1895 into square A12, and then (=A12+1) into square A13. Then I highlight from A13 to A125 and select “Fill down” from the EDIT menu at the top of the Excel page. This now puts the years to 2008 into the A column. So now I need to get the data to put into the table. To get this I go to the US Historical Climatology Network and select Missouri from the scrolling list on the top left of the page.


I then clicked on the Map Sites button to get to the data that I wanted.


The map that now comes up shows the 26 sites that are listed for which there are records going back to 1895 in Missouri. These are listed to the right, and if you click on any one of them then the identification for it shows on the main map. I have done that for the first site on the list, Appleton City, so that you can see what I mean.


From the map information I can get the USHCN Code (230204), the latitude, longitude and elevation of the site. So I can enter those below that station name in my table, which now looks like this:


Now I need the historic temperature data for that site, and to get this I click on the “Get Monthly Data” phrase in the map balloon. This takes me to a new window.


We need to have the data in a form that will fit into the EXCEL spreadsheet, so click on the middle line in the second set of options, to create a download file. This drops you down to the bottom half of the page, and what we want (for today) is the Annual Average Mean Temperature, which is on the upper half of the screen, so I click on that box (a tick mark appears). Then I press on the submit button.



This brings up a response, which tells you the name of the file that will be downloaded to your computer, when you click the blue line, which I did.


This is the data that you want from the file (and why I included the site number when I made the table, so that I could check that I was getting each range copied into the right column, and that when I was finished I had data from all the sites). The file is downloaded into your downloads file on your computer, and when you open it in EXCEL, you get:


The data that you want is in the third column (C ) and you want to make sure that you have the Annual Average Temp so C2 should read as shown. Copy the numbers in the column (Select C3 to C116 and copy or command C). Then re-open the initial EXCEL page where we are storing the data (I call mine Missouri Annual Temp, so I will refer to it as the MAT page from now on). Place the cursor on box C13 and tell the computer to Paste (either from the Edit menu or by using command V). You should get the data pasted into the spreadsheet, as I have shown.


This is the information for the first site, and should fill down to C126. It is now a good idea to save the file.

Now we go back to the USHCN page, close the window that had the data file on it, and you should be looking at the map and list of sites again. Now select the second name on the list (in this case Bowling Green) and repeat the steps to input the site information to the spreadsheet, then select Monthly Data, and the Annual Average Mean Temperature, download the file and copy the information, and then paste it into the spreadsheet.

Keep doing this as you work down the list of stations and by the end you should have 26 station records (if you are doing Missouri – the numbers vary – Illinois has 36 for example). The right end of the EXCEL file now looks like this (except that I have put in some information in boxes W4 and Y4 that I will explain in a minute).

(and the files extend down to row 126).
Now I need the information on population, and for that I went to the State Map (since it has it all in one place), and by each town there is the population from the 2000 Census). So I then inserted this set of numbers as I moved along row 9. I had a problem when I got to Steffenville, since no population is shown. So I went to Google Earth and had a look at the place. It looks as though there is perhaps a dozen or so houses there, so I gave it a population of 30, and for Truman Dam, looking it up on Google, the Corps of Engineers Office is in Web City, so I used the population of that town for the station.


Now I’m not quite done getting data, since the Goddard Institute for Space Studies, GISS, actually doesn’t use any of these, but uses information from three of the largest cities in the state, Columbia, Springfield and Saint Louis. So I need to get the data that they use, but since they record the data in degrees C, and the rest are in degrees F, I don’t want to put their data right next to the information in the table, so while I create the station names, and put them in boxes AC3, AD4 and AE4 I am going to create the initial data columns further over in the table (actually in columns AN, AO and AP) so that after I insert the numbers I can convert them back to Farenheit.

So where do I get the GISS temperatures? Unfortunately I have forgotten the exact route I used, but you can start with the GISTEMP Station Selector page and if you click on the shape of Missouri on the map


You will get a list of the stations that they have in Missouri, though, as I said above, apparently they only use 3 of them - in part because some of the other series are not complete. (I am only going to show the top of the list down past Columbia. ) You click on the name of the station you want (we are going to use the three listed above, so the procedure is the same for each from this point).


Clicking on the station, at GISS, gives you a plot of the average temperature over the past , but you need to go one step further and click on the bottom phrase of those shown to get to the data page. Oh, and it is this page, with the populations that are less than 10,000 being considered as rural, that gave rise to the second hypothesis that we are testing. (Which I guess you might call the “and is James Hansen right?” corollary to the initial question I asked).


And being their usual helpful selves this is not easily downloadable into our table. So, having no other way to proceed, I hand-downloaded the info that I wanted into the table. I have only shown the top of the table, and the numbers that I want started out in the column on the far end, but, as I explain below, I changed the array (If you are doing this, the table can be stretched so that all the data for one year are on a single line, and then you need the data in the far right column). So the data that I am inserting in a column, starting in box AN13 is the metANN (mean temperature Annual) data. (Small hint, to check that I had the years and data correct I squeezed the frame down so that the row data for a year appeared in 2 rows with the metANN in the second row one column over, as shown, and I started with 1895 (12.05) to be consistent with the other tabulated values. (And I only copied the left half of the screen that contains what I need).


This took a little while, since the data had to be hand entered for all three sites, but after maybe an hour (remember I titled this a day as a climate scientist – this is why), we have the data from the three sites that GISS uses in Missouri entered into the table, in Centigrade. The latitude and longitude aren’t given quite as precisely, we don’t have heights, and the population values don’t match the census, but this is the GISS data, and we gratefully take what we are given.


Now we need to go over to the initial columns for these stations (AC to AE) and insert the same data, converted to deg F. (so box AC13 has the equation = 32+(AN13*9/5)). Then I filled down from AC13 to AC126, and then filled right AC to AE, and the conversion was done). In the new columns I also entered the 2000 census data for the three sites, rather than the GISS values.

Now I have the raw data that I need to do the analysis. As they say most research is in the preparation, the fun part comes a lot more rapidly. To get to that we need to add just a few calculation steps into the raw data table. The first one that I am going to add is to simply take three different sets of averages. The first is for each year for the USHCN data without the GISS stations, the second is each year for the GISS stations, and the third is for each station over the full period of the data set. (I am not going to show the screens for this since they are rather straightforward.)

To get the first average I go to column AG – call it Historic ave, without GISS) and in AG13 type =SUM(C13:AB13)/26 which calculates the average or mean value. Then I select from AG13 to AG126 and fill down from the EDIT menu. That gives me the average annual temp for the USHCN data. Then I add, in column AH, which I call Standard Deviation, and in box AH13 a statistical formula (that I get from the Insert menu > Function > STDEV ). This puts =STDEV() into the box, and I give it the range I want examined by either selecting all the boxes from C13 to AE13. Or typing that in, so that the box reads =STDEV(C13:AE13) and click ENTER. This gives me a measure of the scatter in the data for the year 1895. I then select the boxes AH13 to AH126 and filled down. This now has tabulated the change in the scatter of the data over the last 113 years. (And I’ll come back to this in a minute).

Now I want the average of the GISS stations, so I created this in column AJ, typing the formula =(AC13+AD13+AE13)/3 into that box. Then, as before, selecting and filling down the column. And then there is a final column – which I call Difference, which is the difference between the Historic data set and the GISS set. That is created in column AL, and is simply typed into AL13 as =AJ13 – AG13, then filled down to AL129. (It is extended to include the average values that are calculated next). And to provide the overall average for the state I combined all 29 station data into a combined average in column AF.

And the individual station averages are created by typing =sum(c13:C126)/114 into box C129, and then filling that right to column AG129. This formula can then be copied and pasted into box AJ129 to give the average value over the years for the stations that GISS relies on. And immediately it is clear, by looking at box AL129 that there is an average difference, over the years, of 1.19 deg F between the stations that GISS are using, which are in the larger cities, and those of the more rural stations in Missouri. (Which would seem to validate the criticism from E.M. Smith, but that, as they used to say in debate, “is not the question before the house.”)

Now you are going to have to take my word for this, but when I started making this data set I had no knowledge as to how it would turn out, though I had some expectations. Let us now see what happens when we plot the data. (I am going to use the charting function of EXCEL, and add trendlines to the data, with equations and the r-squared value, so that we can see what is happening, since the data is scattered about a bit on each individual plot).

So the first hypothesis we wanted to verify was - the rate of warming does not significantly change, as a function of the size of the community around the weather station. Given that the historic average is for smaller stations and the GISS average is for the larger communities, this would, initially suggest that a plot of difference against time should show no change, if this hypothesis is true.
(Plot of column AL against A)


Well this shows that the difference has been getting less, rather than increasing – which, if anything I suppose initially supports the hypothesis. But out of curiosity I wondered how much temperature change we have had, since the actual relationship hypothesized was about rates of change. So let’s plot average temperature against time.


Now if you look at that plot Missouri has had quite a wimpy warming, less than half a degree F over a hundred and fifteen years, so given that small range, detecting changes in the rates is perhaps not feasible. But let’s plot the difference as a function of temperature just to see if there is anything.


And still it goes down? Wonder if that is trying to tell us something?

Moving on to the second hypothesis, which comes from GISS, and is that temperature is insensitive to adjacent population below a community size of 10,000 folk. This is a plot of row 129 plotted against row 9. I am going to show the plot twice. The first time I am using a log scale for the horizontal axis to cover the range from a population of 30 to that of over a million.


And now I am going to change the scale so that the horizontal scale is linear, and truncate it so that it only shows the data up to a population of 50,000.


Notice how the temperature is much more sensitive to population BELOW a population of 10,000 relative to the sensitivity above that size. Thus the assumption that GISS makes in classifying every town below 10,000 as rural without any sensitivity to population is clearly not correct.

And interestingly this also possibly explains the decline in the temperature difference with time (although it would require inputting data from earlier years census to fully explore the topic). The assumption behind the first two hypotheses was that the larger towns had a greater sensitivity to urban heat, which is getting worse, but in reality, if the smaller towns were growing faster (and require less population change to have an impact on the measured temperature) then they would be gaining temperature, because of that growth, faster than the urban sites – hence the negative slope to the graph.

Which brings me to my hypothesis that the scatter in the data would get larger with time, given the deterioration and urbanization around the weather stations. By using standard deviation to illustrate scatter, the plot, if I am right should have an upward slope, over time.


Hmm! Well it looks as though I got that wrong – it was heading the way I thought until the 1940’s and then it started to bend the other way. Apparently the change from glass thermometers to the automated Maximum/Minimum Temperature System (MMTS) started about then and the changing shape of the curve is perhaps indicative of the spread of the new system.

In all these graphs it should be borne in mind that Missouri has had a relatively stable climate over the past hundred and fifteen years or so. There are also likely influences across the state due to changes in latitude and longitude. And since, with the data table assembled, generating additional plots is easy and relatively fast, we can take a look. It turns out that Longitude doesn’t have that much effect, but the temperature values are much more sensitive to Latitude than anything else that we have discussed.


And it may well be that dependence that hides some of the nuances of the other relationships.

Well there we are, a little exercise in climate science. Of the three hypotheses we looked into, it turned out that the second and third were wrong, and because of that it may be that the data on which the first was based was not focused sufficiently on the changes in the small size of some of the communities (if the sensitivity gets less above a town size of perhaps 15,000.

The procedures that I spelled out in such detail should allow anyone else to run this same series of steps to determine if what I found for Missouri holds true over other states in the Union, and if anyone wants a copy of the spreadsheet, let me know where to send it through comments. (While yes I work at a University and yes I acquire data, I have no clue how to store it on the master servers –all of ours, for lots of good reasons, are stored otherwise, and so I don’t know how to make the file available in other ways than by attaching it to an e-mail).

And so to summarize the exercise, which as I noted in the title took me about a day to do and write up – by analyzing the data from 29 weather stations in Missouri, which have a continuous record of temperature from 1895 to 2008 (and are still running I assume) we have shown that
a) It is not possible to decide if Anthony Watts or Phil Jones is correct, since there may have been an incorrect assumption made in the data collection, which (conclusion b) means that the wrong initial assumptions were made in parsing the data.
b) The assumption that the “urban heat island” effect gets greater with larger conurbations is not correct in Missouri, where the data suggests that the sensitivity is most critical as the community grows to a size of 15,000 people.
c) The hypothesis that the data scatter gets worse with time because of deterioration in station conditions does not hold in Missouri, when the assumption is predicated on increase in the standard deviation of the readout between stations in a community. However this assumption may have been valid where reliance was placed on glass thermometers, since it is possible that the change to automated instrumentation has, at least for the present, over-ridden that deterioration.

So much for my venture into climate science, at least for now. I just wanted to show that it is not that difficult to check things out for yourself, and, provided you have the time, it can yield some unexpected results. (Though I should point out that Anthony Watts and Joseph D’Aleo quoted Oke in their report on surface temperature records, (page 34)

Oke (1973) * found that the urban heat-island (in °C) increases according to the formula –

➢ Urban heat-island warming = 0.317 ln P, where P = population.

Thus a village with a population of 10 has a warm bias of 0.73°C. A village with 100 has a warm bias of 1.46°C and a town with a population of 1000 people has a warm bias of 2.2°C. A large city with a million people has a warm bias of 4.4°C.

It is interesting to note that his coefficient is 0.317 and the one I found is 0.396.

Which is the other thing that you learn when doing research, most of the time someone else has been there before you, and there is little that is new, under the sun.

* Oke, T.R. 1973. City size and the urban heat island. Atmospheric Environment 7: 769-779.