I've been trying to hunt down a bug for months. It's one of those bugs that you can't replicate, that you can really predict, and that only happens in production once every few weeks. It's also a bug that can essentially completely break the system, and at the worst possible time... during periods of heavy load.
The basic story is that I have an application that handles tens or hundreds of thousands of requests a day. These requests are in the form of MSMQ messages which are sent to a set of middle tier machines for processing from a variety of sources and for a variety of purposes.
Once they reach the middle tier, a component called the Dispatch Manager deserializes the contents of the message, examines the .NET type and various properties of that type that adhere to a particular interface called IDispatchedMessage, and then hands the message off to another component for processing. The process of peeking the message on the queue and dispatching it happens in a multithreaded, object pooled environment.
The problem was that sometimes the message types wouldn't get deserialized properly. I would get errors stating that the Dispatch Manager didn't know how to read that particular message. This almost always happened during periods of heavy load, when there were dozens of threads simultaneously dispatching messages. After a fairly random amount of time, but always when the load started to decrease a little, the Dispatch Manager would magically remember how to deserialize those messages again and the problem would go away until the next time I restarted the application or the internal application cache was cleared.
The message types and where they're supposed to get dispatched to are stored in a SQL Server database. This data doesn't change too often, so I cache it using a singleton pattern. The Dispatch Manager constructs an XmlMessageFormatter class instance using the type names from the database, caches this instance, and returns a reference to this static instance when requested. This instance is then assigned to the MSMQ message and the Body property is accessed to get the deserialized data. Something like:
public IDispatchedMessage DeserializeMessage(System.Messaging.Message message)
{ IDispatchedMessage dispatchedMessage;
XmlMessageFormatter messageFormatter;
messageFormatter = MessageFormatterCache.GetMessageFormatter();
message.Formatter = messageFormatter;
dispatchedMessage = message.Body as IDispatchedMessage;
return dispatchedMessage;
}
Looking at this code, I concentrated my attention on the MessageFormatterCache. I figured it had to be the problem. Some how, my singleton pattern was failing under load. I rewrote the code several times. I did extensive research on potential issues with double checked locking in .NET. Not matter what I did, the code still broke.
Well, that wasn't the issue. It was a far more insidious issue, and honestly one I should have caught a lot earlier. The XmlMessageFormatter class, like many classes in the .NET Framework, is not thread safe. Usually, this doesn't actually mean that it won't work, it just means that it hasn't been tested or designed with multithreaded access in mind. In the case of the XmlMessageFormatter, they weren't kidding.
The clue that tipped me off was that just before the Dispatch Manager would start forgetting message types, I would get an exception stating that a collection had been modified and therefore the enumeration could not continue. I was never creating, modifying, or enumerating collections in the code in question, so I took at look at the stack trace. Sure enough, it pointed directly to the Read method of the XmlMessageFormatter.
Here is a slightly abbreviated version of that Read method:
public object Read(Message message)
{ this.CreateTargetSerializerTable();
XmlTextReader xmlReader = new XmlTextReader(message.BodyStream);
foreach (XmlSerializer serializer in this.targetSerializerTable.Values)
{ if (serializer.CanDeserialize(xmlReader))
{ return serializer.Deserialize(xmlReader);
}
}
throw new InvalidOperationException(Res.GetString("InvalidTypeDes..."));Well, there is the enumeration that was generating the exception. Wait a minute, what's that method call at the top of the Read method? That looks like something that might change state, and in a multithreaded context, that's a very bad thing. Let's take a look:
private void CreateTargetSerializerTable()
{ if (!this.typeNamesAdded)
{ for (int i = 0; i < this.targetTypeNames.Length; i++)
{ Type type = Type.GetType(this.targetTypeNames[i], true);
if (type != null)
{ this.targetSerializerTable[type]
= new XmlSerializer(type);
}
}
this.typeNamesAdded = true;
}
if (!this.typesAdded)
{ for (int j = 0; j < this.targetTypes.Length; j++)
{ this.targetSerializerTable[this.targetTypes[j]]
= new XmlSerializer(this.targetTypes[j]);
}
this.typesAdded = true;
}
if (this.targetSerializerTable.Count == 0)
{ throw new InvalidOperationException(Res.GetString("TypeListMissing")); }
}
That's no good at all. This method is clearly not thread safe. Multiple threads can get to the boolean check and continue on into those if blocks before the first thread has finished building the collection and setting the bool to true.
This will cause everything to break until load calms down a little and one thread has time to completely run through the process of deserializing a single message. This is exactly what I was seeing.
The fix was fairly simple. Just use a cloned copy of the XmlMessageFormatter for each thread. I modified the MessageFormatterCache to return a cloned instance, and the problem disappeared. Microsoft even implemented ICloneable for the XmlMessageFormatter. That should have tipped me off right from the beginning. Oh well.
I also "primed" the cached instance by forcing it to deserialize a message in a thread safe context beforing letting it out into the wild world of multiple threads. This improved performance a little since it wasn't building the collection over and over again each time a thread used it.
Just goes to show, you should never assume that a class is thread safe.
I was one of the most pessimistic of pessimists when it came to not just the future of the world's energy supply, but the future of modern society as we know it. While always remaining skeptical, I pretty much bought into the doomsday predictions about peak oil. I surfed the peak oil web sites. I watched and read the peak oil movies and books. Hell, I bought two domain names and started working on a peak oil site of my own. I was sold.
Honestly, I should have known better. People who predict the end of society have a pretty poor track record of being correct. I was probably drawn in by a combination of my decidedly cynical outlook on politics, as well as my just-enough-to-be-dangerous insight into energy alternatives, oil production, macroeconomics, and science.
I am no longer a believer, partly because of a better understanding of new energy technologies, and partly because of a better understanding of our modern economy. I'll leave my knew economic viewpoint for another blog post, but suffice it to say that our economy is not nearly as dependant on oil as it was 30 years ago. The advent of the Internet, information technology, and our shift toward the service sector dramatically dampens the effect that oil shortages have on the economic engine as a whole. Short of most of the world's oil disappearing overnight, we wont' see the collapse of society that is often predicted by peak oil enthusiast.
For this blog post, I'm going to focus on one particular technology that all by itself could allow us to reduce our fossil fuel consumption by 30%[1], and that's only when you use it for electrical power production. This technology is called concentrated solar power.
Concentrated solar power, or CSP, basically amounts to using big mirrors to focus the sun on a small area in order to get something very hot. In other words, it's like what a lot of kids do to ants with magnifying glasses. It really is that simple. Typically, the focal point contains a fluid with a very high specific heat, such as molten salt. This hot liquid then travels to either a heat exchanger so it can turn water into steam for steam turbines, or into something like a sterling engine. (Although at this point our best bet is still steam turbines.) This heat can also be stored for power generation during periods of little or no solar radiation, like at night.
I had heard of CSP before, but had discounted it as not having a high enough efficiency to be a viable alternative. I had always assumed, and indeed read, that CSP, along with most other "green" alternatives, would take up so much land area that we would need to use virtually all the free space in the United States for power production to offset fossil fuels.
As it turns out, this is complete baloney. In fact, a 92 by 92 mile area in the south western US (probably encompassing Death Valley) could power the entire United States electrical grid... and we would even have a little juice left over to sell to our friends in the north. That's just a bit more than 1/3rd of the Mojave Desert, which I'm pretty sure isn't being used for much right now.
Think about that for a minute. Every single watt of electricity that is currently produced by a combination of coal, oil, natural gas, nuclear, hydroelectric, wind, and even photovoltaics could be produced by a big set of mirrors in the desert. That's almost 500,000 megawatts of capacity.[2] We would even have plenty of space left over to keep up with new demand, or with the demand of our neighbors to whom we would undoubtedly be happy to sell our electricity. Of course, most of our exports would probably be heading north, since Mexico's "solar resources" are nearly as good as ours.
What an interesting turn of events it would be if the parts of the world that are currently the most poor and devoid of natural resources suddenly became the Saudi Arabia of the 21st century. (Yes, I realize that Saudi Arabia would also be included in this list.) Huge swaths of North Africa that are currently some of the poorest nations on earth could be sitting on (or, rather, under) the world's largest and most valuable commodity of the future: solar radiation.
I know what you're thinking. You're thinking that CSP plants are expensive to build, or they're difficult to maintain, or they use exotic materials that are extremely expensive. Well, there is some truth to that, but not so much truth as to make it a deal breaker. CSP plants, as I mentioned before, are primarily glass, concrete, and steel. They're less complicated than a modern office building. As a result, CSP plants should last 50 to 100 years without any serious upkeep costs. The only thing that makes them particularly expensive is their massive size.
How expensive, exactly?. Well, if we set out to fully move our electrical grid over to a combination of CSP and the existing alternatives like hydroelectric, nuclear, and wind, we would need to build around 350,000 Mw of capacity. The area required to produce this capacity is about 78 by 78 miles, or about 6,104 square miles. Using the data from a Department of Energy presentation given to the state of California back in 2003, as well as data from the construction of a small CSP plant in Nevada, I've calculated that the current cost per Mw of capacity is about $3,750,000. That's a lot of money. It would cost over $1.5 trillion to build enough capacity to service the entire United States.
No, I'm not suggesting we spend $1.5 trillion. Those costs don't take into account any of the magic of the economies of scale. The DoE presentation I mentioned earlier estimates that a 250 Mw plant would cost around $750 million to construct. A 1000 Mw plant would run you around $2 billion. A 6000 Mw plant would cost about $11 billion. As you can see, our cost per megawatt steadily decreases when you increase plant capacity. Assuming we plateau at 6000 Mw in terms of the economies of scale (which I have little reason to conclude, but for the sake of skeptical argument let's go with it), we're already down to $1.8 million per megawatt. That's about half the cost per Mw of the 65 Mw plant in Nevada.
All of a sudden we're looking at a price tag of around $650 billion to generate all of our electricity from things other than fossil fuels, or around $900 billion to go 100% CSP. Still sounds like a lot of money, doesn't it? Well, it is a lot of money, but it's not money that is simply exiting our economy like some of our other expenditures. If we set out with the goal of no fossil fuels for electrical production in the United States by 2025, it would mean investing around $37 billion a year into CSP. That's about 30% of what we spend in Iraq per year. Does it still sound unreasonable? Remember, one big reason why we're in the mess in the Middle East is because of fossil fuels. We could kill two birds with one stone.
Still not convinced? The DoE presentation also cited data that showed that the construction of a single 1000 Mw facility results in about 7000 new jobs for a period of about a year, and then around 2000 permanent jobs thereafter. We would need to construct over 20 of these plants per year to meet our goal. That's 144,000 jobs a year, 40,000 of which are long term. Can you imagine if our economy added 144k new jobs every year for 17 years in one sector alone? Any President with that on their resume has some serious bragging rights.
Most power plants are heavily "back loaded" in their costs. A coal power plant, for instance, costs 20% "up front" (construction), and 80% in fuel costs and maintenance. A CSP plant is exactly the opposite. 80% of the cost of a CSP plant is in its construction, and 20% in its maintenance. (Fuel is free... as long as the Sun doesn't charge.) This means that unlike fossil fuel plants, CSP has the potential to get cheaper over time. You would think this is a good thing, but in terms of private investment, it's not.
An investor doesn't want to wait 50 years to make his money back. If he can build a 1000 Mw coal plant for $250 million and charge 8 cents per kilowatt hour, he'll make his money back in far less time than if he built a $750 million CSP plant and charged the same amount per kilowatt hour. The investor knows full well that 30 years down the road that CSP plant will be a gold mine since fuel is free and maintenance costs are low. That 8 cents per kilowatt hour will be 80% profit. Alas, the investor isn't going to live to be 150 and enjoy the fruits of that labor, so he is going to stick with coal. At best, the investor might be willing to pony up the $750 million if he can charge 15 cents or 20 cents a kilowatt hour, but those prices aren't competitive in our fossil fuel driven world.
So how do we fix this? The government needs to "make the market". What that means is that there needs to be significant incentives in the form of tax breaks and straight up cash to build these plants. If we can level the playing field between CSP and fossil fuels in terms of initial investment costs or in terms of cost per kilowatt hour, CSP will take over. If we as a nation are willing to spend $120 billion a year on the Iraq war under the slogan that it will make us safer and stronger as a nation, why not $37 billion a year for something that will actually make us safer and stronger as a nation?
The best part is, CSP is just one possible technology that can save us from peak oil. New nuclear reactor designs, innovative wind turbines, and the constantly increasing efficiency of photovoltaics will all help as well.
Peak oil has either arrived or will arrive in the very near future. Very few people still doubt that. How its arrival affects us isn't out of our control. We have a choice.
[1] The US uses 40% of its fossil fuels for electricity production, but only 72% of electrical production is via fossil fuels. So 72% of 40% is about a 30% potential decrease in fossil fuel usage.
[2] Our current grid capacity is somewhere around 500,000 megawatts, or about 4,380 terawatt hours of production per year. It breaks out to: Coal = 245,890 mw, Natural Gas = 89,383 mW, Nuclear = 97,442 mW, Oil = 16,095 mW, Solar/Geothermal/Hydroelectric/Wind/Other = 38,403 mW. For more details, go here.
