Real-time computing. Real-time news.
June 16, 2008
Grids are Dead! Or are they?
1 | 2 All »
It looks like we have to say goodbye to our good, old grids of the past -- at least to all those beautiful features and capabilities envisioned 10 years ago, when grids were supposed to evolve toward coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations, and even to extend beyond their scientific scope. This is a great vision, but it is becoming more and more obvious that in order to make it happen, we need much more time and effort than originally anticipated.
During the past 10 years, we have seen hundreds of grid projects come and go, passing away after government funding ran dry. Most of these projects did not have a realistic (nor pragmatic) sustainability strategy, let alone viable business and operational models for their infrastructures, tools, applications and services, or their intended users. Often, the only asset left after the project was the hands-on grid expertise of the project partners, which certainly is highly valuable but in and of itself does not justify all the effort and funding.
I'm sorry to say it, but, so far, grids have not kept their promise. We are stuck with grids in Gartner's "Trough of Disillusionment."
But maybe I am too pessimistic, or too impatient. Maybe grids by their very nature are so complex to design, build and maintain, and applications are so cumbersome to grid-enable and run, that it will take another 10 years of trial and error (and re-writing grid middleware?) to find the right path through the labyrinth of coming and going technologies and paradigms -- utility computing, autonomic computing, ASP, SOA, SOI, SaaS, PaaS, HaaS, outsourcing, hosting, virtualization, Web 2.0, mashups ... you name it. And just when we think we finally got it right, the technology, and even the cultural landscape, changes again.
So, here's the problem, built in: many of our grids (architecture, technology) are simply so complex that it is almost impossible to adjust them fast enough to take into account the ever-changing IT landscape. Remember, five years ago, the transition from the proprietary grid middleware stacks to service-oriented architectures, and then to include Web services? This killed many grid projects (e.g., UK e-Science projects) -- literally about $100 million worth -- in the midst of their learning and development curve. Was it worth this $100 million? Do we really have better (in the sense of interoperable, user-friendly, flexible, dynamic, etc.) grid middleware today? Looking at Web services, for example, do we really have to expose all the details of the infrastructure to the user, for the benefit of largest-possible flexibility, which then is left to the user? Isn't this kind of explicit flexibility the actual reason for an exponential increase in effort to adjust our grid systems to always-changing technologies and strategies?
And what about cloud computing? Clouds are easier to deploy, more user-friendly, more service-oriented, and more on-demand. Still, if clouds want to replace grids, they will face challenges similar to those that grids experienced. Even with simple clouds (like Amazon's EC2), challenges arise when it comes to reputation and trust. Who trusts what really happens with/to your application and data, behind the portal, within the Amazon or Google clouds? Many of the grid showstoppers we discussed for a long time, they exist for clouds, too; but why are they not being discussed with as much passion this time around? Perhaps because our common sense tells us companies like Amazon and Google are not going to risk their reputations and destroy a potentially big business.
When we think of grids, we immediately think of networks, resources and middleware, in all their wonderful details. When we think of clouds, we think of an elastic service for remote computing and storage -- simple, user-friendly, delivered at your fingertips. William Fellows, principal analyst at The 451 Group, says "clouds are grids done properly," which comes close to the common thinking today. A bit closer to reality (and more in the context of grids), today's clouds (a la Amazon) are service nodes sitting on the Internet, which tomorrow can become compute and storage nodes in your grid for simple tasks of your application workflow. Clouds can be a very useful utility that you can plug into your grid. Fellows calls them the "Third Way," a flexible option that can sit between your in-house infrastructure services on one hand and the complete outsourcing model on the other hand: "Utility 2.0."
But these clouds are not (yet?) ready for a full-blown, complex grid-enabled application workflow, which you usually find in enterprises and computational science today, and perhaps they never will be. Even a single grid service running in a cloud image (or using a service in the cloud) will very quickly face many of the roadblocks we know from grids today.
Concerning cloud computing, the real innovators to me are not Amazon and Google (sorry), who have tons of resources sitting idle in their datacenters while, at the same time, the community is thinking about new ways of doing computing. For me, the real innovator was Sun when, in 2004, it truly built its Sun Grid from scratch, based on the vision that "the network is the computer." As with other technology trailblazers, however, Sun paid a high price for being first and doing all the experiments and evangelization -- but its reputation as an innovator is here to stay. Sun Grid's successor, Network.com, is very popular among its few die-hard clients. This is not only because it is an easy-to use technology, but especially because of its innovative early users (such as CDO2), and because of the instant support users get from the Sun team.
A similar promising example is DEISA (Distributed European Infrastructure for Supercomputing Applications) with its DECI (DEISA Extreme Computing Initiative). Why is DECI so successful in offering millions of supercomputing cycles to the European e-Science community? There are several reasons, in my opinion:
- DEISA has a very targeted focus on specific (long-running) supercomputing applications, with most of the applications just running on one best-suited system.
- It offers user-friendly access (through technology like DESHL and UNICORE).
- It stays away from those more ambitious general-purpose grid efforts.
- Its coordinating function leaves the consortium partners (the European supercomputer centers) fully independent; and
- Application experts (similar to Network.com) help the users with porting their applications to the DEISA infrastructure.
1 | 2 All »
Article Tools
Share Options
(Digg, Technorati, more)
