I think in the end this British sailor kidnapping will prove to have been a big mistake by Iran.
In the short term, their goal of embarrassing Britain may have worked, thanks to the eager greediness of our news media for bad news and their anxiousness to toss barbs at Blair (and indirectly at Bush). Mission accomplished.
However, after that, I'm just not seeing a good move for Iran. They can't just hand these guys back. Well, they could and in some sense it's probably still their best move ("We're so confident that we can kidnap with impunity that we don't even need these hostages; we can take whom we choose, when we choose. Your entire armed force is hostage to us now.") But in reality it'd be seen as weakness by their internal factions, probably.
(Update: QandO points out that having forced the prisoners to make false confessions, they now can't release them because as soon as the prisoners are free they'll repudiate everything they said, making the false confessions worthless. Not that they're worth much to any thinking person anyhow.)
They can't keep them, either, and publicly mistreating them is a dangerous game; they should not puncture the (im)polite fiction that only the Great Satan tortures prisoners and violates the Geneva Convention. And that latter bridge has already been crossed; in the event of war with Iran, the Geneva Conventions need not apply. They've already repudiated them thoroughly with their covert actions, but with this kidnapping and subsequent violations in the public eye, there's no longer even a shred of plausible deniability.
Now it turns out that by their own initial admission, they were indeed in Iraqi waters; later they gave out new coordinates that places them in Iranian waters, but you'd have to be pretty naive to swallow that without question. (Not that some people won't, but still.)
Provocation isn't just a game that Bush and Blair play; the subtle implications that it's so are fundamentally racist. Iran can "provoke" too, and they just did. "Botching" things is also not just a Bush and Blair game; Iran just botched their political cover story, as that previous link discusses, and that's really going to limit their moves now.
I don't see a good endgame for Iran. Meanwhile, by publicly screwing up and handing us a casus belli on a silver platter, they've opened themselves up to "limited reprisal". "The world" may not go for full war (and I wouldn't even necessarily advocate it), but it would take surprisingly little bombing to put a major hurt on Iran's infrastructure, thanks to their neglect of it.
But that's just a first-order effect. Even better would be to make Iran aware of this possible threat and contain them diplomatically, of course. This may however prove difficult as I'm not convinced that Iranian leaders understand economics enough to realize how big a threat that really is, and how easy it would be to follow through.
I have to admit that I don't think we have the will to win a conflict with militant Islam. But we only have to hold out until militant Islam's dumb-ass economic policies cause them to run out of resources to promulgate war. Militant Islam is even worse with their economy than Soviet Russia, in my estimate; at least Communism came from a fundamentally economic background, even if the economic theory was wrong.