Slashdot recently had an article on gibberish in spam. I posted a comment about my work on Bayesian, which reminded me I need to update that. This post will have to do.

First, you'll note a cranky tone in the Slashdot postings. Would you believe that Bayesian filtering has fanboys? A lot of people seem congenitally incapable of reading something about Bayesian once they get the faint idea that I may be a little critical of them. (Completely over their head is the distinction that I'm not critical of Bayesian per se, but of the idea that it will solve the spam problem once and for all.) Instead of reading the words, they seem to suffer some sort of strange vision ailment that renders them incapable of seeing anything but the phrase "Bayesian Filters are bad."

Tell one, tell all, the invaluable Snopes.com has finally gotten an RSS feed!

Snopes is required reading for people on the Internet. If it sounds too good to be true, if it's a little too conveniently in favor (or against) your favorite ideological position, or if it's a little too horrifying to be true, check it on Snopes before you get upset, or worse, spread the claims further. Because you'll meet someone who has nearly the entire site indexed in their head, and there's little that's more damaging to your point then to have it conclusive rebutted on Snopes.

There’s just no nice way to say this: Anyone who can’t make a syndication feed that’s well-formed XML is an incompetent fool. - Timoth Bray on ongoing

I'd like to chip in on this with a couple of hard numbers. Depending on your library support, it is possible to write a basic OPML parser in around half-an-hour using an existing XML parser. I've done it twice now, once directly using a SAX-like XML parser and once with a home-rolled XML<->object library; it's easy. It's even a good way to learn how to parse XML because of OPML's essential simplicity.

This is the first public posting regarding my next major project, Iron Lute (links to a screenshot). Iron Lute is an outliner, written in Python and using Tk as its toolkit; as a result it should run on Mac, Linux, Windows, and anything else that supports Python and Tk. (Off hand I don't know how many platforms that is, but it is plausible this would work on palmtops and some other obscure ones with few changes.) The license is currently undetermined, but will initially be open barring surprises; more details in a forthcoming post.

Thanks to the postscript of my Big Number Fallacy post, I am now the world's sixth leading authority on how to blow up the Earth. (This message may eventually take precedence, though I hope not.)

This is easily my proudest Google moment.

I'm torn between Fisking this essay on how privacy protections are hampering intelligence and programming with the rest of tonight. I can't quite put this essay down but we'll see how little I manage to say about it.

• First, consider the source: Stewart Baker heads the technology law practice at Steptoe & Johnson in Washington, D.C. From 1992 to 1994, he was general counsel of the National Security Agency. A former intelligence official calls for more intelligence powers. A policeman (of sorts) calls for more police powers. Police always think they need more power, and never see (willingly) the abuses the power can and is put to. They can't; it causes cognitive dissonance. This doesn't mean we can simply ignore the arguments, this doesn't mean he can't possibly have insight, but the rest of us certainly need to keep it in mind.
• Second, everywhere he says something to the effect of "privacy laws are hampering our efforts", you may freely substitute the idea "we need more power". Because that's what this boils down to, a cry for more power; the putative thing keeping them from the power doesn't really matter.
• And on Sept. 11, 2001, [the wall of seperation between law enforcement and intelligence] probably cost us 3,000 American lives. That's a highly skewed way of looking at "the wall". -3000 lives? Absolutely nobody else ever hurt by it? Absolutely nobody else ever helped? It's good rhetoric, it's bad logic. It needs to be considered in totality, and remember "the wall" can't take 100% of the credit for those deaths either, if indeed it can take any; it is a hypothesis by Stewart Baker that this is the "root cause" of this complicated event, supported by descriptions of two email messages, which we have to take Stewart's word even accurately reflect the content of these emails from a single frustrated New York agent. Many other failures occurred, and these must take their credit too.
• We couldn't find al-Mihdhar and al-Hazmi in August 2001 because we had imposed too many rules designed to protect against privacy abuses that were mainly theoretical. I can't speak to the specific rules, but I observe that while I can't necessarily pull out specific examples, it is a logical certainty that those rules were put in place because of privacy violations that hurt people. In typical beauracratic fashion, they may have been pointless, contrived, absurd, useless, or even dangerous, but that's a failure of the beauracracy, not the idea of privacy protections. The rules themselves may have been protecting against theoretical privacy violations, but the idea of "privacy violations" themselves are not theoretical in the slightest.
• They were focused on the hypothetical risk to privacy if foreign intelligence and domestic law enforcement were allowed to mix... Again, I challenge this word "hypothetical". I for one would not welcome a return to the days where this can mix. It's such a bad idea that it's almost universally rejected in civilized countries, which is the entire reason why we "needed" Echelon in the first place; so those countries could spy on each other's citizens, getting around these restrictions, and passing intelligence to each other. The very fact that Stewart can refer to these abuses as "hypothetical" is strong evidence that the protections are working, since we apparently have rendered these issues "hypothetical"; we weaken them at our own peril, and it would be foolishness to strip them away again.
• The second lesson is that we cannot write rules that will both protect us from every theoretical risk to privacy and still allow the government to protect us from terrorists. We cannot fine-tune the system to perfection, because systems that ought to work can fail. Emphasis mine. The bolded statement can stand on its own with no further elaboration; arguments that implicitly claim that we could fine-tune the system to "perfection" if we just got rid of that pesky privacy stuff should be recognized for what it is: Sleazy salesmanship. No system can tuned to perfection, those that are designed for perfection fail miserably. Douglas Adams captured this perfectly in his Hitchhiker's Guide book in the phrase "The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at and repair."; this describes beaureucratically-designed systems to a T. A perfect defense is a promise the intelligence community can not make, whether we have privacy or not.
• So the effort to build information technology tools to find terrorists has stalled. And what respect I may have had goes out the window. He's a lawyer, so explanations of why what they were building were useless would probably shoot over his head, but I wish for once they'd listen to the people who know what they're talking about. Casting a dragnet over all data can't work. The tools can't work. It's been established that the data was there, collecting more data will just hurt them.

I couldn't find this on the Internet, so I present to you the "Big Number Fallacy":

Big Number Fallacy: All big numbers are infinite. In particular, they are all larger then each other, smaller then each other, and the same as each other, randomly.

In mathematics, depending on your choice of axioms, infinity is a "number" that can be greater then itself, less then itself, and equal to itself at the same time. Its ability to perform this feat is fundamentally why it's not a number (hence the scarequotes); it does not act like a number, therefore it isn't one.

In light of the won't do and can't do, Microsoft sits there, and watches its market share begin to erode. That's happening slowly at first, but the snowball is rolling. A few people are starting to look up the hill and notice this big thing barreling down at them, and some are bright enough to step out of the way.

I believe this is going to happen. The most convincing arguments are the "arguments from culture":

And now we have reached the point where the science/engineering feedback loop has given engineers the tools and technologies to create the internet, the most recent of my four most important inventions in human history. And just as with the other three (spoken language, writing, movable type printing) it will cause a "knee" in human capabilities and behavior. And because of that, a true superhuman "intelligence" may appear during our lifetime.

But when it does, most of us won't even notice it, because it will be lost amidst the great sea of mediocrity and banality which will always dominate the internet and consume the vast majority of its bandwidth as long as humans exist. - Steven den Beste, Superhuman Intelligence

Disclaimer: I'm not really a fan of Crichton. He was a tolerably good science fiction writer, though not in danger of being called a "Grand Master", which may be why he mostly got out of it. His latest efforts aren't really science fiction so much as Hollywood "sci-fi", where you read a couple of newspaper reports on a new technology, read a couple of summarizations from wild-eyed advocates and equally wild-eyed naysayers, and start writing without particularly caring if you even stay true to those sources. No need for real science. (Contrast this to the Jurassic Park book, which had a reasonably coherent use of chaos theory, for instance.) I went into this speech expecting to dislike it.