Lunch at the Labs

In all the discussions related to whistleblower Edward Snowden's revelations about the NSA and the PRISM surveillance program, first revealed by Glen Greenwald in The Guardian, there are some points I have not seen mentioned and that I wonder about.

Isn't the fact that they did NOT stop the Boston marathon bombers evidence the surveillance program does NOT work very well? It seems it was only helpful AFTER the fact in figuring out the motives.

As the Osama bin Laden raid showed, real terrorists do NOT use the phone system or the internet.

Now that everyone knows about the NSA surveillance, who would be dumb enough to get caught that way? Even if was useful before, how can it still be?

I don't understand the government's reasons for secrecy. If as the government says they foiled more than 50 terrorist plots, surely the people who got caught and their associates (and now terrorists everywhere) know about the surveillance programs. It seems to me the main purpose for secrecy is to keep the American people in the dark about it. (Likewise the secrecy behind the deaths caused by our drones only keeps Americans in the dark; those being attacked know how many terrorists and innocent people are being killed.)

Some things about this massive indiscriminate collection of information reminds me of Reagan's Strategic Defense Initiative (SDI) to shoot down incoming missiles. A big problem there was the enemy could send in hundreds of decoys and there was no good way to distinguish them from the real threats. Similarly why couldn't anyone who wants to foil NSA surveillance, overwhelm the system with dummy phone calls and email messages? Given the amount of email spam and the phone robocalls I get, it doesn't seem that hard to do.

Why not require the government to prove they can do what they claim? Here's a way that doesn't require giving away any secrets. Under the auspices of some legitimate technically knowledgeable organization, like the Electronic Frontier Foundation (EFF) who defends the public's digital rights interests or the Association for Computing Machinery (ACM) for computer professionals, let an unknown group of people use phones and email to plan a mock terrorist attack and see if the NSA can catch them! They would be allowed to use fake message decoys and encryption and other methods to obfuscate the real plot. I bet the government would (a) not accept the challenge, and or (b) not catch them.

Addendum 06/24/2013: For some clues to possible answers see:

The Dirty Little Secret About Mass Surveillance: It Doesn't Keep Us Safe

Top National Security Experts: Spying Program Doesn't Make Us Safer, and Spying Leaks Don't Harm America

In fields such as politics and investing many people are in the business of making predictions and interestingly their reputation does not seem to depend on how right (or wrong) they are. In the June 2011 freakonomics podcast:

The Folly of Prediction

Stephen Dubner summarizes one aspect of it nicely:

So, most predictions we remember are ones which were fabulously, wildly unexpected and then came true. Now, the person who makes that prediction has a strong incentive to remind everyone that they made that crazy prediction which came true. If you look at all the people, the economists, who talked about the financial crisis ahead of time, those guys harp on it constantly. "I was right, I was right, I was right." But if you're wrong, there's no person on the other side of the transaction who draws any real benefit from embarrassing you by bring up the bad prediction over and over. So there's nobody who has a strong incentive, usually, to go back and say, Here's the list of the 118 predictions that were false. And without any sort of market mechanism or incentive for keeping the prediction makers honest, there's lots of incentive to go out and to make these wild predictions.

One participant in that podcast is Philip Tetlock whose book, Expert Political Judgment: How Good Is It? How Can We Know? was reviewed in The New Yorker back in December 2005. A more recent conversation (December 2012) with him can be found on Edge.org:

How To Win At Forecasting

Under the auspices of the Intelligence Advanced Research Projects Activity (IARPA), Tetlock is co-leader of The Good Judgment Project, one of five teams competing in the Aggregative Contingent Estimation (ACE) Program whose aim is to benchmark the accuracy of predictions and discover ways to improve that accuracy by using level playing field forecasting tournaments. To whet your appetite, here are 3 excerpts from the above Edge link.

So, we found three basic things: many pundits were hardpressed to do better than chance, were overconfident, and were reluctant to change their minds in response to new evidence. That combination doesn't exactly make for a flattering portrait of the punditocracy.

. . .
One of the reactions to my work on expert political judgment was that it was politically naive; I was assuming that political analysts were in the business of making accurate predictions, whereas they're really in a different line of business altogether. They're in the business of flattering the prejudices of their base audience and they're in the business of entertaining their base audience and accuracy is a side constraint. They don't want to be caught in making an overt mistake so they generally are pretty skillful in avoiding being caught by using vague verbiage to disguise their predictions.

. . .
The long and the short of the story is that it's very hard for professionals and executives to maintain their status if they can't maintain a certain mystique about their judgment. If they lose that mystique about their judgment, that's profoundly threatening. My inner sociologist says to me that when a good idea comes up against entrenched interests, the good idea typically fails. But this is going to be a hard thing to suppress. Level playing field forecasting tournaments are going to spread. They're going to proliferate. They're fun. They're informative. They're useful in both the private and public sector. There's going to be a movement in that direction. How it all sorts out is interesting. To what extent is it going to destabilize the existing pundit hierarchy? To what extent is it going to destabilize who the big shots are within organizations?