There is a huge difference between a nation's flag or Pledge of Allegiance and the Inaugural Prayer.
Jesus christ, you have to see our flag wherever you go and if you are in school or go to sporting events you have to deal with the pledge or the national anthem.
The inaugural prayer, however, is something that, at most, you have to deal with every four years and even then, only if you find watching a dude on television with his hand on the Bible to be somewhat entertaining.
To be honest, I didn't even know the Inaugural Prayer existed before the Warren thing popped up so I really doubt is has any huge symbolic value.