Some of the jokes work better if you know the history of the players, though Letterman at least does a good job explaining their credentials.
A couple things might help:
Randy Johnson's nickname (he's 6'10") was The Big Unit.
And pitcher Pedro Martinez of the Boston Red Sox (the huge rivals of the New York Yankees)( once famously said -- after being badly beaten by the Yankees a few times in a row -- that the Yankees were his "daddy." From that point on for the rest of the career, whenever Martinez would play in Yankee Stadium, New York fans would rhythmically chant along with the organ playing , "Who's...Your...Da-ddy?"
There's one other joke from Randy Johnson that won't make sense, but they insert a video immediately after to explain.