The classic song "When Johnny Comes Marching Home" is used heavily throughout Die Hard With a Vengeance.
Why is it used so much? What connection does the song have with the film?
First of all, as Oliver_C points out in his comment,
And especially this song with its style of a military march (and the extensive use of drums throughout the movie) fits well to the military background and demeanor of the bad guys, who are more than one time depicted as being (or at least seeing themselves as) soldiers and behaving accordingly:
Add to this that the song is basically about returning from war, and even more so the more pessimistic version of Johnny I Hardly Knew Ye given in dglickler's answer. So it's motifs again fit to the bad guys' background of being ex-soldiers. Seeing that they are depicted as former GDR soldiers and the movie is likely set around its release in 1995, they might feel like having been taken away their homeland and with that basically their existence as soldiers. So they more or less feel a bit like disillusioned veterans returning from war (albeit a rather cold one), not knowing what to do now or what to fight for and trying to get their share from the world. This is also emphasized in Simon's speech at the end:
There's two real links.
First: "When Johnny Comes Marching Home" is also known by another name: "Johnny I Hardly Knew Ye." In both versions of the song, more the second, Johnny is a man who has returned from war...missing many pieces. The song strips away piece by piece of his body, leaving him a lump. The one who he comes back to is left angry, resentful, and hurtful. "Ye haven't an arm and ye haven't a leg, haroo, haroo."
The tie-in there to Die Hard With A Vengeance relates partly to the way the enemy (Simon) picks away at him to try to win (the guards, the false leads, etc.) and partly to the fact that Simon is angry, vengeful over his loss of his brother.
The interesting thing is that this song was left off of the soundtrack.
For interest, the lyrics of "Johnny I hardly knew ye" are: