How come when sometimes I try to knife an enemy, my soldier will sometimes sit there holding the knife up instead of going through the knife animation?
This may have to do with the range and the angle of the knife attack. In order to get a takedown you need to come from either side or the back of the enemy. If you come from the front you will most likely just slash your opponent. You will slash the air if you happen to be out of range.
A second explanation is more likely though: You don't hold the button down long enough. A knife takedown required you to hold down the button instead of tapping it once. Source EDIT: This counts when using the melee button, when having the knife in hand (as the second commenter points out and is said in the source) you'll see a change in the way the knife is held when a takedown is ensured (vertical for takedown vs. horizontal for slash).