Have a look at tcpdump. It can dump all network traffic (not only tcp as the name would suggest), which you can then read with an application like Wireshark. In Wireshark it's very easy to filter certain types of data and even plot graphs of the network I/O.
Another useful tool might be netstat which displays a list of ongoing network connections. Perhaps there are connections that shouldn't be there. Tcpdump is much more useful (capture a few minutes, then check if you can already see the source), but netstat might give you a quick overview.
Upon reading this by the way, my first thoughts are that you have malware on your server or that it is being used for amplification attacks. But to examine this you'll need to run tcpdump first.
Edit: Note that tcpdump probably needs to be run as root, perhaps you need to use
Another edit: Since I can't really find a good webpage to link on what amplification attacks are in general, here's a short version:
Protocols like DNS run on UDP. UDP traffic is connectionless, and thus you can very easily spoof the IP address of someone else. Because a DNS answer is usually larger than the query, this can be used for a DoS attack. The attacker sends a query requesting all records the DNS server has on a given name, and tells the DNS server that the request originated from X. This X is the target that the attacker wants to DoS. The DNS server then kindly replies, sending the (big, say 4kB) reply to X.
This is amplification because the attacker sends less data than X actually receives. DNS is not the only protocol with which this is possible.