The 38-year-old could have gone to the US Supreme Court after his bid to overturn the suspension was rejected by a United States appeals court.
But Brady announced on Facebook that he would "no longer proceed with the legal process."
He was punished by the NFL for his role in a scheme to deflate match balls to give his side an advantage.
Brady, who will miss the first four matches when the new season gets under way in September, denies he or the club did anything wrong during an NFL play-off game.
In his Facebook post, he said it had been a "challenging" 18 months but added he would work hard "to be the best player he can be" on his return to action in the autumn.
Brady was originally suspended in 2015 and was unsuccessful in his first appeal but the decision was later overturned by a federal court.
However this suspension was restored by the Court of Appeal in April after it was challenged by the NFL.
New England Patriots quarterback Tom Brady will not continue to fight his four-match "Deflate-gate" ban.