AEAD modes like GCM are authenticated encryption with associated data; this setting only affects the associated data half of that. The ciphertext itself is still authenticated. The associated data portion is there to provide contextual information for the authentication of the ciphertext. Usually this data is something that's outside of direct control of the party providing the ciphertext to be decrypted.
As a contrived example, it could be the database ID of the user whose data is encrypted. This way, even if another user found a way to copy a victim's secret key and ciphertext into his own account, he wouldn't be able to successfully decrypt the ciphertext since the user IDs would not be the same. Alternatively, in something like an over-the-wire communication protocol, a sequence number could be used as the associated data. This would prevent some types of replay attacks at the cryptographic layer — any attempt to replay a ciphertext would fail, since the other side can reject packets with duplicated sequence numbers, and any other sequence number would fail to decrypt the cryptogram.
To get around to answering your question, "less secure" than what? Not using authenticated data doesn't render the underlying ciphertext inherently weaker than had you used another, non-AEAD mode. But if there's additional context that could be included and you're not using it, it could be more secure if you did use it.
As always, it's important to understand what your threat model is before you involve cryptography. What are you trying to protect? From whom are you protecting it? What are the possible threats, and the vectors an attacker might use to compromise the protected data?