Concerning this, the Lord states in the Vedic literature:

mām anārādhya duḥkhārtaḥ
kuṭumbāsakta-mānasaḥ
sat-saṅga-rahito martyo
vṛddha-sevā-paricyutaḥ

"A person who does not worship Me, who is unduly attached to family and who does not stick to devotional service must be considered a most unhappy person. Similarly, one who does not associate with Vaiṣṇavas, or who does not render service to his superior, is also a most unhappy person."

There is also a statement given in the Bṛhad-bhāgavatāmṛta (1.5.51):

sva-jīvanādhikaṁ prārthyaṁ
śrī-viṣṇu-jana-saṅgataḥ
vicchedena kṣaṇaṁ cātra
na sukhāṁśaṁ labhāmahe

"[King Yudhiṣṭhira said:] 'Out of all kinds of desirable things experienced in the life of a living entity, association with the devotees of the Lord is the greatest. When we are separated from a devotee even for a moment, we cannot enjoy happiness.' "