AWS SES - Able to receive few emails but NOT ALL, very random

Question

I'm trying to receive emails using AWS SES and trying to process them through SNS/SQS, it works fine for few (20-30% cases), but not all.

Below is the current setup:

a) MX record has been set up on AWS Route 53.

b) 1 Email receipt rule has been set up to point to SNS topic, and SQS is subscribing to SNS, so that email messages are visible in SQS queue.

Now when emails are attempted to receive, some emails end up in SQS queue and some don't. The ones which do not end up in SQS queue, they are visible on the original mail server whose MX record was set up on AWS route 53.

Exactly same email from same email address with same text will land up in SQS queue, and after 1 minute on trying again will not land up in SQS queue, and will be visible on original mail server. So it is very random.

I have even unticked "Apply spam rules" in the email receipt rules, in case some emails are being refused by receipt rules because they are being identified as spam.

Searched through the forums, unable to find any help.

Any hint or help some one can provide will be much appreciated!!