In ordinary conversations, literal meanings of an utterance are often quite different from implicated meanings and the inference about implicated meanings is essentially required for successful comprehension of the speaker's utterances. Inference of finding implicated meanings is based on the listener's assumption that the conversational partner says only relevant matters according to the maxim of relevance in Grice's theory of conversational implicature. To investigate the neural correlates of comprehending implicated meanings under the maxim of relevance, a total of 23 participants underwent an fMRI task with a series of conversational pairs, each consisting of a question and an answer. The experimental paradigm was composed of three conditions: explicit answers, moderately implicit answers, and highly implicit answers. Participants were asked to decide whether the answer to the Yes/No question meant 'Yes' or 'No'. Longer reaction time was required for the highly implicit answers than for the moderately implicit answers without affecting the accuracy. The fMRI results show that the left anterior temporal lobe, left angular gyrus, and left posterior middle temporal gyrus had stronger activation in both moderately and highly implicit conditions than in the explicit condition. Comprehension of highly implicit answers had increased activations in additional regions including the left inferior frontal gyrus, left medial prefrontal cortex, left posterior cingulate cortex and right anterior temporal lobe. The activation results indicate involvement of these regions in the inference process to build coherence between literally irrelevant but pragmatically associated utterances under the maxim of relevance. Especially, the left anterior temporal lobe showed high sensitivity to the level of implicitness and showed increased activation for highly versus moderately implicit conditions, which imply its central role in inference such as semantic integration. The right hemisphere activation, uniquely found in the anterior temporal lobe for highly implicit utterances, suggests its competence for integrating distant concepts in implied utterances under the relevance principle.