We report a lexical decision task experiment, in which words were manipulated such that two different sounds had been switched with each other: the voiceless pharyngeal and uvular fricatives. The former is a marked sound of some dialects of Modern Hebrew, the latter is a merged category corresponding to both the historical pharyngeal and the uvular in the production of most speakers. The two categories are represented by different letters in the orthographic system and each is associated with unique phonological processes. Socially, the pharyngeal is stereotyped; merging the categories is both more common and more prestigious in most social contexts. Speakers of Modern Hebrew with varied linguistic backgrounds, including Merged speakers who have not been exposed to non-merged dialects during most of their lives, are very good at acoustically distinguishing between these sounds (only slightly underperforming compared with Non-merged speakers). Nevertheless, we found that manipulated stimuli - which were not part of the input for language learners of either dialect - provoke different acceptance rates and reaction times, depending on the listener's home dialect, in certain cases regardless of their production grammar. In particular, Non-merged speakers and Merged speakers who are 2nd generation listeners to non-merged dialects rejected switched category items at much higher rates and took longer to process them compared with Merged speakers who did not have early experience with the categorical distinction. We discuss these findings in the context of models of phonological representation and auditory word recognition.