A method and system are disclosed for reducing perplexity in a speech recognition system within a telephonic network based upon determined caller identity. In a speech recognition system which processes input frames of speech against stored templates representing speech, a core library of speech templates is created and stored representing a basic vocabulary of speech. Multiple caller-specific libraries of speech templates are also created and stored, each library containing speech templates which represent a specialized vocabulary and pronunciations for a specific geographic location and a particular individual. Additionally, the caller-specific libraries of speech templates are preferably processed to reflect the reduced bandwidth, transmission channel variations and other signal variations introduced into the system via a telephonic network. The identification of a caller is determined upon connection to the network via standard caller identification circuitry and upon detection of a spoken utterance, that utterance is processed against the core library, if the caller's identity cannot be determined, or against a particular caller-specific library, if the caller's identity can be determined, thereby greatly enhancing the efficiency and accuracy of speech recognition by the system.