Influenza A viruses (IAVs) evolve continuously, often leading to antigenic drift - the accumulation of substitutions in viral surface proteins that alter antigenic properties and enable immune evasion. Here, we have investigated the antibody response to recent seasonal influenza viruses and used serologic assays to find correlates of protection (CoP) in influenza-infected individuals that sought care at the Hospital of the University of Pennsylvania (HUP) during the 2019-2020, 2021-2022, and 2022-2023 seasons. We collected nasopharyngeal (NP) swabs and serum samples from individuals infected with H1N1 virus in the 2019-2020 season. We sequenced viruses isolated from NP swabs and used serum samples to perform hemagglutination inhibition assays with vaccine and circulating virus strains to assess antibody specificities. We collected serum samples from a control group infected with influenza B virus (IBV) that presented at the same hospital during the same time frame. We found that antibodies from H1N1 and IBV infected individuals had higher titers against the H1N1 vaccine strain compared to circulating viruses, with evidence of antigenic drift leading to reduced antibody recognition. H3N2 viruses were dominant in the 2021-2022 and 2022-2023 seasons in our samples collected from HUP and nationally. We collected NP swabs and serum from infected individuals at HUP and sequenced H3N2 viral genomes. Using multiple serologic assays, we assessed influenza-binding and neutralizing antibody levels. In the 2021-2022 season, a significant antigenic mismatch between vaccine and circulating H3N2 strains was identified, likely due to a specific substitution in HA antigenic site B that led to a loss of glycosylation. This mismatch corresponded with low antibody titers against circulating strains in infected and control individuals. Logistic regression analyses demonstrated that higher antibody titers correlated with reduced odds of influenza infection, particularly in the 2022-2023 season when the antigenic mismatch was less pronounced. Our findings highlight the impact of antigenic drift on protection from influenza and the importance of standardized serologic assays for assessing immune correlates of protection. This study underscores the need for new influenza vaccine strategies, particularly in high-risk populations.