Real-world data often conceal meaningful signals beneath both random and structured noise. Structured noise arises in many settings, from batch effects in biomedical studies to background variation in image classification. Interestingly, algorithms that encourage diversity or uniformity in their learned representations tend to generalize better across contexts. To investigate this phenomenon, we study linear representation learning with two views, comparing classical and contrastive methods, both with and without a uniformity constraint. We find that classical non-contrastive algorithms fail in the presence of structured noise. Contrastive learning with only an alignment loss performs well when background variation is mild but breaks down under strong structured noise. In contrast, contrastive learning that enforces a uniformity constraint remains robust regardless of the magnitude of the background noise. Building on these insights, we further explore strategies for designing algorithms that maintain robustness under broader conditions, including random noise and nonstationary environments, by appropriately augmenting the data and problem conditions.