Current models of visual perception suggest that during scene categorization, low spatial frequencies (LSF) are rapidly processed and activate plausible interpretations of visual input. This coarse analysis would be used to guide subsequent processing of high spatial frequencies (HSF). The present study aimed to further examine how information from LSF and HSF interact during scene categorization. We used hybrid scenes as stimuli by combining LSF and HSF from two different scenes which were semantically similar or dissimilar. Hybrids were presented 100 or 1000 ms and participants had to attend and categorize either the LSF or HSF scene. Results showed impaired categorization when the two scenes were semantically dissimilar, indicating that the semantic content of the unattended scene interfered with the categorization of the attended one. At short exposure duration (100 ms), this semantic interference effect was greater when participants attended HSF than LSF scenes, suggesting that information from LSF interfered more with HSF categorization than HSF did with LSF categorization. This reversed at longer exposure duration (1000 ms) where HSF interfered more with LSF categorization. These results suggest that the relative weight of LSF and HSF content varies over time during scene categorization, in accordance with a coarse-to-fine processing sequence.