The thickness of graphene films can be accurately determined by optical contrast spectroscopy. However, this becomes challenging and complicated when the flake size reduces to the micrometer scale, where the contrast spectrum is sensitively dependent on the polarization and incident angle of light. Here, we report accurate measurement of the optical contrast spectra of micrometer-sized few-layer graphene flakes on Au substrate. Using a high-resolution optical microscopy with a 100× magnification objective, we accurately determined the layer numbers of flakes as small as one micrometer in lateral size. We developed a theoretical model to accurately take into account the appropriate contribution of light incident at various angles and polarizations, which matched the experimental results extremely well. Furthermore, we demonstrate that the optical contrast spectroscopy is highly sensitive to detect the adsorption of submonolayer airborne hydrocarbon molecules, which can reveal whether graphene is contaminated. Though the technique was demonstrated on graphene, it can be readily generalized to many other two-dimensional materials, which opens new avenues for developing miniaturized and ultrasensitive label-free molecular sensors.