Coffee is the most consumed processed beverage aside from water, and green coffee beans are the most traded agriculture commodity after petroleum in the world. The agricultural production of green coffee beans and consumption of coffee have been increasing by 17% and 2% at an annual rate during the previous decades, respectively. The credit of increasing coffee production and consumption goes to its alluring organoleptic characteristics. The organoleptic or final cup quality characteristic of coffee is a multifactorial and complex trait, and both agricultural and postharvest processing factors influence this multifaceted trait significantly. Agroproduction technology of coffee influences 40% cup quality attributes of coffee beverage, whereas remaining 60% quality attributes are determined by postharvest processing technology. In this chapter, the relationship of organoleptic or final cup quality attributes with agricultural and environmental factors was reviewed. The analysis focused on how these factors affect the physical quality attributes of coffee beans in addition to the biochemical cup quality attributes. An overview of agricultural and environmental factors of coffee identified a critical impact of these factors in determining the physical and biochemical cup quality attributes. Geographical topography (especially altitude, slope of attitude, its steepness) was found to be the major element which also dictated the scope of influence of subsequent agricultural and environmental factors. Coffee verities or genetics, rainfall, frost, temperature, soil fertilization status, sun and shade ecosystems, and harvesting strategies played a decisive role in shaping not only the final physical and biochemical cup quality attributes but also in postharvest processing approaches. Each coffee variety (both C. arabica and C. robusta) is specified to a specific region with a set of its own inherent quality characteristics which played an important role in the production of certified specialty, organic, or other same kind of coffees. Moreover, there are still some bottlenecks that need to be addressed in order to fully understand the critical relationship of agricultural and environmental factors with final physical and biochemical cup quality attributes.