Computer-based technologies are now commonplace in classrooms, and the integration of these media into the teaching and learning of mathematics is supported by government policy in most developed countries. However, many questions about the impact of computer-based technologies on classroom mathematics learning remain unanswered, and debates about when and how they ought to be used continue. An increasing number of studies seek to identify the effects of technology usage on classroom learning, and at a time when governments are calling for 'evidence-based' policy development, many studies applying quasi- scientific methodologies to this field of practice are emerging. By analysing a series of conceptual frameworks for assessing the use of computer-based technologies to support school learning, this article emphasises the value of research into the relationship between technical and conceptual aspects of technology use in mathematics education and beyond, and challenges the usefulness of large-scale, quasi-scientific studies that focus on educational inputs and outputs.