Existing reading comprehension assessments have been criticized by researchers, educators, and policy makers, especially regarding their coverage, utility, and authenticity. The purpose of the current study was to evaluate a new type of assessment of reading comprehension that was designed to broaden the construct of reading. In light of these issues, we developed a Scenario-Based Assessment (SBA) of reading comprehension that was inspired by the cognitive literature in reading, learning, and instruction. The SBA was designed to measure students’ ability to integrate and evaluate a set of thematically related sources for the purpose of achieving an overarching goal. The SBA also measured students’ ability to form an integrated and global mental model of the text, student background knowledge, and social interactions in a digital environment (e.g., perspective taking; classifying and evaluating simulated peer claims). A sample of 426 sixth grade students completed the SBA form and a subsample of 161 students completed a reading component skills battery. Results indicated adequate psychometric properties of the SBA, evidence generally in support of the alignment of the SBA to the assessment design, and strong correlations between the SBA and traditional reading comprehension assessments. While students were able to engage with a variety of complex tasks, items that measured students’ ability to form a coherent mental model (e.g., write a summary) and digital literacy tasks that required students to integrate perspectives from multiple texts were particularly difficult.