Optimising retail food environments to make healthy options available, affordable, accessible and appealing is recognised as a promising pathway to improving population diet and health outcomes. Appraisal of the retail food environment can be a catalyst for retailers to take action by allowing benchmarking against other stores and over time, and identification of areas for improvement. We developed a mobile app tool (Store Scout) to enable rapid assessment of the retail food environment against best practice, with results provided immediately as scores ranging from 0-100 (higher more health-enabling) overall and for seven food groups. We aimed to assess validity and sensitivity to change of this scoring system using measurements from Healthy Stores 2020 (HS2020), which tested a strategy restricting promotion and reducing visibility of unhealthy foods in 20 NT/QLD remote stores (10 control; 10 intervention). Sensitivity to change was assessed using t-test of change in scores from baseline to follow-up in intervention versus control stores, with expectation of increased drinks and snack foods scores with HS2020 implementation. Validity was assessed using linear regression of scores versus unhealthy foods sales using electronic sales data (last 4-weeks of study period) expressed as grams sold of discretionary products per total megajoule energy from all products (g/MJ). Implementation of the HS2020 strategy resulted in increased scores (more health-enabling) for overall (mean difference 4.8 [95% CI 1.1-8.5] p=0.020), drinks (16.0 [5.5-26.6] p=0.008) and snack foods (12.8 [5.6-20.1] p=0.003) scores, while other food group scores were not significantly changed. A one-point increase in Store Scout score was associated with reduced unhealthy food and beverage purchasing at baseline (-1.44 g/MJ; 95% CI -2.69,-0.19; R2 =0.22) and change from baseline to follow-up (-0.90 (-1.31,-0.50) R squared =0.52). These findings indicate potential of Store Scout for enabling benchmarking of the consumer food environment between stores and over time.