📝 SummarXiv

Abstract

Smart homes, powered by programmable IoT platforms, often face safety and security issues. A class of defense solutions dynamically enforces policies that capture the expected behavior of the IoT system. Despite numerous innovations, these solutions are under-vetted. The primary reason lies in their evaluation approach -- they are self-assessed in isolated virtual testbeds with hand-crafted orchestrated scenarios that require manual interactions using the platform's user-interface (UI). Such non-uniform evaluation setups limit reproducibility and comparative analysis. Closing this gap in the traditional way requires a significant upfront manual effort, causing researchers to turn away from large-scale comparative empirical evaluation. To address this, we propose VetIoT -- a highly automated, uniform evaluation platform -- to vet the defense solutions that hinge on runtime policy enforcement. Given a defense solution, VetIoT readily instantiates a virtual testbed to deploy and evaluate the solution. VetIoT replaces manual UI-based interactions with an automated event simulator and manual inspection of test outcomes with an automated comparator. VetIoT incorporates automated event generators to feed events to the event simulator. We developed a prototype of VetIoT, which successfully reproduced and comparatively assessed four runtime policy enforcement solutions. VetIoT's stress testing and differential testing capabilities make it a promising tool for future research and evaluation.