We present BotSIM, a data-efficient end-to-end Bot SIMulation framework for commercial task-oriented dialog (TOD) systems. BotSIM consists of three major components: 1) a Generator that can infer semantic-level dialog acts and entities from bot definitions and generate conversations via model-based paraphrasing; 2) an agenda-based dialog user Simulator to communicate with the dialog agents; 3) a Remediator to analyze and visualize the bot health reports and provide actionable remediation suggestions for troubleshooting and improving the dialog system. We demonstrate BotSIM's effectiveness in end-to-end evaluation, remediation and multi-intent dialog generation via case studies on two commercial bot platforms. BotSIM's "generation-simulation-remediation'" paradigm accelerates the end-to-end bot evaluation and iteration process by: 1) reducing the effort needed to create test cases; 2) enabling a better understanding of both NLU and end-to-end performance via extensive dialog simulation; 3) improving the bot troubleshooting process with actionable suggestions from simulation results analysis. A demo of our system can be found at https://tinyurl.com/mryu74cd and a demo video at https://youtu.be/qLPJm6_UOKY.