Randomized experiments – often called A/B tests in industrial settings – are an increasingly important element in the management of many organizations. While some firms have long had both the managerial and technical know-how to use experiments for making key decisions, new forms of software and internet infrastructure have dramatically lowered the cost of conducting A/B tests online, opening up the practice to an entirely new set of organizations. This dissertation studies the practice of A/B testing among this new wave of practitioners, characterized primarily as e-commerce businesses that have adopted new forms of low cost, easy to use, third-party experimentation software. The first two chapters of this document study A/B testing as its own distinct phenomena in digital business, answering questions about the prevalence of p-hacking among e-commerce practitioners and the nature of how firms use A/B testing software in the real world. The final chapter demonstrates how e-commerce firms can use A/B tests and recent developments in causal machine learning for improved customer targeting and price discrimination. As a whole, this work demonstrates the growing importance of A/B testing and causal reasoning as a key factor in the future of managerial decision making.