A central factor in trustworthy autonomous systems is the presence of humans in different stages of an AI system’s deployment cycle - from data curation, to model training, evaluation, and deployment. In real world applications, an AI system requires different experts in each of these stages. For instance, in medical applications, doctors evaluate the decisions made by AI systems but are not well versed with its training. ML engineers can select the best possible models but may not be able to curate unbiased data for these models. Existing human-in-the-loop systems do not account for this variety in the required expertise. Since deep learning based AI systems require many specialized experts, their applicability is limited to only a few non-specialized applications. At OLIVES, we leverage limited experts to train large scalable systems that learn from limited data. We move away from a generic human-in-the-loop framework towards experts-in-the-loop.