Xinchen Yang

Curriculum learning has been a quiet yet crucial component of many of the high-profile successes of reinforcement learning. Despite this, none of the major reinforcement learning libraries directly support curriculum learning or include curriculum learning implementations. These methods can improve the capabilities and robustness of RL agents, but often require significant, complex changes to agent training code. We introduce Syllabus, a library for training RL agents with curriculum learning, as a solution to this problem. Syllabus provides a universal API for curriculum learning algorithms, implementations of popular curriculum learning methods, and infrastructure for easily integrating them with distributed training code written in nearly any RL library. Syllabus provides a minimal API for each of the core components of curriculum learning, dramatically simplifying the process of designing new algorithms and applying existing algorithms to new environments. We demonstrate that the same Syllabus code can be used to train agents written in multiple different RL libraries on numerous domains. In doing so, we present the first examples of curriculum learning in NetHack and Neural MMO, two of the premier challenges for single-agent and multi-agent RL respectively, achieving strong results compared to state of the art baselines.

The Automated Machine Learning (AutoML) System has powered domain experts with its efficient model discovery capabilities and helped address shortages of qualified data scientists. Popular AutoML Systems include the AutoⁿML developed by the Auton Lab at Carnegie Mellon University (CMU), Tree-based Pipeline Optimization Tool (TPOT), AutoGluon, auto-sklearn, etc. We hope to learn about how the performances of AutoML systems compared to each other and how factors such as dataset characteristics and algorithm selection can affect AutoML performances. Motivated by this goal, we conducted experiments to evaluate the performance of 4 popular AutoML systems, including AutoⁿML, TPOT, AutoGluon and auto-sklearn on 270 OpenML regression tasks, using R-Squared score as the evaluation metric. We analyzed the experimental data from various aspects, including relative rankings of the AutoML systems, and relationship between performance of AutoML systems and dataset characteristics. we have also run repeating experiments for selected AutoML systems on a few selected datasets to analyze the stability of AutoML performances under the same condition.

The Automated Machine Learning (AutoML) System has powered data scientists and domain experts with its efficient model discovery capabilities and helped address shortages of qualified data scientists. Widely used AutoML Systems include the AutoⁿML developed by the Auton Lab at Carnegie Mellon University (CMU), H₂O AutoML, Auto-Sklearn, etc. Various performance evaluations have been done on these AutoML systems. However, as AutoML systems get updated with improved model searching ability and newly added functionalities, we need to obtain a new map to depict the performance of the AutoML systems. Motivated by this goal, we conducted experiments to evaluate the performance of 4 popular AutoML systems, including AutoⁿML, H₂O AutoML, TPOT and AutoGluon, on 177 OpenML binary-classification tasks, using Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC) curve as the evaluation metric. We analyzed the experimental data from various aspects, including the relative rankings of the AutoML systems, training- testing performance discrepancies, relationship between the performance of AutoML systems and dataset characteristics, and the winning algorithms used by each AutoML system.