ROADS to Mega-AI Models Workshop

Appearance

On this page

Research On Algorithms & Data Structures (ROADS) to Mega-AI Models Workshop ​

Workshop Summary ​

The state-of-the-art on numerous machine learning (ML) benchmarks comes from training enormous neural network models on expensive, specialized hardware with massive quantities of data. However, this route to success in deep learning is unsustainable. Training a large transformer model in natural language processing, for instance, can incur a higher carbon footprint than the total lifetime cost of five cars. In addition, these massive models require immense memory and computing resources during deployment, which hinders their practical impact. To realize the full promise and benefits of artificial intelligence, we must solve these scalability challenges prevalent in both training and inference and design new algorithms with step-function improvements in efficiency. This workshop aims to bring together both computer science researchers and practitioners focused on ML efficiency to offer innovative solutions towards efficient modeling workflows grounded in principled algorithm design.

Keynote Speakers ​

Schedule (Thursday, June 8) ​

  • 8:30am-8:40am - Welcome and Opening Remarks (Workshop Organizers)
  • 8:40am-9:40am - Keynote #1: Prof. Michael Mitzenmacher (Harvard)
  • 9:40am-10:05am - Invited Talk: Hongyi Wang Cuttlefish: Low-Rank Model Training without All the Tuning
  • 10:05am-10:30am - Invited Talk: Daochen Zha Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Models
  • 10:30am-10:50am - Coffee Break and Socialize
  • 10:50am-11:30am - Keynote #2: Dr. Bilge Acun (Meta)
  • 11:30am-11:55am - Invited Talk: Trevor Gale MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
  • 11:55am-12:20pm - Invited Talk: Daochen Zha RSC: Accelerating Graph Neural Networks Training via Randomized Sparse Computations
  • 12:20pm-1:20pm - Lunch Break
  • 1:20pm-2:00pm - Keynote #3: Prof. Jonathan Frankle (Harvard)
  • 2:00pm-2:30pm - Keynote #4: Prof. Furong Huang (UMD)
  • 2:30pm-3:10pm - Keynote #5: Dr. Chen Luo (Amazon)
  • 3:10pm-3:30pm - Coffee Break and Socialize
  • 3:30pm-4:30pm - Panel Discussion (Mitzenmacher, Frankle, Acun, Luo, Shrivastava)
  • 4:30pm - Social

Organizing Committee ​

Call for Papers ​

This workshop encourages submissions on original research, benchmarks, in-progress research results, and position papers. We also invite submissions of previously accepted papers where authors had limited presentation opportunities (e.g. due to pandemic-related constraints). We do not have strict requirements on paper lengths, but encourage authors to adhere to a maximum of 10-pages (excluding references) in MLSys 2023 format. We will manage submissions through OpenReview, but the review-process will not be open.

Scope ​

The technical topics of interest at this workshop include (but are not limited to):

  • Algorithms and data structures to improve the computational efficiency of neural network training and inference
  • Algorithmic solutions for deploying machine learning models on resource-constrained devices
  • Model compression approaches for training and inference, including pruning, quantization, and parameter sharing
  • Data reduction techniques (e.g. sketching, sampling, coresets) for efficient training and inference
  • Algorithmic techniques to enable longer sequence language models, higher resolution images for vision models, wide and deep neural networks, and other model architectures of interest to the community

Submission ​

OpenReview Link

Please format submissions using the MLSys 2023 style files.

The reviewing process will be double-blind. Please submit anonymized papers that omit all information about author identities. There are no formal proceedings generated from this workshop. Accepted papers will be made available on OpenReview.

Contact ​

Contact the organizers: zx22@rice.edu