Events & Conferences
ICLR: Why does deep learning work, and what are its limits?
At this year’s International Conference on Learning Representations (ICLR), René Vidal, a professor of radiology and electrical engineering at the University of Pennsylvania and an Amazon Scholar, was a senior area chair, overseeing a team of reviewers charged with evaluating paper submissions to the conference. And the paper topic that his team focused on, Vidal says, was the theory of deep learning.
“While representation learning and deep learning have been incredibly successful and have produced spectacular results for many application domains, deep networks remain black boxes,” Vidal explains. “How you design deep networks remains an art; there is a lot of trial and error on each and every dataset. So by and large, the area of mathematics of deep learning aims to have theorems, mathematical proofs, that guarantee the performance of deep networks.
“You can ask questions such as ‘Why is it the case that deep networks generalize from one data set to another?’ ‘Can you have a theorem that tells you the classification error on a new dataset versus the classification error on the training data set?’ ‘Can you derive a bound on that error as, say, a function of the number of training examples?’
“There are questions that pertain to optimization. These days, you are minimizing a loss function over, sometimes, billions of parameters. And because the optimization problems are so large, and you have so many training examples, for computational reasons, you are limited to very simple optimization methods. Can you prove convergence for these nonconvex problems? Can you understand what you converge to? Why is it the case that these very simple optimization methods are so successful for these very complex problems?’”
Double descent
In particular, Vidal says, two topics in the theory of deep learning have been drawing increased attention recently. The first is the so-called double-descent phenomenon. The conventional wisdom in AI used to hold that the size of a neural network had to be carefully tailored to both the problem it addressed and the amount of training data available. If the network was too small, it couldn’t learn complex patterns in the data; but if it got too large, it could simply memorize the correct answers for all the data in its training set — a particularly egregious case of overfitting — and it wouldn’t generalize to new inputs.
As a consequence, for a given problem and a given set of training data, as the size of a neural network grows, its error rate on the previously unseen data of the test set goes down. At some point, however, the error rate starts to go up again, as the network begins to overfit the data.
In the last few years, however, a number of papers have reported the surprising result that as the network continues to grow, the error rate goes back down again. This the double-descent phenomenon — and no one is sure why it happens.
“The error goes down as the size of the model grows, then back up as it overfits,” Vidal explains. “And it gets to a peak at the so-called interpolation limit, which is exactly when, during training, you can achieve zero error, because the network is big enough that it can memorize. But from then on, the testing error goes down again. There have been a lot of papers trying to explain why this happens.”
The neural tangent kernel
Another interesting recent trend in the theory of deep networks, Vidal says, involves new forms of analysis based on the neural tangent kernel.
“In the past — say, the year 2000 — the way we did learning was by using so-called kernel methods,” Vidal explains. “Kernel methods are based on taking your data and embedding it with a fixed embedding into a very-high-dimensional space, where everything looks linear. We can use classical linear learning techniques in that embedding space, but the embedding space was fixed.
“You can think of deep learning as learning that embedding — mapping the input data to some high-dimensional space. In fact, that’s exactly representation learning. The neural-tangent-kernel regime — a type of initialization, a type of neural network, a type of training — is a regime under which you can approximate the learning dynamics of a deep network using kernels. And therefore you can use classical techniques to understand why they generalize and why not.
“That regime is very unrealistic — networks with infinite width or initializations that don’t change the weights too much during training. In this very contrived and specialized setting, things are easier and we can understand them better. The current trend is how we go away from these unrealistic assumptions and acknowledge that the problem is hard: you do want weights to change during training, because if they don’t, you’re not learning much.”
Indeed, Vidal has engaged this topic himself, in a paper accepted to this year’s Conference on Artificial Intelligence and Statistics (AISTATS), whose coauthors are his old research team from Johns Hopkins University.
“The three assumptions we are trying to get rid of are, one, can we get theorems for networks with finite width as opposed to infinite width?” Vidal says. “Number two is, can we get theorems for gradient-descent-like methods that have a finite step size? Because many earlier theorems assumed a really teeny tiny step size — like, infinitesimally small. And the third assumption we are relaxing is this assumption on the initialization, which becomes much more general.”
The limits of representation learning
When ICLR was founded, in 2013, it was a venue for researchers to explore alternatives to machine learning methods, such as kernel methods, that represented data in fixed, prespecified ways. Now, however, deep learning — which uses learned representations — has taken over the field of machine learning, and the difference between ICLR and the other major machine learning conferences has shrunk.
As someone who spent 20 years as a professor of biomedical engineering at Hopkins, however, Vidal has a keen awareness of the limitations of representation learning. For some applications, he says, domain knowledge is still essential.
“It happens in domains where data or labels may not be abundant,” he explains. “This is the case, for example, in the medical domain, where maybe there are 100 patients in a study, or maybe you can’t put the data on a website where everyone can annotate it.
“Just to give you one concrete example, I had a project where we needed to produce a blood test, and we needed to classify white blood cells into different kinds. No one is ever going to take videos of millions of cells, and you’re not going to have a pathologist annotate each and every cell to do object detection the way we do in computer vision.
“So all we could get were the actual results of the blood test: what are the concentrations? And you might have a million cells of class one, class two, and class three, and you just have these very weak labels. But the domain experts said, we can do cell purification by adding these chemicals here and there, and we do centrifugation and I don’t know what, and then we get cells of only one type in this specimen. Therefore you can now pretend that you have labels, because we know that cells that had different labels didn’t survive this chemistry. And we said, ‘Wow, that’s great!’
“If you do things with 100% people who are all data scientists and machine learning people, they tend to think that all you need is a bigger network and more data. But I think, as at Amazon, where you need to think backwards from the customer, you need to solve real problems, and the solution isn’t always more data and more annotations.”
- We’re sharing how Meta is applying machine learning (ML) and diversity algorithms to improve notification quality and user experience.
- We’ve introduced a diversity-aware notification ranking framework to reduce uniformity and deliver a more varied and engaging mix of notifications.
- This new framework reduces the volume of notifications and drives higher engagement rates through more diverse outreach.
Notifications are one of the most powerful tools for bringing people back to Instagram and enhancing engagement. Whether it’s a friend liking your photo, another close friend posting a story, or a suggestion for a reel you might enjoy, notifications help surface moments that matter in real time.
Instagram leverages machine learning (ML) models to decide who should get a notification, when to send it, and what content to include. These models are trained to optimize for user positive engagement such as click-through-rate (CTR) – the probability of a user clicking a notification – as well as other metrics like time spent.
However, while engagement-optimized models are effective at driving interactions, there’s a risk that they might overprioritize the product types and authors someone has previously engaged with. This can lead to overexposure to the same creators or the same product types while overlooking other valuable and diverse experiences.
This means people could miss out on content that would give them a more balanced, satisfying, and enriched experience. Over time, this can make notifications feel spammy and increase the likelihood that people will disable them altogether.
The real challenge lies in finding the right balance: How can we introduce meaningful diversity into the notification experience without sacrificing the personalization and relevance people on Instagram have come to expect?
To tackle this, we’ve introduced a diversity-aware notification ranking framework that helps deliver more diverse, better curated, and less repetitive notifications. This framework has significantly reduced daily notification volume while improving CTR. It also introduces several benefits:
- The extensibility of incorporating customized soft penalty (demotion) logic for each dimension, enabling more adaptive and sophisticated diversity strategies.
- The flexibility of tuning demotion strength across dimensions like content, author, and product type via adjustable weights.
- The integration of balancing personalization and diversity, ensuring notifications remain both relevant and varied.
The Risks of Notifications without Diversity
The issue of overexposure in notifications often shows up in two major ways:
Overexposure to the same author: People might receive notifications that are mostly about the same friend. For example, if someone often interacts with content from a particular friend, the system may continue surfacing notifications from that person alone – ignoring other friends they also engage with. This can feel repetitive and one-dimensional, reducing the overall value of notifications.
Overexposure to the same product surface: People might mostly receive notifications from the same product surface such as Stories, even when Feed or Reels could provide value. For example, someone may be interested in both reel and story notifications but has recently interacted more often with stories. Because the system heavily prioritizes past engagement, it sends only story notifications, overlooking the person’s broader interests.
Introducing Instagram’s Diversity-Aware Notification Ranking Framework
Instagram’s diversity-aware notification ranking framework is designed to enhance the notification experience by balancing the predicted potential for user engagement with the need for content diversity. This framework introduces a diversity layer on top of the existing engagement ML models, applying multiplicative penalties to the candidate scores generated by these models, as figure1, below, shows.
The diversity layer evaluates each notification candidate’s similarity to recently sent notifications across multiple dimensions such as content, author, notification type, and product surface. It then applies carefully calibrated penalties—expressed as multiplicative demotion factors—to downrank candidates that are too similar or repetitive. The adjusted scores are used to re-rank the candidates, enabling the system to select notifications that maintain high engagement potential while introducing meaningful diversity. In the end, the quality bar selects the top-ranked candidate that passes both the ranking and diversity criteria.
Mathematical Formulation
Within the diversity layer, we apply a multiplicative demotion factor to the base relevance score of each candidate. Given a notification candidate 𝑐, we compute its final score as the product of its base ranking score and a diversity demotion multiplier:
where R(c) represents the candidate’s base relevance score, and D(c) ∈ [0,1] is a penalty factor that reduces the score based on similarity to recently sent notifications. We define a set of semantic dimensions (e.g., author, product type) along which we want to promote diversity. For each dimension i, we compute a similarity signal pi(c) between candidate c and the set of historical notifications H, using a maximal marginal relevance (MMR) approach:
where simi(·,·) is a predefined similarity function for dimension i. In our baseline implementation, pi(c) is binary: it equals 1 if the similarity exceeds a threshold 𝜏i and 0 otherwise.
The final demotion multiplier is defined as:
where each wi ∈ [0,1] controls the strength of demotion for its respective dimension. This formulation ensures that candidates similar to previously delivered notifications along one or more dimensions are proportionally down-weighted, reducing redundancy and promoting content variation. The use of a multiplicative penalty allows for flexible control across multiple dimensions, while still preserving high-relevance candidates.
The Future of Diversity-Aware Ranking
As we continue evolving our notification diversity-aware ranking system, a next step is to introduce more adaptive, dynamic demotion strategies. Instead of relying on static rules, we plan to make demotion strength responsive to notification volume and delivery timing. For example, as a user receives more notifications—especially of similar type or in rapid succession—the system progressively applies stronger penalties to new notification candidates, effectively mitigating overwhelming experiences caused by high notification volume or tightly spaced deliveries.
Longer term, we see an opportunity to bring large language models (LLMs) into the diversity pipeline. LLMs can help us go beyond surface-level rules by understanding semantic similarity between messages and rephrasing content in more varied, user-friendly ways. This would allow us to personalize notification experiences with richer language and improved relevance while maintaining diversity across topics, tone, and timing.
Every day, millions of customers search for books in various formats (audiobooks, e-books, and physical books) across Amazon and Audible. Traditional keyword autocomplete suggestions, while helpful, usually require several steps before customers find their desired content. Audible took on the challenge of making book discovery more intuitive and personalized while reducing the number of steps to purchase.
We developed an instant visual autocomplete system that enhances the search experience across Amazon and Audible. As the user begins typing a query, our solution provides visual previews with book covers, enabling direct navigation to relevant landing pages instead of the search result page. It also delivers real-time personalized format recommendations and incorporates multiple searchable entities, such as book pages, author pages, and series pages.
Our system needed to understand user intent from just a few keystrokes and determine the most relevant books to display, all while maintaining low latency for millions of queries. Using historical search data, we match keystrokes to products, transforming partial inputs into meaningful search suggestions. To ensure quality, we implemented confidence-based filtering mechanisms, which are particularly important for distinguishing between general queries like “mystery” and specific title searches. To reflect customers’ most recent interests, the system applies time-decay functions to long historical user interaction data.
To meet the unique requirements of each use case, we developed two distinct technical approaches. On Audible, we deployed a deep pairwise-learning-to-rank (DeepPLTR) model. The DeepPLTR model considers pairs of books and learns to assign a higher score to the one that better matches the customer query.
The DeepPLTR model’s architecture consists of three specialized towers. The left tower factors in contextual features and recent search patterns using a long-short-term-memory model, which processes data sequentially and considers its prior decisions when issuing a new term in the sequence. The middle tower handles keyword and item engagement history. The right tower factors in customer taste preferences and product descriptions to enable personalization. The model learns from paired examples, but at runtime, it relies on books’ absolute scores to assemble a ranked list.
For Amazon, we implemented a two-stage modeling approach involving a probabilistic information-retrieval model to determine the book title that best matches each keyword and a second model that personalizes the book format (audiobooks, e-books, and physical books). This dual-strategy approach maintains low latency while still enabling personalization.
In practice, a customer who types “dungeon craw” in the search bar now sees a visual recommendation for the book Dungeon Crawler Carl, complete with book cover, reducing friction by bypassing a search results page and sending the customer directly to the product detail page. On Audible, the system also personalizes autocomplete results and enriches the discovery experience with relevant connections. These include links to the author’s complete works (Matt Dinniman’s author page) and, for titles that belong to a series, links to the full collection (such as the Dungeon Crawler Carl series).
On Amazon, when the customer clicks on the title, the model personalizes the right book-format (audiobooks, e-books, physical books) recommendation and directs the customer to the right product detail page.
In both cases, after the customer has entered a certain number of keystrokes, the system employs a model to detect customer intent (e.g., book title intent for Amazon or author intent for Audible) and determine which visual widget should be displayed.
Audible and Amazon books’ visual autocomplete provides customers with more relevant content more rapidly than traditional autocomplete, and its direct navigation reduces the number of steps to find and access desired books — all while handling millions of queries at low latency.
This technology is not just about making book discovery easier; it is laying the foundation for future improvements in search personalization and visual discovery across Amazon’s ecosystem.
Acknowledgements: Jiun Kim, Sumit Khetan, Armen Stepanyan, Jack Xuan, Nathan Brothers, Eddie Chen, Vincent Lee, Soumy Ladha, Justine Luo, Yuchen Zeng, David Torres, Gali Deutsch, Chaitra Ramdas, Christopher Gomez, Sharmila Tamby, Melissa Ma, Cheng Luo, Jeffrey Jiang, Pavel Fedorov, Ronald Denaux, Aishwarya Vasanth, Azad Bajaj, Mary Heer, Adam Lowe, Jenny Wang, Cameron Cramer, Emmanuel Ankrah, Lydia Diaz, Suzette Islam, Fei Gu, Phil Weaver, Huan Xue, Kimmy Dai, Evangeline Yang, Chao Zhu, Anvy Tran, Jessica Wu, Xiaoxiong Huang, Jiushan Yang
Modern warehouses rely on complex networks of sensors to enable safe and efficient operations. These sensors must detect everything from packages and containers to robots and vehicles, often in changing environments with varying lighting conditions. More important for Amazon, we need to be able to detect barcodes in an efficient way.
The Amazon Robotics ID (ARID) team focuses on solving this problem. When we first started working on it, we faced a significant bottleneck: optimizing sensor placement required weeks or months of physical prototyping and real-world testing, severely limiting our ability to explore innovative solutions.
To transform this process, we developed Sensor Workbench (SWB), a sensor simulation platform built on NVIDIA’s Isaac Sim that combines parallel processing, physics-based sensor modeling, and high-fidelity 3-D environments. By providing virtual testing environments that mirror real-world conditions with unprecedented accuracy, SWB allows our teams to explore hundreds of configurations in the same amount of time it previously took to test just a few physical setups.
Camera and target selection/positioning
Sensor Workbench users can select different cameras and targets and position them in 3-D space to receive real-time feedback on barcode decodability.
Three key innovations enabled SWB: a specialized parallel-computing architecture that performs simulation tasks across the GPU; a custom CAD-to-OpenUSD (Universal Scene Description) pipeline; and the use of OpenUSD as the ground truth throughout the simulation process.
Parallel-computing architecture
Our parallel-processing pipeline leverages NVIDIA’s Warp library with custom computation kernels to maximize GPU utilization. By maintaining 3-D objects persistently in GPU memory and updating transforms only when objects move, we eliminate redundant data transfers. We also perform computations only when needed — when, for instance, a sensor parameter changes, or something moves. By these means, we achieve real-time performance.
Visualization methods
Sensor Workbench users can pick sphere- or plane-based visualizations, to see how the positions and rotations of individual barcodes affect performance.
This architecture allows us to perform complex calculations for multiple sensors simultaneously, enabling instant feedback in the form of immersive 3-D visuals. Those visuals represent metrics that barcode-detection machine-learning models need to work, as teams adjust sensor positions and parameters in the environment.
CAD to USD
Our second innovation involved developing a custom CAD-to-OpenUSD pipeline that automatically converts detailed warehouse models into optimized 3-D assets. Our CAD-to-USD conversion pipeline replicates the structure and content of models created in the modeling program SolidWorks with a 1:1 mapping. We start by extracting essential data — including world transforms, mesh geometry, material properties, and joint information — from the CAD file. The full assembly-and-part hierarchy is preserved so that the resulting USD stage mirrors the CAD tree structure exactly.
To ensure modularity and maintainability, we organize the data into separate USD layers covering mesh, materials, joints, and transforms. This layered approach ensures that the converted USD file faithfully retains the asset structure, geometry, and visual fidelity of the original CAD model, enabling accurate and scalable integration for real-time visualization, simulation, and collaboration.
OpenUSD as ground truth
The third important factor was our novel approach to using OpenUSD as the ground truth throughout the entire simulation process. We developed custom schemas that extend beyond basic 3-D-asset information to include enriched environment descriptions and simulation parameters. Our system continuously records all scene activities — from sensor positions and orientations to object movements and parameter changes — directly into the USD stage in real time. We even maintain user interface elements and their states within USD, enabling us to restore not just the simulation configuration but the complete user interface state as well.
This architecture ensures that when USD initial configurations change, the simulation automatically adapts without requiring modifications to the core software. By maintaining this live synchronization between the simulation state and the USD representation, we create a reliable source of truth that captures the complete state of the simulation environment, allowing users to save and re-create simulation configurations exactly as needed. The interfaces simply reflect the state of the world, creating a flexible and maintainable system that can evolve with our needs.
Application
With SWB, our teams can now rapidly evaluate sensor mounting positions and verify overall concepts in a fraction of the time previously required. More importantly, SWB has become a powerful platform for cross-functional collaboration, allowing engineers, scientists, and operational teams to work together in real time, visualizing and adjusting sensor configurations while immediately seeing the impact of their changes and sharing their results with each other.
New perspectives
In projection mode, an explicit target is not needed. Instead, Sensor Workbench uses the whole environment as a target, projecting rays from the camera to identify locations for barcode placement. Users can also switch between a comprehensive three-quarters view and the perspectives of individual cameras.
Due to the initial success in simulating barcode-reading scenarios, we have expanded SWB’s capabilities to incorporate high-fidelity lighting simulations. This allows teams to iterate on new baffle and light designs, further optimizing the conditions for reliable barcode detection, while ensuring that lighting conditions are safe for human eyes, too. Teams can now explore various lighting conditions, target positions, and sensor configurations simultaneously, gleaning insights that would take months to accumulate through traditional testing methods.
Looking ahead, we are working on several exciting enhancements to the system. Our current focus is on integrating more-advanced sensor simulations that combine analytical models with real-world measurement feedback from the ARID team, further increasing the system’s accuracy and practical utility. We are also exploring the use of AI to suggest optimal sensor placements for new station designs, which could potentially identify novel configurations that users of the tool might not consider.
Additionally, we are looking to expand the system to serve as a comprehensive synthetic-data generation platform. This will go beyond just simulating barcode-detection scenarios, providing a full digital environment for testing sensors and algorithms. This capability will let teams validate and train their systems using diverse, automatically generated datasets that capture the full range of conditions they might encounter in real-world operations.
By combining advanced scientific computing with practical industrial applications, SWB represents a significant step forward in warehouse automation development. The platform demonstrates how sophisticated simulation tools can dramatically accelerate innovation in complex industrial systems. As we continue to enhance the system with new capabilities, we are excited about its potential to further transform and set new standards for warehouse automation.
-
Business1 week agoThe Guardian view on Trump and the Fed: independence is no substitute for accountability | Editorial
-
Tools & Platforms4 weeks ago
Building Trust in Military AI Starts with Opening the Black Box – War on the Rocks
-
Ethics & Policy1 month agoSDAIA Supports Saudi Arabia’s Leadership in Shaping Global AI Ethics, Policy, and Research – وكالة الأنباء السعودية
-
Events & Conferences4 months agoJourney to 1000 models: Scaling Instagram’s recommendation system
-
Jobs & Careers2 months agoMumbai-based Perplexity Alternative Has 60k+ Users Without Funding
-
Education2 months agoVEX Robotics launches AI-powered classroom robotics system
-
Funding & Business2 months agoKayak and Expedia race to build AI travel agents that turn social posts into itineraries
-
Podcasts & Talks2 months agoHappy 4th of July! 🎆 Made with Veo 3 in Gemini
-
Education2 months agoMacron says UK and France have duty to tackle illegal migration ‘with humanity, solidarity and firmness’ – UK politics live | Politics
-
Podcasts & Talks2 months agoOpenAI 🤝 @teamganassi