Events & Conferences

CVPR: Deep learning has more gas in the tank

Published

5 years ago

June 11, 2020

The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), which will be held virtually this year and starts next week, is the major conference in the field of computer vision. But when Larry Davis, senior principal scientist for Amazon Fashion, started attending it, “computer vision” wasn’t even part of its title.

“I think 1981 was my first time having a formal role at the conference,” Davis says. “Back then the meeting was called Pattern Recognition and Image Processing. My advisor [Azriel Rosenfeld] was pretty much regarded as the founder of the field of computer vision, but when he founded it, he called it ‘picture processing’. Then he and a couple other senior people decided to rename it computer vision, and they changed the name of the conference to CVPR.”

In his four decades attending CVPR, Davis has witnessed several waves of change sweep through the computer vision field.

Larry Davis, senior principal scientist for Amazon Fashion.

“The 1980s was the decade of what we would call segmentation,” Davis says. “How do you take a picture and break it into parts? One of the motivations was computational — trying to reduce the combinatorics of finding objects in pictures. What’s an object? You might say it’s some connected set of pixels. There are a lot of connected subsets of pixels in images. You can’t begin to look for objects by enumerating the connected subsets. But suppose you could break the image up into 100 pieces via color and texture segmentation. Now you can begin to think of a more-or-less brute-force algorithm that would look at combinations of pieces and ask, ‘Is this the right shape for a dog?’

“Alternatively, you could try to outline objects. Perceptual psychologists will tell you that most of the information that people perceive from images is from the boundaries of objects or the boundaries between the objects and background. So people tried for decades to build edge detectors that would enclose objects. That kind of died out. People weren’t making much progress.

“In the ’90s there was an interest in geometry — multiview geometry, how do you build 3-D models from multiple images from a camera moving through the environment. There was a ton of great work on that.

“In the 2000s, computers became fast enough, mass storage became cheap enough, that people were doing video, so there was a lot of interest in video surveillance. Once the social-media companies took hold, then the emphasis in video was more on consumer videos, being able to caption them, summarize them visually, index into them, all kinds of things.

“And then, the last seven or eight years it’s been all deep learning for all these problems. And that’s when the field took off in terms of size and sophistication.”

Paradigm shift

As it has in other areas of AI, deep learning has revolutionized computer vision.

“Instead of sitting around trying to design new feature detectors, you design new architectures that learn the features,” Davis says.

For researchers of Davis’s generation, that sometimes meant the abandonment of long-standing research programs.

“I saw in a number of my colleagues a resistance to the deep-learning wave,” Davis says. “I met [deep-learning pioneer] Geoff Hinton and a researcher then at Hopkins named Terry Sejnowski — he was more a biomedical type who was interested in neural computing — when they were young. I always thought they were brilliant. It wasn’t what I did, but I was willing to believe them. So it never really bothered me to vacate classical computer vision and just parachute into deep-learning territory. And in any event, any faculty members who didn’t do it were dragged in by their students, because they certainly knew what kind of skills they needed to develop in order to be successful.”

For all of deep learning’s successes, however, some researchers have begun to question whether we can expect to get much more mileage out of it.

“I think we might have hit a wall on some problems — for example, there have been no dramatic improvements recently in the ability to detect objects in pictures,” Davis says. “If you want to build the world’s ultimate flower detector, that never makes a mistake, that sees every flower from any angle, even when only a very small part of the flower is visible, or its image is very small, you realize that might never happen. Even if the problem is just the unavailability of training data to cover all edge cases — if they could ever be known — the cost of doing this for all object types, maybe even just for flowers, would be prohibitive today.”

It never really bothered me to vacate classical computer vision and just parachute into deep-learning territory. … I think the field attracts people who are not challenged by change.

Davis, however, remains optimistic about the continued applicability of deep-learning techniques. While object recognition may be running up against its limits, he says, “the interesting thing is that there are still so many more problems that you can successfully approach. They’re just different problems, and even the current object detectors are good enough to solve lots of important problems.”

Within Amazon Fashion, for instance, “there are several different things going on that involve different types of deep-learning architecture, and they all have some scientific creativity, something new that somebody’s developed,” Davis says.

Indeed, three of the ten Amazon papers at CVPR this year concern ways in which computer vision can help improve customers’ experience while shopping for clothing online.

“The number of deep-learning problems whose solution would improve our customers’ shopping experience is enormous, and we’ve expanded our science team to tackle more and more of them,” Davis says. “We’re always looking for more ML scientists excited about fashion to join us.”

Still, if some new, more powerful artificial-intelligence paradigm arises, Davis will remain no more attached to deep learning than he was to pixel clustering and boundary detection.

“In computer science generally, things change very rapidly,” he says. “You can’t hold on to old stuff. I think the field attracts people who are not challenged by change.”

Source link

Related Topics:Amazon Fashion CVPR Deep Learning

Up Next

Michael J. Black awarded CVPR “test of time” honor

Don't Miss

Computer vision: a look at the past, present, and future

Larry Hardesty

Click to comment

Events & Conferences

A New Ranking Framework for Better Notification Quality on Instagram

Published

4 days ago

September 2, 2025

Xian Sun

We’re sharing how Meta is applying machine learning (ML) and diversity algorithms to improve notification quality and user experience.
We’ve introduced a diversity-aware notification ranking framework to reduce uniformity and deliver a more varied and engaging mix of notifications.
This new framework reduces the volume of notifications and drives higher engagement rates through more diverse outreach.

Notifications are one of the most powerful tools for bringing people back to Instagram and enhancing engagement. Whether it’s a friend liking your photo, another close friend posting a story, or a suggestion for a reel you might enjoy, notifications help surface moments that matter in real time.

Instagram leverages machine learning (ML) models to decide who should get a notification, when to send it, and what content to include. These models are trained to optimize for user positive engagement such as click-through-rate (CTR) – the probability of a user clicking a notification – as well as other metrics like time spent.

However, while engagement-optimized models are effective at driving interactions, there’s a risk that they might overprioritize the product types and authors someone has previously engaged with. This can lead to overexposure to the same creators or the same product types while overlooking other valuable and diverse experiences.

This means people could miss out on content that would give them a more balanced, satisfying, and enriched experience. Over time, this can make notifications feel spammy and increase the likelihood that people will disable them altogether.

The real challenge lies in finding the right balance: How can we introduce meaningful diversity into the notification experience without sacrificing the personalization and relevance people on Instagram have come to expect?

To tackle this, we’ve introduced a diversity-aware notification ranking framework that helps deliver more diverse, better curated, and less repetitive notifications. This framework has significantly reduced daily notification volume while improving CTR. It also introduces several benefits:

The extensibility of incorporating customized soft penalty (demotion) logic for each dimension, enabling more adaptive and sophisticated diversity strategies.
The flexibility of tuning demotion strength across dimensions like content, author, and product type via adjustable weights.
The integration of balancing personalization and diversity, ensuring notifications remain both relevant and varied.

The Risks of Notifications without Diversity

The issue of overexposure in notifications often shows up in two major ways:

Overexposure to the same author: People might receive notifications that are mostly about the same friend. For example, if someone often interacts with content from a particular friend, the system may continue surfacing notifications from that person alone – ignoring other friends they also engage with. This can feel repetitive and one-dimensional, reducing the overall value of notifications.

Overexposure to the same product surface: People might mostly receive notifications from the same product surface such as Stories, even when Feed or Reels could provide value. For example, someone may be interested in both reel and story notifications but has recently interacted more often with stories. Because the system heavily prioritizes past engagement, it sends only story notifications, overlooking the person’s broader interests.

Introducing Instagram’s Diversity-Aware Notification Ranking Framework

Instagram’s diversity-aware notification ranking framework is designed to enhance the notification experience by balancing the predicted potential for user engagement with the need for content diversity. This framework introduces a diversity layer on top of the existing engagement ML models, applying multiplicative penalties to the candidate scores generated by these models, as figure1, below, shows.

The diversity layer evaluates each notification candidate’s similarity to recently sent notifications across multiple dimensions such as content, author, notification type, and product surface. It then applies carefully calibrated penalties—expressed as multiplicative demotion factors—to downrank candidates that are too similar or repetitive. The adjusted scores are used to re-rank the candidates, enabling the system to select notifications that maintain high engagement potential while introducing meaningful diversity. In the end, the quality bar selects the top-ranked candidate that passes both the ranking and diversity criteria.

Figure.1: Instagram’s diversity-aware ranking framework where the diversity layer sits on top of the existing modeling layer and penalizes notifications that are too similar to recently sent ones.

Mathematical Formulation

Within the diversity layer, we apply a multiplicative demotion factor to the base relevance score of each candidate. Given a notification candidate 𝑐, we compute its final score as the product of its base ranking score and a diversity demotion multiplier:

$\text{Score}(c) = R(c) \times D(c)$

where R(c) represents the candidate’s base relevance score, and D(c) ∈ [0,1] is a penalty factor that reduces the score based on similarity to recently sent notifications. We define a set of semantic dimensions (e.g., author, product type) along which we want to promote diversity. For each dimension i, we compute a similarity signal p_i(c) between candidate c and the set of historical notifications H, using a maximal marginal relevance (MMR) approach:

$p_i(c) = \mathrm{max}_{h \in H}\mathrm{sim}_i(c, h)$

where sim_i(·,·) is a predefined similarity function for dimension i. In our baseline implementation, p_i(c) is binary: it equals 1 if the similarity exceeds a threshold 𝜏_i and 0 otherwise.

The final demotion multiplier is defined as:

$D(c) = \prod_{i=1}^{m} \left( 1 - w_i \cdot p_i(c) \right)$

where each w_i∈ [0,1] controls the strength of demotion for its respective dimension. This formulation ensures that candidates similar to previously delivered notifications along one or more dimensions are proportionally down-weighted, reducing redundancy and promoting content variation. The use of a multiplicative penalty allows for flexible control across multiple dimensions, while still preserving high-relevance candidates.

The Future of Diversity-Aware Ranking

As we continue evolving our notification diversity-aware ranking system, a next step is to introduce more adaptive, dynamic demotion strategies. Instead of relying on static rules, we plan to make demotion strength responsive to notification volume and delivery timing. For example, as a user receives more notifications—especially of similar type or in rapid succession—the system progressively applies stronger penalties to new notification candidates, effectively mitigating overwhelming experiences caused by high notification volume or tightly spaced deliveries.

Longer term, we see an opportunity to bring large language models (LLMs) into the diversity pipeline. LLMs can help us go beyond surface-level rules by understanding semantic similarity between messages and rephrasing content in more varied, user-friendly ways. This would allow us to personalize notification experiences with richer language and improved relevance while maintaining diversity across topics, tone, and timing.

Source link

Events & Conferences

Simplifying book discovery with ML-powered visual autocomplete suggestions

Published

4 days ago

September 2, 2025

Mao Sheng Liu

Every day, millions of customers search for books in various formats (audiobooks, e-books, and physical books) across Amazon and Audible. Traditional keyword autocomplete suggestions, while helpful, usually require several steps before customers find their desired content. Audible took on the challenge of making book discovery more intuitive and personalized while reducing the number of steps to purchase.

We developed an instant visual autocomplete system that enhances the search experience across Amazon and Audible. As the user begins typing a query, our solution provides visual previews with book covers, enabling direct navigation to relevant landing pages instead of the search result page. It also delivers real-time personalized format recommendations and incorporates multiple searchable entities, such as book pages, author pages, and series pages.

1 of 2

Audible’s visual-autocomplete experience.

2 of 2

Amazon’s visual-autocomplete experience.

Our system needed to understand user intent from just a few keystrokes and determine the most relevant books to display, all while maintaining low latency for millions of queries. Using historical search data, we match keystrokes to products, transforming partial inputs into meaningful search suggestions. To ensure quality, we implemented confidence-based filtering mechanisms, which are particularly important for distinguishing between general queries like “mystery” and specific title searches. To reflect customers’ most recent interests, the system applies time-decay functions to long historical user interaction data.

Events & Conferences

Revolutionizing warehouse automation with scientific simulation

Published

1 week ago

August 26, 2025

Deniz Akyildiz

Modern warehouses rely on complex networks of sensors to enable safe and efficient operations. These sensors must detect everything from packages and containers to robots and vehicles, often in changing environments with varying lighting conditions. More important for Amazon, we need to be able to detect barcodes in an efficient way.