Enhancing eCommerce Search With AI-Powered Product Discovery Engine

Project Requirements

A Better Site Search Experience

Every year, retailers miss out on over $2 trillion in potential sales in the U.S. alone due to poor online search experiences. Our client faced a similar challenge with their site search experience. Only 10-15% of users searching for products actually made a purchase.

They also lost potential sales due to irrelevant search results. Precise queries like "hand faucets for kitchen sink, 1 meter long, with long flexible tube & wall hook" often displayed unrelated items, and broad queries such as "sink water outlet" or "kitchen faucet" overwhelmed users with too many options.

To address these challenges, the client needed a product discovery solution to:

Auto-suggest relevant products in real time when the user starts typing a search term
Predict and display the most likely products, improving search relevance and sales likelihood
Analyze user inputs to detect potential synonyms and match search queries accurately
Personalize search results and product recommendations for each user
Showcase related products through search recommendations

Project Challenges

Conceptualizing the AI-Powered Product Discovery Engine

Our primary challenge was to create a search system that returned comprehensive yet precise results and continued to improve over time. To achieve this, we needed to

determine what content to include and exclude from the search, given the client's extensive product catalog and website content
collate and transform various site content types (product catalogs, articles, manuals, etc.) to make them searchable
establish how the AI model should personalize search results for different users
integrate conversational semantic capabilities into the search user interface
prepare for future increases in data volumes, query loads, and interface complexity.

The Solution

Developing a Custom AI Site Search Feature and Training the Corresponding AI Model

Search AI

We have built a product discovery engine with AI and natural language processing capabilities. This engine can understand user queries and identify the underlying intent. This allows for a more personalized search experience, with more relevant results. This engine also continues to refine its understanding of individual user preferences over time and improves its ability to anticipate their desired results. This has led to more relevant search experiences, increased customer satisfaction, and higher conversion rates.

Recommendation AI

We have also developed a Recommendation Engine that has significantly enhanced the user experience by improving engagement, basket size, and average order value. As users type their search query, the engine provides intelligent auto-fill suggestions, guiding them towards the most relevant results. It also leverages the user's search intent to dynamically recommend related and complementary products, encouraging them to add more items to their cart. Even if the user's initial search query does not yield an exact match, the engine can provide relevant product suggestions, eliminating the frustration of encountering a "no results found" page.

Data Enrichment and Management

Data enrichment was required to equip the product discovery engine with the necessary information to deliver relevant search results. We appended the client's catalog dataset with additional attributes, descriptions, categorizations, and metadata. This has provided the AI engine with the context to match products to user queries. User data like browsing history, purchase patterns, and demographics were integrated to help the engine personalize recommendations. We constructed comprehensive taxonomies mapping relationships between products, categories, and concepts, which has provided the AI model with a better semantic understanding. Incorporating external knowledge bases such as glossaries and manuals has further enhanced the engine's understanding of products and terms used in queries.

Our Workflow

Here's How the AI Search and Recommendation Engine Was Put Together

We assembled a couple of nopCommerce developers (since the client's website was built on nopCommerce, hosted on Oracle cloud) and our AI/ML consultants and developers. This team spent the initial week analyzing the scope and creating blueprints. One we finalized the plan of action, we-

Assembled Necessary Data

Collated website content like product catalogs, articles, manuals, etc. that need to be searchable
Compiled search logs of actual user queries and clicked results from existing site search
Assembled categorization data like product types, content tags, metadata, etc.
To address discrepancies, the collected data was cleaned and standardized

Performed Text Annotation on Processed Data

Labeled words/tokens extracted through NLTK
Tagged words with parts of speech, semantics, roles etc.
This enabled the AI model to understand relationships and meaning between the text

Performed Exploratory Data Analysis (EDA) on Annotated Data

Created word frequency distribution plots to analyze common terms
Visualized query patterns and relationships in search logs
Identified any data quality issues or biases to address
Verified that the extracted keywords and entities make sense

Vectorized Text Data

Converted processed text into numeric vectors using inverse document frequency (Tf-IDF), and Word2Vec, and Bag Of Words (BOW)
Used cosine similarity and cosine distance to measure the distance and angle between the embedded vectors
This enabled retrieving the closest-matching results for a given search query (by comparing vector alignments)
Enriched vectors with extracted entities and keywords as features

Used Python NLTK Toolkit for Text Processing

Tokenized text into sentences and words using NLTK tokenizers
Removed stop words and punctuations
Used the Porter Stemmer algorithm to remove common suffixes and endings from words to get their base form or stem.
This helped determine semantic relationship between different word forms
Applied Part-Of-Speech tagging (POS) to identify verbs, nouns, entities etc.
Extracted keyphrases and entities using NLTK chunking and NER (Named Entity Recognition)

Trained the AI Search Model

Used the processed text, vectors, and user search logs to train a Transformer neural network-based ranking model
Fine-tuned the pre-trained model with domain-specific data to adapt it to the website's content
Evaluated the model's recall, precision, and relevance metrics to ensure high-quality results
Made iterations by modifying model architecture and hyperparameters until optimal performance was achieved

Developed the AI Search Interface

Built an intuitive conversational UI with auto-complete and search suggestions
Ensured UI responsiveness through iterative testing and improvements based on usability studies
Implemented query spelling correction and error handling to make the interface robust
Implemented intent detection to make the search experience more relevant
Provided explanations of the model's ranking logic to the client for transparency
The user feedback is continuously incorporated into the model to retrain and improve its outcome

eCommerce Search and Product Discovery: AI Powered Search Technology Bridges the Gap between Consumer and Merchant

A Bathroom and Lighting Solutions Company

A Better Site Search Experience

Conceptualizing the AI-Powered Product Discovery Engine

Developing a Custom AI Site Search Feature and Training the Corresponding AI Model

Search AI

Recommendation AI

Data Enrichment and Management

Industry-Leading Tools, Technologies, & Frameworks we Work with

Swift

Kotlin

Java

SQLite 3

Xcode

Android Studio

Azure Repos

Azure DevOps

Angular JS

ASP.NET Core (Web APIs)

SQL Server

Here's How the AI Search and Recommendation Engine Was Put Together

Assembled Necessary Data

Performed Text Annotation on Processed Data

Performed Exploratory Data Analysis (EDA) on Annotated Data

Vectorized Text Data

Used Python NLTK Toolkit for Text Processing

Trained the AI Search Model

Developed the AI Search Interface

Business Impact

Achieve Operational Excellence with TechnoScore

Share your Requirements

India

USA

UK