Every chat tool demo looks good. The sales rep picks a friendly question, the assistant gives a perfect answer, and you think "yes, this is what I need." Then you sign up, connect your store, and discover the demo was the high point.
I've been on both sides of this. I've evaluated chat tools for stores, and I've built one. Vendors answer the easy questions during the pitch. These five are the ones they hope you don't ask until after you've committed.
1. How Does It Learn My Products?
This sounds basic, but the answer reveals everything about how well the assistant will work. There are three common approaches:
- Web scraping: The vendor crawls your site and extracts product data from HTML. This breaks whenever you change your template, misses products behind filters or pagination, and has no reliable way to detect when a product goes out of stock. Your assistant might recommend items you stopped selling last month.
- Manual upload: You export a CSV and upload it. Fine for the initial setup, terrible for maintenance. Your catalog changes daily. Nobody re-uploads a spreadsheet every time a price changes.
- Webhook sync: Your store pushes updates in real time. Product created, updated, deleted, price changed, stock changed. The assistant always has current data because your store tells it what changed, when it changed.
Ask the vendor: "If I mark a product out of stock at 2 PM, when does the chat assistant stop recommending it?" If the answer involves "our next crawl" or "after your next upload," that delay is a window where your assistant is actively misleading customers.
Product search quality depends on data freshness. I wrote about how product search actually works in an earlier post, including the difference between keyword matching and understanding what a customer means by "something warm for hiking."
2. What Happens When It Gets the Answer Wrong?
Every chat assistant will get something wrong eventually. Chat assistants sometimes get things wrong. They make up products, invent specifications, or confidently state a return policy that doesn't exist. This will happen. What matters is what the tool does when it happens.
Two things to look for:
- Grounding: Does the assistant only reference products and policies that exist in your data? Or can it fabricate information that sounds plausible but has no source?
- Escalation: When the assistant isn't confident, does it hand the conversation to a human? Or does it keep guessing until the customer gives up?
Confidence-based escalation means the assistant recognizes when it's out of its depth. Instead of saying "I don't know" or making something up, it transfers the conversation to your team with the full context, so the human agent doesn't start from scratch.
More on how human handoff works, including what "confidence-based" means in practice and why a binary "knows / doesn't know" model falls short.
3. What Do I Actually Pay Per Month?
Chat pricing models vary more than you'd expect, and the headline number rarely tells the full story.
- Per-message pricing: You pay for every message sent and received. A single conversation might be 8-12 messages. At $0.01 per message, a 10-message conversation costs $0.10. But some vendors count system messages, tool calls, and retries. Your bill depends on implementation details you can't see.
- Per-conversation pricing: Simpler to predict. But at $0.10 per conversation, a store with 3,500 conversations per month pays $350. That's $4,200 per year before you've added any features.
- Flat rate + included conversations: You pay a fixed subscription that includes a set number of conversations. Overages are billed at a transparent per-conversation rate. For 3,500 conversations on a plan that includes them, you pay the subscription and nothing more. If you go over, the overage rate is clear upfront.
The difference compounds: for the same 3,500 monthly conversations, the spread between pricing models can be 3-5x. Ask for a calculator, not a pricing page. And ask what's included in each "conversation" or "message" count.
4. Does It Work in My Customers' Language?
Most vendors say "yes, we support multiple languages." But the question that matters is more specific: "If my product catalog is in German and a customer writes in French, does the assistant find the right products?"
Translation and cross-language search are different things. Translation means the assistant can respond in French. Cross-language search means the assistant can understand a French query, search German product data, and return relevant results. Translation is table stakes for any modern chat tool, but cross-language search requires the search infrastructure to handle it.
If you sell internationally, test this during evaluation. Write a query in one language about a product described in another. If the assistant returns nothing or returns wrong results, the "multilingual support" is surface-level only.
I wrote about cross-language search and multilingual support in detail, including how the search handles queries across different languages.
5. Can I Prove It's Working?
After you've been running a chat assistant for a month, someone will ask: "Is this making us money?"
If all you have is conversation counts and message volume, you're stuck citing vendor case studies and industry benchmarks. "Studies show chat tools increase conversion by 20%" doesn't answer the question about your store.
What you need is a funnel: chat session started, product added to cart, checkout initiated, purchase completed. Each step linked to the conversation that preceded it. That gives you a number: "Last month, 200 chat sessions led to completed purchases totaling $18,000. The chat cost us $50 (200 conversations × $0.25)."
Ask the vendor: "Can I see which specific conversations in my store led to purchases?" Your conversations, your revenue, your attribution data — not aggregate stats or industry averages.
Without this, you're investing in something you can't measure. More on what conversion tracking looks like in practice, including its limitations (correlation vs. causation, cross-device gaps, attribution windows).
What This Post Doesn't Cover
These five questions help you evaluate before you buy. They don't cover implementation, platform-specific setup, or how to customize the assistant for your store. Those details depend on whether you're running WooCommerce, Magento, Drupal Commerce, Sylius, or something else entirely.
If you're past the evaluation stage and ready for setup specifics, the documentation covers integration for each platform.
Want to test these questions against a real product? Install free, sync your real catalog, and see how the answers hold up. No credit card required. Platform guides for WooCommerce, Drupal Commerce, Sylius, and Magento 2 in the documentation.
