2026-09-08

Comparing AI customer service bots: how enterprises should choose

A comparison of AI customer service bot solutions across real business metrics, human handoff, knowledge base quality, private deployment capability and POC validation, to help enterprises make selection and procurement decisions on solid ground.

1. Change how you compare: don't start with feature counts

When procuring AI customer service, a common approach is to tally features side by side: how many channels are supported, how many document types can be imported, whether there is voice capability, how many parameters the model has. This order of comparison easily leads to misjudgment. A feature existing doesn't mean it covers the company's main inquiries, and fluent answers in a demo don't mean the system can actually handle business transactions after launch.

A more reliable method is to map scenarios first, then decide on the product form. Sample recent customer service records and organize scenarios by inquiry volume and peak distribution, the channels users come in through, how stable the questions are, and whether system operations need to be executed. Don't just list departmental categories like "pre-sales" and "after-sales"; break them down into testable tasks such as "check shipping status," "determine whether a return is allowed," "change the delivery address" and "explain membership benefits."

Different automation capabilities are not simply ranked higher or lower. When standard questions make up a large share and business rules are stable, rule-based solutions are usually easier to control, and their costs are more predictable. Generative Q&A suits questions with complex phrasing and scattered source material, but its answer boundaries must be constrained. Only when customer service tasks require reading real-time status and taking action is it worth introducing an agent; otherwise, the extra system integration, auditing and failure recovery mechanisms may not deliver matching returns.

In comparing solutions, it is especially important to distinguish "can answer" from "can resolve." For example, a bot that can explain the return policy has only proven that it understands the knowledge content. If the goal is to complete a return, it also needs to verify the order status, check the product's condition, create an after-sales ticket, and write the result back to the order or CRM system. The same goes for tasks such as shipping inquiries, stock confirmation and order cancellation: the natural language reply is only the interaction layer, and closing the loop depends on whether systems such as orders, products and customers can be read and written, and whether operations can be rolled back or handed off to a human agent when they fail.

So the requirements list shouldn't just say "supports order lookup"; it should spell out the execution boundaries:

• Which systems the bot reads, and how much data latency is acceptable;
• Which operations can be executed automatically and which require the user to confirm again;
• How API timeouts, data conflicts or insufficient permissions are handled;
• Whether results are written back to the ticket, with call records and a chain of responsibility retained;
• Whether high-risk actions can be restricted by conditions such as amount, user type or business status.

Industry figures such as market growth, faster response times or lower customer service costs can only help you judge the direction of the technology; they cannot go straight into your own ROI expectations. Companies differ greatly in inquiry mix, cost per human interaction, system maturity and self-service foundations. If many requests inherently require human judgment, or backend APIs are not yet open, even the most powerful language model will struggle to replicate the cost savings other companies achieved.

Before procurement, establish your own baseline: monthly valid inquiry volume, human handling time per interaction, first-contact resolution rate, human handoff rate, repeat contact rate, and the time spent on system operations for each type of task. Then estimate, for each solution, the share of inquiries it can cover and the share of tasks it can complete automatically. The former measures how much the bot "says"; the latter measures how much it "gets done." Only by separating these two metrics does a feature list turn from marketing material into verifiable engineering requirements.

2. Dimension 1: measure with real business metrics, not demo performance

Demo environments usually contain only curated questions, complete knowledge materials and preset dialogue paths, and they rarely reflect the colloquial phrasing, missing context, system errors and emotional complaints of production. Procurement evaluation should shift from "what can the bot answer" to "how many problems can the bot resolve on its own, and what does each resolution cost."

We recommend establishing at least the following metrics, with the calculation method spelled out in the tender documents, the POC and the acceptance clauses:

These metrics must share the same measurement period, and the deduplication rules for conversations, the definition of invalid inquiries, how identities are merged across channels and the criteria for "resolved" must all be explicit. In particular, don't mix results calculated by number of users, conversations, questions and messages. With different denominators, solutions can't be compared even when the metrics share a name.

Before launch, also keep a representative set of human service data as a baseline, stratified by business scenario. Pre-sales inquiries tend to be repetitive and low risk; order lookups depend on real-time APIs; after-sales service involves rule-based judgments and process execution; complaints rely more on emotion recognition, permission boundaries and human collaboration. Looking only at an overall average lets the large volume of simple, high-frequency questions inflate the automated resolution rate and mask failures in complex after-sales and complaint scenarios. A more reliable approach is to look at inquiry volume, standalone resolution rate, handoff rate, handling time and satisfaction for each scenario, then compute a combined result weighted by the company's actual business mix.

Vendor case studies can help you decide what to observe, but they cannot serve as a promise of returns. A public BetterYeah AI case study described improvements in efficiency, problem resolution and satisfaction; a case study of Guanjiapo software's integration with Alibaba Cloud's Tongyi bot also reported gains in service efficiency. Results like these show that AI customer service can produce business value, but they cannot prove that improvements of the same magnitude will carry over to another company. Final performance is usually shaped jointly by inquiry mix, the quality of existing knowledge, how deeply order and ticketing systems are integrated, the human takeover process and ongoing operational investment. If the improvement percentages in a case study lack a verifiable report year, sample scope and metric definitions, they should not go into the company's own budget model.

Cost comparison also can't stop at monthly fees or per-seat prices. Buyers should account over the full lifecycle for subscriptions or licenses, model inference calls, API and workflow development, knowledge curation and ongoing updates, human fallback, monitoring and evaluation, system maintenance, and upgrade adaptation. We recommend using "cost per valid resolution" as a unified financial metric, while separately calculating three cost types: resolved by the bot alone, resolved after handoff to a human, and resolved purely by humans. Only when resolution quality does not decline and total cost keeps improving relative to the pre-launch baseline has the solution produced verifiable business value.

3. Dimension 2: human handoff determines whether the bot actually reduces workload

When evaluating customer service bots, "fewer handoffs to humans" can't be equated with good performance. The bot should keep answering only if it correctly understands the question and can give a reliable conclusion within its current permissions and knowledge scope. When confidence is low, the user explicitly asks for a human, negative feedback keeps coming, or the conversation involves complaints, money, compliance or personal safety, the system should promptly exit automated handling. Forcibly blocking handoffs may reduce human workload on the surface, but in practice it can increase repeat inquiries, customer churn and complaint escalation.

Procurement testing should check both "does it hand off when it should" and "does it hand off when it shouldn't." The former reflects risk recognition; the latter determines whether the bot really shares the workload. Sample ordinary inquiries, complex business cases, emotionally charged conversations and high-risk cases from historical conversations, pre-label the expected handling, and run candidate solutions on the same samples, rather than relying only on demo scripts prepared by vendors.

Of these, the repeat explanation rate usually does the most to expose integration depth. A handoff shouldn't just send a chat transcript; it should produce a context package the agent can use directly, containing at least a conversation summary, the identified business intent, the user's identity and authentication status, related orders or tickets, the lookups the bot has already performed, and the reason that triggered escalation. Sensitive fields should be masked according to agent permissions rather than displayed in full on every workstation.

Routing capability can't be validated just by checking whether the bot can connect to the human agent system. Companies should check whether queues can be assigned by issue category, customer service tier, business hours, language and agent skills. For example, refund disputes should go to a team with the corresponding authority, high-value customers can be given a different priority, and outside service hours the system should offer scheduled callbacks or ticket intake. If every request lands in the same shared queue, the bot has merely added an entry point without optimizing human resources.

Conversation continuity after human takeover also needs hands-on testing. Once an agent replies, the bot should stop jumping in; when the agent steps away, the conversation is transferred to another group or the channel switches, the conversation state must not be lost. Testing should cover human takeover, return to the bot, second escalation and transfers across skill groups, checking that message order, ticket status and customer identity stay consistent throughout.

Finally, the two kinds of failure should be costed separately. Unnecessary handoffs consume agent hours and lengthen queues; handoffs that come too late lengthen the customer's path to resolution and turn what was an ordinary problem into a complaint. For accounting, you can use "unnecessary handoff volume × average human handling cost," and separately track the repeat contacts, complaint escalations and compensation payouts caused by late handoffs. The two cannot offset each other, and neither should be masked by the overall handoff rate.

POC acceptance criteria should therefore cover escalation recognition, context sync, skill-based routing, queue performance and resolution after human takeover all at once. A truly effective solution is not one that keeps users on the bot side as long as possible, but one that completes the job when automation is confident and, when continued automation isn't appropriate, hands the problem, together with full context, to the most suitable agent.

4. Dimension 3: judge the knowledge base by answer quality, not import formats

Support for uploading documents, crawling web pages, maintaining Q&A pairs, reading spreadsheets or connecting to business databases only shows that the system can ingest knowledge; it does not prove that it can answer customer questions reliably. If procurement treats "how many formats are supported" as a main scoring item, it is easy to pick a solution that imports material smoothly but gives wrong answers frequently after launch.

Knowledge base evaluation should work backward from the output: does the answer comply with current business rules, does it cover the key conditions in the question, can it point to the policy, product documentation or data record it relies on? Especially for high-risk content such as pricing, benefits, after-sales policies and compliance requirements, a smoothly worded conclusion alone is not enough. In the POC, require the system to show source documents, specific paragraphs or data fields, and verify that the cited content actually matches the answer's conclusion. Showing only a file name or a web link is not enough to form an auditable evidence chain.

The test set should not be written on the spot by the vendor, nor should it consist only of standard questions that knowledge base titles can match directly. A more effective approach is to sample from the company's historical tickets, online conversations and call summaries, mask the data, and compile a fixed question bank. Questions should cover at least standard phrasing, colloquial abbreviations, typos, follow-up questions, products with similar names, and cases where different documents contradict each other. For questions that require looking up order, membership or account status, also distinguish "knowledge retrieval failure" from "business API failure"; otherwise you can't tell whether the problem lies in the knowledge base or in system integration.

For each question, predefine the expected answer, required information points, acceptable wording range, materials that should be cited, and whether the system should refuse. Record evaluation results separately as correct answer, reasonable refusal, wrong answer and missing key information, rather than merging them into a vague "hit rate." Companies should also weight by business risk: omitting one condition of a return deadline clearly has different consequences from leaving out a sentence in a brand introduction. For conflicting materials, focus on whether the system follows the configured authority levels and effective dates, rather than arbitrarily choosing the chunk with the higher retrieval score.

Knowledge quality also depends on the update pipeline. During the POC, deliberately change a policy, record the time from content submission to actual effect in each service channel, and check whether the old answer is still retrieved. Buyers need to confirm, item by item, version retention, role permissions, publishing approval, takedown at expiry and rollback capability, and channels such as the website, app, WeCom and call center should all use the same published version. Otherwise, even if the knowledge base answers accurately, customers may receive conflicting answers because channels publish at different times.

"Continuous learning" likewise needs clear boundaries. Real conversations can be used to discover unknown questions, add synonymous phrasings and identify low-quality answers, but they should not become official knowledge without review. Customer conversations may contain agents' ad hoc promises, incorrect explanations, sensitive personal information or expired policies. An acceptable process looks like this: the system proposes candidate knowledge or optimization suggestions, the business owner reviews them, and after masking, conflict checks and approval they are published, with a record of who made the change, when it was published and how to roll it back.

• Contract acceptance should be tied to the company's own test set, not the vendor's demo question bank.
• Acceptance metrics should separate wrong answers, omissions, refusals and citation validity, and specify handling rules for high-risk questions.
• Knowledge update latency, retention of historical versions, approval records and multi-channel sync should be written into the acceptance clauses.
• Any new knowledge generated from customer conversations should go through masking and human review before it goes into production.

Ultimately, what should be compared is not how much material a system can "hold," but whether it can find the right grounding under complex phrasing, give answers with clear boundaries, and keep knowledge changes in a process the company can control, inspect and roll back.

5. Dimension 4: break private deployment down into data, models, integration and operations

Buyers first need to rewrite "private deployment" as verifiable technical boundaries. A customer service application running on company-dedicated servers or in a dedicated cloud account does not mean the entire processing chain stays on the internal network. Conversations may still call external models, document chunks may go into a vendor-hosted vector database, and runtime logs, quality analytics data and alerts may be sent to external platforms. So confirming the deployment location alone means little; you must confirm, item by item, which components the data passes through, which network boundaries it crosses, and who holds administrative privileges.

Data clauses can't stop at generalities like "meets security requirements." The contract and technical annexes should at least specify storage region, transmission path, access authorization, audit log retention rules, handling of sensitive fields, data destruction deadlines, and whether the vendor may use company data to improve models. They should also set notification deadlines, forensic cooperation, remediation responsibilities and liability for losses in the event of a leak, mistaken authorization or a breach of the service. If the vendor uses external models or cloud services, it should also disclose its subcontractors and the scope of their data processing.

Private model deployment is also not simply a matter of putting model files in the company's data center. Buyers need to determine whether model inference, vector retrieval, reranking, content moderation and quality analytics can run independently, and confirm whether business data must be re-uploaded after upgrades. If the solution is locked to a specific model service, assess the risks of price changes, API deprecation and regional unavailability. A safer acceptance standard: key models are replaceable, endpoints are configurable, the scope of data use is auditable, and there is a clear degradation path when external services go down.

Integration capability directly determines whether a private deployment project can go into production. A large number of APIs doesn't mean they are usable; focus on verifying customer identification, order lookup, ticket creation, status write-back and inheritance of agent permissions. Testing can't follow only the happy path; it must also cover API timeouts, expired credentials, field changes, duplicate requests and unavailable downstream systems. Write operations must have idempotency controls, an audit trail and human confirmation, to prevent the bot from issuing duplicate refunds, creating duplicate orders or wrongly modifying customer data.

Operational responsibilities must be pinned to specific boundaries. The company needs to be clear about who scales up when compute runs short, who restores the database after corruption, who installs security patches, and who rolls back after a failed upgrade. Monitoring should cover at least request success rate, response latency, model and retrieval service status, API errors, resource usage and message backlog. If the vendor is responsible only for the application while the operating system, database, middleware and model services all fall to the company, actual operations effort is usually significantly higher than the quoted price suggests.

Cost comparison should use total cost of ownership across the build, use and maintenance phases, not just the first-year contract value. Market quotes usually split private deployment fees into software licensing and ongoing maintenance, but these are only a preliminary reference. A complete budget should also include inference compute, storage and backup, implementation and delivery, business API modifications, security assessments, environment expansion, version upgrades, and the time of internal development and operations staff. We recommend booking one-time costs, annual fixed fees, usage-based fees and internal labor separately, and modeling them against business growth scenarios.

• If core data still flows to external services, the solution should be treated as a hybrid deployment, not a fully closed internal-network loop.
• If offline operation, permission isolation and backup recovery can't be demonstrated in the POC, the solution should not pass acceptance on the strength of an architecture description alone.
• If fault responsibility, upgrade windows and data exit mechanisms are not written into the contract, subsequent operational risk usually falls on the buyer.
• If the long-term total cost exceeds the labor savings and risk benefits it can deliver, private deployment is not inherently better than other deployment models.

The final criterion is not whether the "supports private deployment" box is checked, but whether the company can control its data, replace models independently and connect business systems reliably, and whether someone will restore service as agreed when something fails. Only when all four capabilities are testable, auditable and accountable does private deployment have procurement value.

6. How to compare mainstream solutions

Procurement teams shouldn't put every AI customer service product into the same feature scoring sheet. Different solutions solve different problems: some focus on carrying the full customer service workflow, some aim for fast, low-cost activation, and others provide a foundation for knowledge engineering and system integration. A more effective approach is to group products by form first, then validate business results within each group.

The value of a mature suite lies in workflow completeness, not just the bot's ability to answer. Intercom and Zendesk AI are generally worth evaluating first for companies with an established customer service operation, many channels or a large agent team. The comparison should focus on how conversations enter queues, how tickets flow, how the bot and humans hand off, and whether managers can track service quality. Answer quality in a demo environment covers only part of the actual procurement value.

The hidden costs of these products often come from migration and billing. Companies need to take stock of whether historical tickets, user fields, knowledge content, channel configurations and automation rules can be migrated smoothly, and confirm whether AI assistants, automated resolutions, message volume, agent seats and advanced analytics are billed separately. The monthly fees on public pricing pages usually correspond to different billing bases, usage caps and contract terms, and can't be converted directly into a company's annual total cost.

Lightweight SaaS has the advantage of a fast start, but its capability limits have to be exposed through stress scenarios. Freshchat and Tidio can make the shortlist for companies that value ease of use, implementation speed and budget control, and they are especially suited to starting with website inquiries or small service teams. However, "can go live in a few days" is not the same as "can reliably handle complex business." The POC shouldn't test only high-frequency Q&A; it should include inquiries with many conditions, questions with insufficient information, follow-up questions, newly updated knowledge, and conversations that need a human to take over.

If the bot frequently gives generic replies in complex scenarios, or agents can't see the full context after a handoff, the deployment cost saved up front turns into a human workload later. Companies should also simulate growth in inquiry volume and check the cost curve created by plan upgrades, additional seats, AI usage packs and automation modules.

Platform products are better evaluated by "connectability." According to public product materials for Alibaba Cloud's intelligent dialogue bot, its knowledge sources can include files, site content, structured tables and databases; it can connect to websites, mobile apps and instant messaging entry points, and it provides APIs for companies to extend. For projects with many Chinese business terms, answers that depend on internal data, or a need to embed into order, membership, after-sales and other systems, this type of product can be included in the POC.

But the number of APIs isn't the conclusion. During testing, actually connect one business data source and verify field permissions, query correctness, response latency, graceful degradation and call auditing; then update a batch of knowledge and observe the full pipeline of parsing, indexing and answers taking effect. Only then can you judge whether the platform's capabilities truly translate into delivery capability.

The final comparison should follow one principle: judge suites by workflow closure, lightweight SaaS by how far its capabilities reach at a low entry barrier, and platform products by the depth of their knowledge and system connectivity. Media ratings, the monthly fee of a single tier and the number of checked features are useful only for initial screening; they can't replace POC results based on the same test set, the same inquiry volume and the same service goals.

7. Use the POC and contract terms to make the final procurement decision

A product demo only proves that the system can run along a preset path; it doesn't prove that it can handle the company's own customer questions. Once procurement reaches the final shortlist, stop comparing feature lists, switch to POC validation under uniform conditions, and turn the results into contract requirements that acceptance can be measured against.

Build a unified test set from historical conversations

Test questions should be drawn from real historical conversations rather than taken from sample questions supplied by candidate vendors. We recommend stratified sampling by business type, inquiry frequency and handling difficulty, covering high-frequency standard questions, contextual follow-ups, vaguely phrased questions, tasks that require querying business systems, and high-risk scenarios that should be handed off to a human. Test data should be masked first, with the correct answers, handling actions and handoff conditions retained.

All candidate solutions must use the same version of knowledge materials, business APIs, access channels and human agent rules. Which fields the model can access, when the knowledge base is updated and how the system degrades after failures must also be kept consistent. Otherwise, the test results may reflect differences in implementation effort rather than differences in the capabilities of the solutions themselves.

The test period must cover real traffic variation

A single burst of Q&A is not enough to support a procurement decision. The POC should span at least normal working hours, unattended hours and business peaks, and preserve real multi-turn context and concurrency pressure as far as possible. During testing, don't just count how many questions the bot answered; also record the downstream handling costs caused by errors.

Use pass/fail thresholds; don't let a total feature score hide weaknesses

The scoring sheet can keep its weights, but the final decision should set entry thresholds around actual business results, human-bot collaboration quality, knowledge reliability, and deployment and operations constraints. A failure involving security, critical business errors or a breakdown of human fallback should never be offset by extra points for other features.

Also distinguish between "achieved now" and "promised to be achievable." The former can be counted directly in POC results; the latter must list the scope of changes, owner, delivery date and retest method. If a candidate solution depends on extensive manual tuning or on-site maintenance to meet the bar, that ongoing investment should be included in total cost of ownership.

Write the POC conclusions into acceptance clauses

• Unify metric definitions: specify conversation boundaries, success criteria, anomalous samples and formulas, to avoid reinterpretation at acceptance.
• Lock down data scope: agree on where data is stored, what it is used for, the retention period, the deletion method, and whether it may be used for model training.
• Agree on service levels: spell out how availability is calculated, fault severity levels, response and recovery deadlines, and remedies for missed targets.
• Define security responsibilities: cover access control, log retention, vulnerability handling, data breach notification and responsibility for third-party components.
• Clarify delivery boundaries: list who is responsible for knowledge curation, API development, channel integration, monitoring and alerting, and staff training.
• Preserve exit capability: require that knowledge, configurations, logs and necessary conversation data can be exported, and specify migration formats, fees and the assistance period.

Procurement scheduling should also be broken down by delivery depth. Basic integration can usually be completed fairly quickly, but when proprietary processes, multiple business systems, permission frameworks or on-premises deployment are involved, the timeline stretches considerably. The contract shouldn't specify just one "go-live date"; it should set milestones such as environment readiness, API integration testing, trial operation, metric retesting and formal acceptance.

In the end, choose the solution that passes the key thresholds under the same test conditions, has explainable implementation costs and offers a clear exit path, not the one with the smoothest demo or the most features. The POC reduces uncertainty in judging capability; the contract controls uncertainty about responsibility after delivery. Without either one, the procurement conclusion is on shaky ground.

8. FAQ: common questions about buying an AI customer service bot

What automated resolution rate counts as acceptable for AI customer service?

There is no single passing line that applies to every company. Inquiry-type business, after-sales troubleshooting, complaint handling and transaction changes differ in difficulty, and comparing automated resolution rates directly easily leads to the wrong conclusions. Companies should first define what "resolved" means: the bot giving a reply is not resolution, and the user not following up doesn't mean the issue is closed either.

A more usable definition: within an agreed observation period, the user's goal has been achieved, with no handoff to a human, no repeat contact and no correction ticket created. The evaluation should also look at the handoff rate, repeat inquiries, answer acceptance, complaint trends and human handling time. If the automated resolution rate rises but repeat contacts or complaints rise along with it, the bot is usually just delaying human involvement.

Acceptance criteria should be set separately by question type, using the current human service baseline as the reference. High-frequency questions with stable rules should deliver most of the automation gains; for questions involving authorization, complex judgment or high-risk operations, accurate recognition and smooth handoff should come first, rather than a higher automation rate.

How should we choose between SaaS and private deployment?

The choice shouldn't rest only on budget or company size; break it down into four areas: data boundaries, model control, system connectivity and ongoing operations. If inquiry content is of low sensitivity, you want to validate business results quickly, and you lack in-house algorithm and platform operations teams, SaaS is usually the better starting point. Before buying, still confirm where data is stored, the log retention policy, whether data is used for model training, export and deletion mechanisms, and how data is disposed of when the service ends.

When conversations involve regulated data, core business information or strict internal network isolation requirements, or the company needs to control model versions, inference resources and release cadence, private deployment is worth considering. But private deployment doesn't end with installing software on the internal network; you also take on capacity planning, model upgrades, security fixes, monitoring and alerting, failure recovery and knowledge base maintenance. The decision should compare full lifecycle cost, not just first-year quotes.

A hybrid approach is also possible: sensitive data and critical processes stay in a controlled environment, while general capabilities use external services. The prerequisites are clear data classification, auditable call chains and a defined degradation plan for failures.

How many real questions should we prepare for a POC?

A POC shouldn't start by chasing a fixed number of questions; what matters is whether the sample covers real traffic and the main failure modes. The question bank should include at least high-frequency standard phrasings, colloquial expressions, typos, contextual follow-ups, missing information, similar intents, conflicting knowledge, out-of-scope questions, and high-risk scenarios that must be handed off to a human. Copying standard questions from FAQ documents alone will significantly overestimate post-launch performance.

Samples should be drawn from historical conversations, tickets and search logs, stratified by business topic, handling difficulty and risk level. Keep adding new error types as they appear during testing; only when new samples no longer noticeably change the distribution of results across question types has coverage stabilized. Review results also shouldn't record only "right or wrong"; distinguish wrong citations, incomplete answers, wrongful refusals, answers beyond authorized scope, handoff failures and slow responses, so you can pinpoint whether the problem lies in knowledge, retrieval, prompting strategy or workflow integration.

We already have a human customer service system. Do we need to replace it entirely?

Usually not. A safer approach is to keep the existing agent system and plug the bot into the entry layer, the routing layer or agent assistance. The existing system continues to handle queuing, conversations, tickets, quality inspection and permission management, while the bot takes on intent recognition, knowledge retrieval, handling of common questions and information gathering before handoff.

Whether replacement is needed depends on whether the existing system can provide stable APIs, pass complete context, support two-way handoff between the bot and humans, and allow records to flow into a unified audit and reporting system. If it can only redirect to a human entry point without carrying the user's identity, what has already been asked, the bot's answers and the reason for failure, agents still have to ask again, and the reduction in workload will be limited.

We recommend first choosing a channel or business queue with clear boundaries for a parallel integration, validating conversation continuity, failback and metric definitions before deciding whether to expand. Only when the existing system has long resisted integration, key data can't be exported, or maintenance costs have become unacceptable might full replacement make more sense than gradual modernization.