Almost every AI development firm can show a polished demo. A chatbot answers a few questions about a sample PDF, an agent books a test meeting, a dashboard lights up. Demos are cheap to build. What separates partners is what happens when the real data is messy, the model gets something wrong in front of a customer, and you need to change vendors two years later.
The questions below are grouped by what they reveal. For each group, we describe what a good answer sounds like, so you can tell the difference between a team that has shipped production AI systems and one that has mostly built prototypes.
Before the first call
Write down the one workflow you want to improve, a handful of real examples, and the systems it touches. Share it with each candidate in advance. A partner who comes back with specific questions about your examples is already showing you how they work. One who sends a generic capabilities deck is showing you something too.
Technical depth and production experience
Questions on technical depth
- How would you approach this specific workflow, and what would you build first?
- Which models would you consider, and why one over another for this task?
- How do you decide whether this needs retrieval, fine-tuning, an agent, or plain automation with no AI at all?
- How do you test whether the AI’s outputs are correct?
A good answer is specific to your case and includes trade-offs. Expect to hear something like “for extracting fields from your invoices, a mid-sized model with structured output is probably enough; we would test it against 100 of your real invoices before committing.” A strong partner will also tell you when part of the problem does not need AI. Be wary of anyone who proposes the most complex architecture by default, or who cannot explain how they measure accuracy beyond “it looked good in testing.”
Questions on production experience
- What AI systems have you built that real users rely on today?
- What broke after launch, and how did you fix it?
- How do you monitor model behavior and cost once a system is live?
- What happens when a model provider changes or retires a model you depend on?
A good answer includes failure stories. Teams with real production experience talk about outputs that drifted after a provider update, retrieval returning stale documents, costs spiking because context grew, or users finding inputs nobody anticipated. They describe logging, evaluation sets that rerun on every change, and fallbacks. If every story ends in a flawless launch, they have either not shipped much or are not being straight with you.
Data, security, and ownership
Questions on data and security
- Where will our data be stored and processed, and by which providers?
- Will our data be used to train any third-party model?
- How do you handle access control, secrets, and personal data?
- What gets logged, who can see the logs, and how long are they kept?
A good answer names the actual services and accounts involved, explains the provider’s data retention and training terms, and proposes running things in accounts you control where possible. For regulated sectors, they should raise compliance questions themselves rather than waiting for you to ask. Vague reassurance such as “it’s all secure” is not an answer.
Questions on ownership, IP, and handover
- Who owns the code, prompts, evaluation sets, and configuration?
- Whose accounts will the hosting, model API keys, and domains sit in?
- What documentation do we get at handover?
- If we move to another team or bring development in house, what does that process look like?
A good answer is simple: you own the code and the accounts, the repository is in your organization, and handover includes setup instructions, architecture notes, and the evaluation set. Prompts and evaluation data are part of your intellectual property, not the vendor’s secret sauce. If a partner wants to keep the code, host it in their own accounts with no transfer path, or license it back to you, understand that you are renting, not buying.
Process, communication, and pricing
Questions on process and communication
- When will we first see working software?
- How often do we get updates, and in what form?
- Who exactly will work on our project, and who is our day-to-day contact?
- How do you handle a change in scope halfway through?
A good answer puts working software in front of you early, within days or a couple of weeks rather than months, and describes a regular rhythm such as a short weekly demo plus a shared channel. You should be able to talk directly to the people building the system. Scope changes should go through a clear process: written, estimated, and agreed before work starts, not discovered on the invoice.
Questions on the pricing model
- Is this fixed price, time and materials, or a retainer, and why that model for this project?
- What exactly is included in the scope, and what is not?
- What will ongoing costs look like for hosting, model usage, and maintenance?
- Who pays for model API usage during development and after launch?
A good answer fixes the scope before the price and states assumptions in writing. Either fixed price or time and materials can work. The difference is who carries the risk of overruns, and a good partner will be clear about it. They should also give you a rough monthly estimate for model usage based on your expected volume, showing the arithmetic, rather than leaving it as an unknown that appears after launch.
Red flags to take seriously
- Guaranteed accuracy. Nobody can promise an AI system will never make a mistake. A partner who does is either inexperienced or selling.
- No questions about your data. If they quote before seeing real examples, the quote is a guess.
- Everything needs AI. Some steps are better handled with rules or plain integration. A partner who never says so is optimizing for scope, not results.
- They keep the code or accounts. Lock-in should be a choice you make knowingly, not a default buried in the contract.
- No evaluation plan. “We’ll test it” without describing how means testing will be informal and incomplete.
- Long silence before the first demo. Months of work with nothing to see is how budgets disappear.
- The people pitching are not the people building. Ask to meet the engineers.
A practical way to decide
Narrow your list to two or three partners and ask each to show working software around one of your real processes before you sign a larger contract. A short paid discovery or a small fixed-scope first phase works well. You learn more from a week of working together than from any number of proposals.
WeaverSol offers a free working AI agent demo within 24 hours of a first call, built around one of your manual processes with no commitment. If you would like to put these questions to us directly, book a call and bring your hardest ones.