Start with people
Who is this meant to help?
Before evaluating a model or feature, the review should establish the user, the problem, and the decision the system will influence. “AI-powered” describes an implementation choice, not a user benefit. If a simple form, a search function, a transparent rule, or a better explanation solves the problem, the added model may not be justified.
For a real proposal, the review should record who benefits, who may be harmed, who bears the cost of a wrong output, and who can stop or correct the process. Those answers matter most when the feature can affect money, services, education, employment, safety, or a person’s reputation.
Risk before demo
Test the downside before celebrating the first good answer
A useful demo proves that one example can work; it does not establish how often the feature will fail or who will absorb the cost. Depending on the task, failures might include false information, missing context, prompt injection, exposed data, stale results, or an answer where the system should have stopped and asked for review. The risk depends on what the output is allowed to trigger.
- Define acceptable and unacceptable failure modes.
- Test ordinary, edge, ambiguous, and adversarial inputs.
- Measure against a baseline that does not use AI.
- Set a confidence or escalation policy that matches the risk; do not rely on a model’s self-reported confidence alone.
- Decide when a human must review the output before action.
Data handling
Do not send information just because a field accepts it
Data sent to an AI service may be processed according to that service’s plan, settings, retention rules, and connected-app permissions. Review those arrangements before sending confidential or personal information. Minimize what is shared, redact identifiers where practical, and avoid credentials or secrets entirely.
For internal systems, define who can access prompts, outputs, logs, attachments, and audit records; how long they are retained; and how deletion works. Treat retrieval, plugins, file uploads, and connected data sources as part of the security boundary.
Human control
Make uncertainty and correction part of the design
People affected by an AI-assisted decision need a meaningful path to ask questions, correct input data, challenge an output, or reach a responsible person. A generic “AI may make mistakes” footer is not enough when the output is used to take an action.
- Show where information came from when source-backed answers are expected.
- Label generated drafts or automated summaries so users know what they are reading.
- Provide a clear human fallback for high-impact cases.
- Keep an audit trail appropriate to the risk without over-collecting personal data.
- Monitor failures after launch and make it possible to disable a harmful feature.
A short review
Questions to answer before deployment
- What user problem does this solve better than the simpler alternative?
- What evidence supports the claimed improvement?
- Which users or situations perform worse?
- What data leaves the local system, and under whose control?
- How can a person verify, correct, contest, or reverse the outcome?
- Who will monitor the system and respond when it fails?
For a structured risk-management vocabulary, start with the NIST AI Risk Management Framework resources. A framework does not make a product responsible by itself, but it can help turn broad principles into reviewable decisions.