Sample report
AI infrastructure audit: a sample report
This is what the one-week audit gives you: where your setup stands, what to fix first, and how. Your report covers your own setup, in the same shape.
This is a sample. Larkspur Home Goods is a made-up company, and its findings are examples of what an audit often turns up.
At a glance
- Prepared for
- Larkspur Home Goods (made up). An online home goods shop with 35 staff.
- What was reviewed
- A customer support assistant, built as a prototype: it answers questions from 1,200 help and policy documents (retrieval-augmented generation) and runs on Azure OpenAI, Azure AI Search, App Service and Blob Storage, with the code in GitHub.
- How
- One week, with read-only access to Azure and GitHub, and two calls with the team.
In short: Ready for a pilot after three fixes: secrets, deployments and a cost alert. About a third of the monthly bill can go without changing what customers see.
Scorecard
| Area | Rating | Why |
|---|---|---|
| Security | API keys in the code and app settings; every service open to the internet | |
| Cost | Search tier larger than the index needs; no budget alert; failed calls retried five times | |
| Deployments | Deployed by hand from one laptop; no staging copy | |
| Reliability | One region, and nothing handles “too many requests” (429) errors | |
| Monitoring | Logs exist but hold customers’ email addresses; no alerts | |
| Answer quality | No test questions; prompt changes judged by trying a few by hand | |
| Data | Documents stay in their own storage account, and the index is rebuilt every night |
Findings, most important first
-
1API keys are in the code and the app settings
- What I found
- The Azure OpenAI and AI Search keys are in App Service settings, and an older key is in the repository’s history.
- Why it matters
- Anyone with access to the repository, or a leaked copy of it, can run up the bill or read the index.
- The fix
- Turn on a managed identity for the app and give it access to each service, so it needs no keys. Move anything left into Key Vault, then rotate both keys.
-
2Every service is open to the internet
- What I found
- Azure OpenAI, AI Search and Storage all accept requests from any network.
- Why it matters
- A stolen key works from anywhere, and the services are exposed to scanning.
- The fix
- Put the services on a virtual network with private endpoints, or at least limit them to the app’s outbound addresses.
-
3Deployments are done by hand
- What I found
- One developer deploys from their laptop. Prompts and settings change in production without a record.
- Why it matters
- A bad change reaches customers straight away, and there’s no quick way back.
- The fix
- A GitHub Actions pipeline that tests, deploys to a staging slot, and swaps to production after an approval, with the prompt and model version kept in the repository.
-
4No budget alert, and retries multiply the bill
- What I found
- The client retries a failed call five times with no wait in between, and nothing warns when spending jumps.
- Why it matters
- During an outage or a traffic spike, each question can cost up to six times as much, and nobody notices until the invoice.
- The fix
- A monthly budget with alerts at 50%, 80% and 100%; retries capped at three, waiting longer each time.
-
5The search tier is larger than the index needs
- What I found
- The index is about 3% of the search service’s capacity, on a tier two sizes up from what it needs.
- Why it matters
- It’s the largest line on the bill and buys nothing.
- The fix
- Move to the smallest tier that fits the index with room to grow, and check again when the documents double.
-
6Logs keep customers’ email addresses
- What I found
- Application Insights stores every question in full for 90 days, including email addresses and order numbers.
- Why it matters
- Personal data kept longer than needed is a privacy risk and harder to answer for under data protection rules.
- The fix
- Mask email addresses and order numbers before logging, and keep question logs for 30 days.
-
7Rate limits aren’t handled
- What I found
- When Azure OpenAI answers “too many requests”, the customer sees an error.
- Why it matters
- Busy hours are exactly when support questions arrive.
- The fix
- Wait and retry as the response asks, show a friendly “one moment” message, and add a second deployment in another region to fall back to.
-
8Answer quality isn’t measured
- What I found
- Changes are tested by asking a few questions by hand.
- Why it matters
- A change that fixes one answer can quietly break ten others.
- The fix
- Fifty real questions with their right answers, scored on every change in the pipeline, so quality is a number you can watch.
Roadmap
- This weekFindings 1, 4 and 5: no keys in code, a budget alert, and the right search tier. Small changes, most of the risk and cost.
- Weeks 2–4Findings 2, 3 and 6: private networking, a deployment pipeline with staging, and safer logs. This is the work in Prototype to production.
- LaterFindings 7 and 8: rate-limit handling with a second region, and an answer-quality check on every change.
What happens next
We go through the report together on a 60-minute call, and you keep it whether or not you work with me after. Its fee comes off Prototype to production if you go ahead within 30 days. The audit starts from $500 (₹25,000 in India).