Skip to content
Back to blog
Engineering

A Modern Stack for Shipping AI Products

AHAshir Haroon · CEOMay 8, 202612 min read
Engineering

An AI product is an application that calls a model. The model is one dependency. The product is the interface, the data, the permissions, the jobs that run after the user leaves, and the way you notice when answers get worse. This guide maps those layers and the choices that keep a first version from becoming a rewrite.

The defaults below match how we build: TypeScript, a web application framework, a relational database, and a provider-agnostic place to call models. They are defaults, not a requirement that every project copy them. The point is to know which layer owns which problem.

Frontend

The interface has to show uncertainty. Streaming tokens, a citation the user can open, a disabled button while a tool runs, and an explicit error when retrieval finds nothing are product decisions. A single spinner over a thirty-second call is how people learn not to trust the feature.

Keep the client thin. The browser should not hold API keys, and it should not decide which documents the user may see. Send the user’s action to your server, and let the server assemble context. For most products we use a React and Next.js frontend because it can render the marketing site, the signed-in app, and the server handlers in one codebase. A separate native client can call the same APIs later.

Backend and APIs

The backend is where policy lives. It authenticates the user, checks what they may do, loads or retrieves data, calls the model, validates the model’s proposed action, and writes the result. If that logic sits in a prompt, you cannot test it and you cannot audit it.

Expose a small API: one endpoint to start a task, one to fetch status, and explicit endpoints for actions such as “create the draft” or “send the message.” Do not give the model a generic “run SQL” route. API integrations with the systems you already run (CRM, billing, scheduling, a fleet TMS) belong behind those same checks.

LLM providers and model routing

Call models through one internal interface. The application asks for a capability (“draft a reply,” “extract these fields,” “embed this passage”) and a router picks a provider and a model. When a vendor changes price, latency, or terms, you change the router, not every screen.

Routing rules that earn their keep are boring: a smaller model for classification, a stronger model for long reasoning, a fallback when the primary provider errors, and a hard stop when the request contains a data class that provider is not allowed to see. Log the model name and the token counts on every call. Without that, cost reviews are guesswork.

Embeddings, vector databases, and RAG

Embeddings turn text into vectors so you can search by meaning. A vector database stores those vectors and returns nearest neighbors. That pair is the retrieval half of RAG. It is the right tool when the model must answer from documents that change and that were not in its training data.

It is the wrong tool when the question is a lookup you can do in SQL: “what is this customer’s open balance,” “which loads delivered yesterday.” Exact facts belong in the database, with the model used only to phrase the result if a sentence is even needed. The longer explanation is in RAG knowledge systems.

Queues and background jobs

Ingestion, re-embedding, evaluation runs, and outbound messages should not live inside the user’s HTTP request. Put them on a queue. A job can retry, it can resume, and it can record each step. A request that times out at a proxy leaves you with a half-written index and no record of why.

Anything multi-step (fetch documents, chunk, embed, then mark the source as ready) needs a status the operator can see. “The worker crashed” is an acceptable state if the UI says so and the job can be rerun without duplicating chunks.

Databases

Use a relational database for users, tenants, permissions, documents’ metadata, job state, and the audit log. Use object storage for the original files. Use the vector store for similarity search only. Teams get into trouble when the vector store becomes the only copy of a policy, because it is a poor system of record: updates are awkward, transactions are limited, and permissions are easy to forget.

Postgres is our usual core because it is well understood, it can store JSON when a shape is still moving, and it keeps operational data next to the metadata you will join against retrieval results. The vector index can sit beside it or in a dedicated engine. Either is fine if the document ID in the index matches a row you can authorize.

Authentication

Use the identity system the rest of the product uses. AI routes are not a side door. Session or token, role, and tenant should be available before any retrieval or tool call. Service accounts for background jobs need their own identity and a narrower scope than a human admin.

Observability, evaluation, and caching

Observability for an AI feature is traces plus content you are allowed to store. A trace should show the user, the route, the retrieval query’s document IDs, the model, the latency, the token use, and whether a tool ran. Storing full prompts is a security decision, covered in AI security and compliance. At minimum, store enough to reproduce a failure on synthetic data.

Evaluation is a set of examples with expected behavior, run when you change a prompt, a chunking rule, or a model. Include cases that should refuse, cases that should cite a specific document, and cases that should call no tools. A vibe check in a demo is not an evaluation set. You do not need hundreds of rows to start. You need rows that represent the failures you cannot afford.

Cache stable work: embeddings for unchanged chunks, and identical classification calls if the input is not sensitive and the policy allows it. Do not cache personalized answers across users. A shared cache key that omits the tenant is a data leak.

Rate limiting

Limit by user and by tenant, not only by IP. Model calls are expensive and they are an abuse path. Return a clear error when the limit hits, and apply a stricter limit to unauthenticated endpoints. Background jobs need their own budget so a re-index cannot starve the product.

Deployment, CI/CD, and security

Ship prompts and evaluation fixtures in the same repository as the code, so a change is reviewable. Run the evaluation set in CI when those files change, and block the deploy on the failures you marked as critical. Secrets stay in the host’s secret store. Preview environments should use synthetic data or a provider tier that does not retain customer prompts.

The production checklist is the same as any other web app, plus the model boundary: dependency updates, least-privilege roles, backups of the system of record, and an owner for vendor status. Enterprise web applications and custom software engagements at Vyntrix start from that production bar rather than from a notebook that happens to call an API.

Hosted models or self-hosted models

A decision aid, not a rule. Data-residency and contract terms can override it.
SituationUsually a better fit
You need to ship a feature and the data is not highly restrictedHosted API, enterprise terms reviewed
Prompts must stay in infrastructure you operateSelf-hosted or a provider region and contract that meets the requirement
The task is narrow and high volumeA smaller hosted model, or a self-hosted model if unit cost dominates
You do not have people to patch and scale GPU servicesHosted, until that operating cost is real
You need the newest general-purpose qualityHosted frontier model, behind your router so you can leave

Self-hosting is an operations commitment: hardware or GPU instances, model updates, and capacity planning. Hosting is a vendor commitment: retention, subprocessors, and outages. Write down which commitment you are accepting. “We will decide later” usually means customer data is already going to whichever key was pasted into the demo.

A minimum architecture

  1. Web app and server API in one reviewed codebase.
  2. Relational database for users, permissions, and audit metadata.
  3. Object storage for source files.
  4. A single model client with routing, timeouts, and token logs.
  5. A queue for ingestion and other multi-step work.
  6. Vector search only where similarity is actually the query.
  7. An evaluation set that runs before you change prompts or models.
  8. Rate limits and an off switch per feature.

Conclusion

The stack is modern because the boundaries are clear, not because every layer uses a new product. Put policy in the server, facts in a database, files in object storage, similarity search in a vector index, and model choice behind a router you can change. AI agents sit on top of that stack. They do not replace it.

Frequently asked questions

Do we need a vector database on day one?

Only if the feature answers from a changing document set using similarity. Lookups, transactions, and workflows should use your existing database and APIs. Adding a vector index before you know the questions usually creates a second, weaker copy of the data.

Should prompts live in the database or in code?

Start in code so changes are reviewed and tied to an evaluation run. Move a prompt to a database when non-engineers must edit it and you still have a review and a way to roll back.

Is serverless required?

No. Serverless is a convenient default for spiky web traffic. Long ingestion jobs and self-hosted models often fit a worker and a queue better. Choose the runtime for the job, not for the label.

AH

Ashir Haroon

CEO at Vyntrix Labs

Let's build something intelligent

Ready to automate the busywork and scale with AI?

Book a free discovery call and we'll map your highest-impact automation opportunities, no obligation, no jargon.