Why streaming changes the system design
A normal API request returns one complete payload. An AI request may run for many seconds and produce hundreds of incremental events. The browser can disconnect, the model provider can throttle, or a user can start a second request before the first one finishes. Your application must treat the stream as a lifecycle rather than a single response.
A practical production architecture
React interface
Submits input, renders partial output, supports cancellation, and communicates state clearly.
Next.js API boundary
Authenticates the user, validates payloads, enforces quotas, and returns a stable event stream.
AI service layer
Selects the model, builds prompts, translates provider events, and records usage.
Persistence and telemetry
Stores completed messages and captures latency, errors, tokens, cost, and safety outcomes.
This separation lets you change models without rewriting the UI. It also gives the API layer one place to enforce business rules before expensive work begins.
Design the Node.js stream as a protocol
Do not send anonymous text fragments with no meaning. Define a small event contract such as start, delta, usage, complete, and error. Include a request ID on every event so logs, traces, and user reports can be correlated.
{ "type": "delta", "requestId": "req_123", "text": "partial output" }
{ "type": "usage", "requestId": "req_123", "inputTokens": 420 }
{ "type": "complete", "requestId": "req_123" }Handle cancellation end to end
When the browser aborts a fetch, propagate that signal to the model request. Stop downstream work, close the stream, and avoid saving a partial answer as complete. Cancellation reduces wasted compute and prevents an old response from overwriting a newer one.
Build a predictable React experience
Model the interface with explicit states: idle, submitting, streaming, complete, cancelled, and failed. Disable only the controls that would create invalid actions; always keep the stop control accessible while a request is active.
- Batch UI updates when token frequency causes excessive renders.
- Keep partial output separate from the last confirmed server record.
- Show a useful retry action without silently submitting twice.
- Preserve the user prompt when a request fails.
- Announce streamed status appropriately for assistive technology.
Reliability patterns that matter
Timeout budgets
Set separate limits for connection, first token, idle stream, and total duration.
Selective retries
Retry only transient failures and only before content has been delivered to the user.
Backpressure
Respect writable-stream pressure so a slow client does not grow memory without limits.
Graceful errors
Send a safe terminal event, retain diagnostics server-side, and close resources once.
Measure the experience, not only uptime
Track time to first token, total response time, completion rate, cancellation rate, provider errors, token usage, and cost per successful task. Segment results by model, feature, and deployment version. A healthy endpoint can still feel broken if first token latency drifts upward.
Security and cost controls belong in the API
- Keep provider credentials on the server and rotate them regularly.
- Validate message shape, size, attachment type, and conversation ownership.
- Apply per-user and per-workspace rate limits before calling a model.
- Use allowlists for tools and validate every tool argument independently.
- Redact sensitive data from logs and define a retention policy.
- Set token and spending ceilings with alerts for unusual consumption.
Prompt injection is an application-security concern, not merely a prompt-writing problem. Treat model output as untrusted data and require authorization again at the moment a tool attempts a privileged action.
Pre-launch checklist
✓ A documented, versioned event contract
✓ Authentication, input validation, quotas, and token limits
✓ Cancellation propagated from browser to provider
✓ Timeouts for first token, idle streams, and total duration
✓ Tests for disconnects, malformed events, throttling, and partial failures
✓ Dashboards for latency, completion, cost, and safety outcomes
✓ Accessible loading, stop, retry, and error states
✓ A fallback plan for provider or model degradation
Build an AI feature users can trust
Endurance Softwares designs and develops production-ready Next.js, Node.js, and AI applications with security, performance, and observability built in.