How do we know whether technical problems are affecting customers?
Short answer
Observe the customer journeys that matter, not just whether servers are running. Connect product events with errors, latency and operational changes so you can see where people fail or abandon a task. Define success and failure consistently, then use those signals to investigate quality and prioritise improvements with the team.
Begin with a useful customer outcome
For an event product, a useful signal might be whether a participant can join and hear the session. A successful HTTP response alone does not establish that outcome. For another product, the critical journey might be a purchase or a completed analysis.
Agree the start and end of the journey, what counts as failure and which users or environments may be affected differently. Avoid collecting personal data merely because instrumentation makes it possible.
Connect product signals with operational evidence
Look at completion and abandonment alongside response times, errors and recent releases. Use consistent identifiers where appropriate to trace the affected workflow without turning monitoring into a store of unnecessary customer content.
A change in conversion is not automatically a technical failure: customer mix, pricing and product changes can also explain it. Product analytics identifies a question; operational evidence helps test the explanation.
Work backwards to quality
- Identify a customer journey and agree its success signal.
- Establish a baseline and check the instrumentation against real behaviour.
- Investigate the failures with the greatest customer or business impact.
- Improve the relevant code, tests, deployment process or decision.
- Check whether the customer outcome improved after the change.
This is how I approach quality: define the outcome, establish how we will know it is happening, then work backwards to engineering practice.
Related questions
Do we need another monitoring tool?
Not necessarily. Start with the questions your existing tools cannot answer and check whether the missing piece is instrumentation, access to data or agreement about what matters.
Should business and engineering use the same dashboard?
They should share definitions and be able to connect the evidence. Different views can serve different decisions, provided they describe the same customer journey consistently.