From two-day data requests to self-serve dashboards

The problem

Every question about how customers used Cloud One Conformity went through engineering. Someone raised a ticket, an engineer wrote a query and ran it on the production database, and the answer came back as a raw table. Depending on priority and who was free, that took one to two days.

That process had four costs. Questions went unasked because answers were slow. Engineers spent time on queries instead of the product. Every query ran against the live database that customers depended on. And the database held only the current state, so questions about trends could not be answered at all, such as how a customer’s compliance score had changed since customer success started working with them.

The same kinds of request kept reaching me: product managers deciding where engineering time should go, designers looking for drop-off in workflows, customer success tracking their accounts, and engineers asking whether what they built was used. I wrote a proposal for an analytics platform, arguing that without data-driven decisions our competitors could out-execute us, and got buy-in to build it, with me as the product manager.

Defining what to build

I treated the platform as a product with internal users. The goal for the first version was to give teams the ability to explore the data, and to show with a dashboard what that made possible. It had three requirements that pulled against each other: answer questions without touching production, never expose customer data, and keep the initial cost low with room to scale. I gathered requirements from security and compliance (retention, governance and anonymisation), from customer success (the dashboards they needed first) and from product managers (usage data), and drew the workflow that became the basis of the architecture.

The first build covered Conformity alone; the plan to extend it to the rest of Cloud One came once it was working. Because the architecture was new to us, I spent the planning phase on a proof of concept with the data engineers, the security team and an AWS specialist before committing to the build. The work ran in four stages: requirements and sign-off (March to May 2021), proof of concept and planning (May to August), build (September to November), and testing and rollout (December 2021 to January 2022).

The trade-offs

AWS’s own services, not a third-party tool. Commercial pipeline tools would have been quicker to set up. But we were a security company handling customer data, and after working it through with the security and compliance team, we ruled them out. We built the pipeline with AWS Glue feeding a data lake on Amazon S3. It took a little longer and kept the data inside our own controls.

Serverless, not a data warehouse. With a data engineer, I sized our data, its expected growth, the computing power we needed and the cost. Amazon Redshift, AWS’s data warehouse, suits very large datasets; ours were small to medium. The serverless design was forecast at 50% lower cost on our data and query volumes. A warehouse would have been easier to maintain, but a low initial build cost with room to scale was the better bet for a first version.

Correct data over less work. Exploring the data, I found that the production database counted far more customers than we had, because its sync with the subscription system was faulty. Engineering’s planned fix would have made paying customers hard to tell from trial users: simpler for them, but it would have broken every customer analysis. I agreed with the team that owned the subscription system to load its data straight into the data lake, and took the change through approval. It added scope, and it meant our numbers came from the source of truth. For the same reason, I added a step that flagged internal staff accounts before anonymisation, while they could still be identified.

Pause when customer data is at risk. When the Log4j vulnerability broke in December 2021, part of our pipeline was exposed. I switched off the scheduled data extraction until it was safe; a delay was a smaller risk than exposing customer information.

Customer data could not leave its country of origin, so the stack ran in each AWS region where we had customers.

What changed

Product, marketing and business teams answered routine questions from dashboards instead of raising a ticket, and people who wanted to build their own dashboards had access to the tools. Analysis moved off the production database, and for the first time we had history to analyse. The QuickSight dashboards reduced processing time by 25% and informed decisions that lifted feature adoption by 15%. That rise did not come from one feature: it came from the actions product managers took once they could see how customers used the product.

The data changed product decisions. Usage showed which security rules to keep or retire, and some rules for IAM services were deprecated. We tracked which communication channels customers used, and segmented customers by account size, number of rules run and risk, which gave us a clearer picture of who our customers were. The threat research team used the data in its published research, including a report on the cloud misconfigurations companies most often make.

Once the first build was working for Conformity, I extended the same setup to other Cloud One products. In all, the BI infrastructure was implemented across five product teams in 15 months.

What I would do differently

I would put the dashboard tool first. We started with Athena for queries and SageMaker for exploration. SageMaker is flexible but poor for sharing results, so QuickSight came later than it should have, and so did the value for stakeholders.

I would also write a data strategy before building: goals, roles, architecture and data management. We were understaffed, and a strategy would have made the case for the right team and sharper priorities.