How Wealthsimple decides what to test and what to just ship

Wealthsimple moves fast. The Canadian financial services company started as a brokerage, recently launched a banking arm, and now serves 4 million clients with everything from checking accounts and credit cards to portfolio loans, trading and investing. Launching that much, that quickly, raises a question every experimentation team eventually faces: which changes deserve a test, and which should simply ship?
On this episode of The Experimentation Edge, Delina Ivanova, who leads data science for growth and investing at Wealthsimple, tells host Ashley Stirrup how her teams answer it, and what a recent margin promotion taught them about designing experiments that measure the thing they are actually trying to drive.
Two years of making experimentation a discipline
Wealthsimple has run some version of A/B testing for as long as it has existed. What changed over the last 2 years is the rigor. The company brought in people who know how to design and run good experiments, built processes around them, and started educating stakeholders on how to make decisions with results.
There is no separate experimentation team. Wealthsimple's data scientists are full-stack, and experimentation is a core part of the job. A central group of experimentation experts sits across 3 data science teams: growth and investing, banking (including credit and market risk models), and an enablement team that supports client support, sales and brokerage operations. Each data scientist works with specific product managers or business partners to decide when a test is needed, how to design it, what success looks like and what the go/no-go decision will be.
When a test is a waste of time
Delina learned the "test or don't test" lesson at her previous company, Misplay, which she calls the strongest experimentation culture she has worked in. Misplay launched nearly everything with a test so it could measure lift and speak to the value each feature added. Eventually the team realized it was testing features that didn't need it.
Her rule of thumb today is simple. If you are removing an obvious blocker, ship it. In Canada, financial institutions require a lot of documentation, and clients are often asked to upload forms, scan PDFs or email files. Automating that friction away needs no experiment. "Anytime you have any situation like that, that's a no-brainer. I would say just go for it and don't bother testing."
Optional products are different. For a must-have like a checking account, a 10-step funnel and a 5-step funnel may convert similarly, because the client needs the account. For an optional feature, a long funnel will lose people, and that is where testing the lift pays off. Company context matters too: teams backed by PE or VC investors who must show the value of their work should test more so they can measure and report it.
Picking metrics that actually connect to the outcome
At Wealthsimple, every feature and campaign ultimately aims at deposits or revenue. But an engagement feature rarely moves revenue right away. The theory is that engagement builds familiarity, familiarity leads to account openings, accounts lead to deposits, and deposits lead to revenue.
"Sometimes those metrics are lagging metrics. We're not going to see them until 60 days out, 90 days out, especially in financial services." No experiment can run that long. So data scientists need to understand how the behavior they are driving relates to the outcome they want, and choose in-between metrics, like engagement or account opening, that signal the outcome is likely. When teams design tests quickly, Delina says, the most common gap is testing something that isn't connected to the metric they are trying to move.
What a margin promotion revealed
Wealthsimple recently launched margin borrowing, a product that is newly available to many Canadians. To grow it, the team ran promotions targeting people it believed were a strong fit, building on a small user research effort where a product manager reached out directly and saw a very high conversion rate.
The results surprised them. Instead of margin transfers, they saw a wave of deposits into all kinds of other accounts. The reason: a broad deposit offer with a 1% match was live in market at the same time, and the messaging combined both incentives. Clients responded to the most prominent message. The outcome was positive, but it wasn't the outcome the test was designed to drive.
The team took 3 lessons from it. First, decouple incentives so each message isolates one thing to learn. Second, refine the messaging itself, because many of the people targeted didn't yet understand what margin is or why it could help them; it was an education problem, not a rate problem. Third, rethink targeting. "When you have 30 or 40 offers out in the market, it gets lost." A client opening the app has limited attention, so the most relevant offer, like a margin discount for an active margin user, needs to be the one they see.
Why surveys need experiments
Wealthsimple talks to clients and runs surveys, but Delina is careful about how much weight they carry. People describe what they would ideally do, and that is often different from what they actually do. "You need to hear the thing that people tell you, but then see the behavior that they exhibit." A research strategy combined with experimentation keeps teams from investing in features based on feedback that doesn't match real behavior.
Diagnosing a losing test
When a feature you believe in loses, Delina narrows the cause to 3 options. "If something doesn't work, it's either going to be a messaging problem, a targeting problem, or an experience problem."
From individual tests to a learning system
Wealthsimple shares results in product reviews and across the organization, but Delina wants more systemic learning. In a hypergrowth phase, the company tests many ideas looking for big wins rather than optimizing incrementally, and team changes sometimes mean similar tests get repeated.
Her goal for the coming year is a meta-learning structure: putting every past experiment in one place so the team can see how each client responded to discounts, match offers, pricing and messaging over time. "Experimentation is actually a great way to learn about people." Done at scale, it becomes a built-in mega test and the foundation for a personalization engine.
She also wants more uplift modeling, because some marketing turns clients into detractors. "Sometimes when you reach out to people with marketing or a new product, they actually become detractors." Learning who wants more, who wants less and who wants nothing at all protects the relationship, especially now that AI makes it easy to produce far more marketing copy.
AI makes ideas cheaper to test
AI has already changed how her team works. It speeds up testing of messaging variants, and the team is experimenting with large language models as predictive engines to suggest the next best product for a client or personalize a message with a concrete saving. Instead of building a full machine learning model before knowing whether an idea works, the team can test the concept quickly and invest in a proper model only when it proves itself. "It's much cheaper to test ideas, which is exactly what we want with a good experimentation culture."
The takeaway
Wealthsimple's approach is a practical one for any fast-growing company: ship the obvious fixes, test what's uncertain, choose metrics that truly lead to the outcome, and design each experiment to isolate one lesson. Over time, every test becomes part of a larger system for understanding clients.
Related articles
Ready to ship faster?
No credit card required. Start with feature flags, experimentation, and product analytics — free.





