Introducing Effort, a New Product Metric. Here’s What It Is and How to Measure It.
Tl;dr: “Effort” is a previously hidden measurement in funnels, which in tandem with conversion rate can instruct you on where and how to invest in your product. Without quantifying Effort, it’s easy to misinterpret what’s happening in a given user flow, which can lead a team to miss opportunities or prioritize the wrong product improvements. For every doubling of effort, we’ve observed conversion rate drops by about 16 percentage points.
The easier it is to do something, the more likely we are to do it. This simple claim underpins the biggest successes of the digital era. (See: 1-click, Amazon.) But product analytics tools today can only point you to the result of a hard-to-use product: low conversion rate in specific parts of a flow. They tell us nothing about why: is conversion low because users are encountering friction in part of a product, or because they’re not interested in continuing on?
We’re excited to now offer Effort Analysis in Heap, which pairs your funnel’s conversion rate with a new measurement that illuminates potential reasons for dropoff: Effort.
Before walking through use cases, let’s explain what Effort is and how we measure it.
What is Effort?
Whether you call it “effort,” friction, or even difficulty, Effort is meant to capture how difficult it is for users to complete a given part of a user flow.
To quantify Effort, we measure the number of interactions the average user takes to complete a step in your flow. (Interactions here means any engagements with the site or app: clicks, form changes, form submits, touches on mobile, etc.).
In many ways, the idea is simple: when lots of interactions are needed to complete a step, it’s a strong signal that the step is difficult to complete. There may be too many tasks for a user to complete. Key information may be hidden, and users are clicking around to figure out what to do next. Forms may be broken, producing consistent “rage clicks.” Directions may be non-obvious.
When steps are normally completed in one or two interactions, in contrast, it’s a strong signal that the step offers users little to no friction.
Why is this new? (And why can’t other tools do this?)
Until now, the only ways to see how much difficulty a step in a funnel posed to users — to see how hard any step was to complete — were to 1) stand over a user’s shoulder and watch them as they navigated your site; 2) use a session replay tool; or 3) wait for users to tell you. All of these methods are useful, but at bottom they all deliver qualitative signals. To understand what’s happening in the aggregate — to see what the average user experiences — requires an enormous time commitment.
Why not just measure conversion for each step? Of course you can, but conversion rates on their own tell you little about whether users are dropping off because they encounter difficulty, or because they’re simply not interested in continuing. There may be any number of reasons why people drop off, many of which have nothing at all to do with how difficult a flow is to complete. Because of this, simple conversion rates are a poor proxy for Effort.
That’s why having an immediate, quantitative metric for Effort is so useful. It can show you what parts of a funnel give users trouble, and why. It can also show you relative levels of difficulty across different funnels, so you can benchmark funnels (and individual funnel steps) against one another, and prioritize the right product fixes.
How we track Effort
We have two technical capabilities that allow us to measure Effort. The first is Autocapture, which allows us to automatically capture all user interactions with your site or app. The second is the fact that our data science layer can sift through raw interactions that haven’t been defined.
Together, these two capabilities let us see every interaction every user has in every single user flow on your site or app, and to aggregate this information to capture the average number of interactions between any two steps in any funnel.
How to use Effort (a few ways!)
Where would you invest to improve conversion?
Let’s look at a concrete example. Here’s a registration funnel with two steps. The first involves getting from the homepage to the registration page, and the second involves submitting a completed registration.
As we can see, the steps exhibit similar conversion rates. So: where should we invest if we want to increase the number of submitted registrations?
Well, we might start with two potential opportunities:
- Make the registration button more visible on the homepage, in case users are missing it and not getting to the Registration page. This would ideally increase conversion between step 1 and step 2.
- Change the registration process so it’s less difficult and more users can complete it successfully. This would ideally increase conversion between step 2 and step 3.
We may have some intuition that (1) or (2) is the bigger product opportunity, but to really know we would either need to start experimenting or look at a few dozen session replays to see what’s actually going on between the steps.
If you’ve worked with many funnels, maybe you’ve adopted a rule of thumb like this one:
- Conversion in early steps may be increased by improving the quality of traffic or visibility of calls-to-action.
- Conversion in later steps may be increased through improvements to the core product, likely by making it easier to accomplish key goals.
With Effort, we can replace that rule of thumb with real data. What if that early step actually has low conversion because of a core usability issue? What if users are stumbling on a late step not because of friction in your product but because they were never properly qualified in the first place?
In this case, Effort Measurement backs up our rule of thumb:
There’s not much to be done to better expose users on the homepage to the registration process. Most users are getting from the homepage to the registration page in just one “interaction,” which could be a click, form change, or form submission. This means this step has low motivation users: the main factor affecting conversion is how interested the user is in registering, not how hard it is to do so.
On the other hand, the average user must do 80 interactions to get from the registration page to a submitted application. Completing this step is difficult: there may be many opportunities for improving conversion by making it easier. Reducing effort can likely improve conversion through this step, and we can even predict potential improvements in conversion.
When rules of thumb fail: Effort to the rescue
Here’s an example funnel based on education content. How could we get more users to view our online courses?
Our handy rule of thumb would tell us that most users aren’t very interested in looking at courses. Those that view a listing of courses are more interested, and the low conversion rate from the listings page to a particular course may be an opportunity to make navigation simpler.
Wrong! Users are navigating from the listings page to courses just fine: the average user can do it in just one interaction. So this means that these users have low motivation.
There may be an unexpected opportunity on the homepage, however: the few users who are making it to the courses listing page are taking a surprising number of actions to do so, considering it’s linked directly. It’s surprisingly difficult to get from the homepage to the courses listing.
Without visibility on Effort, it would seem reasonable to invest in the part of this flow from courses listing to specific courses. However, those investments would be unlikely to increase the number of users viewing courses.
Classifying your steps to take the right action: the Effort matrix
The combination of Effort and conversion rate can give you a complete matrix and direct you to the correct action:
Product changes aimed at reducing friction are unlikely to help users with low motivation move from the homepage to the registration page, or move from the courses listing to a particular course. Our new “Group Suggestions” can help you drive better users by finding the referrers or UTMs most likely to convert, so you can invest more in those channels.
For difficult steps like completing a registration, we will cover in a future blog post how “Step Suggestions” can automatically “zoom in” to those parts of the flow where the most users are dropping off, in order to specifically target reductions in friction.
Estimating the impact of making difficult steps easier
Not only can Effort suggest where in a flow you should invest or not, it can estimate the potential impact of an investment.
We found that there is a strong relationship between conversion and Effort when a step requires more than four actions. Beyond four actions, every doubling in effort yields an average 16 percentage point decrease in median conversion.
For example, steps with 2–4 actions have a median conversion rate of 87%; doubling to 5–10 actions decreases conversion to 72%; doubling again to 11–21 actions decreases conversion to 61%; doubling again to 22–44 actions decreases conversion to 40%. The red slope line below estimates conversion based on this increase in effort.
The first four actions are “free”: reductions in effort below this are no longer correlated with higher conversion.
Validating our rule of thumb against real data
How universal is our rule of thumb that earlier steps with low conversion are due to low motivation, and later steps with low conversion are because of difficulty?
It’s right most of the time: first steps are more likely than any other step to have low motivation users. But many of them (30%) are difficult steps instead! If you’ve ever assumed that low conversion in a first step is because of low motivation, rather than product friction, there’s a good chance that you missed an opportunity like we saw in our online courses example above.
Likewise, while low motivation users are less likely later in funnels, there are still plenty (15%) of high conversion, low effort steps in step 3 and beyond. In these cases, Effort data can save you potential trouble trying to optimize a product experience when the real reason users aren’t advancing may be outside your product entirely.
What’s next
You can measure Effort in Heap! All existing Funnel Analyses are now supplemented with an Effort Analysis, with no work needed by you.
Effort Analysis is just the first of an exciting new array of Heap features that leverage Autocapture to suggest the most valuable interventions in your product without any manual tracking.
In our next post, we’ll cover Step Suggestions, currently in Beta, which enables you to automatically zoom in on complex or difficult parts of a flow to determine where a product change may have the most impact.
