Today, Dreem’s Head of Analytics, Artemis Llamosi PhD, takes us through the story behind the Adventurers program, Dreem’s initiative to broaden the horizons of sleep science.
This duo of articles tells the story of how we designed tools, processes — and I would even say a culture — in order to embrace the wisdom of the crowd on a larger scale than any other sleep company.
It’s about how and why we started a program involving thousands of sleepers to collectively map the unknown territory of sleep and ultimately invent solutions tailored to each individual.
This article covers the why, the vision, the scope, the second part covers the how and the who (not the band!).
The Bad Sleep Epidemic
The Nightmare- Henry Fuseli
Bad sleep is a killer. It’s a global epidemic with disastrous consequences, both in its short-term acute form (with people dying from lack of sleep, or sleep-related incidents) and in its long-term chronic form (with people suffering from bad sleep for years lowering their life expectancy, increasing risk of other diseases etc.). What’s more, bad sleep preys on the weak. It’s frequent bedfellow to other physical and mental health conditions, creating particularly toxic vicious cycles (a phenomenon known as comorbidity).
At Dreem we fight every day to provide users with means to resolve or at least alleviate their bad sleep without relying on sleeping pills which do more harm than good in most cases. To fight bad sleep, we make the best weapons and techniques from neuro and sleep science accessible to the largest number of people possible.
At the heart of our approach is the Dreem Band a device which hides a top-notch miniaturized sleep lab beneath its soft, lightweight exterior. This wearable measures EEG (electrical brain activity), heart rate, respiration and movements. Forget approximative actimetry (Fitbit, Apple Watch etc.) or even more suspicious sound-based sleep staging, Dreem is the only consumer device producing scientific grade sleep data. Great measurement is not the goal per se, more like a requirement in helping people fixing their sleep through active features (deep sleep stimulation, sleep onset relaxation techniques, smart alarm etc.) and dedicated coaching (from both algorithms and human experts).
Your humble servant and his Dreem headband
The myth of the average man and the wisdom of the crowd
Just like any company with a broad and bold mission, we’d love to have a one-size-fits-all, silver-bullet solution to all problems. This is impossible, because bad sleep is not a single problem, but myriad stories, causes and consequences.
To say that nearly everything in biology is variable is to state the obvious. But many people don’t realize just how wild it is in reality. Biological variability comes in two flavors:
- differences between individuals (a.k.a. inter-individual variability)
- differences in time for a same person (a.k.a intra-individual variability)
Some of our clients experience bad sleep on a daily basis, some suffer from erratic crises of insomnia while some have chaotic sleep rhythms because of a newborn baby or a pressing deadline. Personal and temporal context are key to understanding and possibly improving sleep.
One of the biggest issues when translating state-of-the-art science into mainstream products is that we lack context. We are only given a recipe for the average man. Average solutions designed for the average man are therefore partially satisfying at best. There are some instances where the average man represents no one really. The average is therefore only a starting point from which we must explore to find individualised solutions that fit personal singularities.
To grasp the reality of sleep diversity and propose concrete improvement requires help from the wisdom of the crowd. User testing has always been the central source of truth at Dreem, guiding our R&D and with the Adventurers program, we took it to the next level.
A scalpel and a shotgun: a complementary toolset
Sleep was, until rather recently, a very mysterious phenomenon (and it still is for a large part). To decipher its fundamental components like sleep stages, or its relation to overall physiology, cognition and memory, the most precise machines and measurements have been necessary. Meticulous experiments were needed to control and remove as much as possible any variability so as to reveal trends and mechanisms. In this respect, clinical research is like a scalpel: sharp, precise and a bit cold actually.
Best night ever!
Experiments in a sleep lab aren’t the most fun way to spend a night. You’re hooked up to a dozens of electrodes — literally glued to your skin — on your scalp, your face, your torso and legs. You feel the pressure of breathing belts or breathing through masks. All this is connected by a mess of wires and tubes to large recorders. You’re in an unfamiliar place, in an unfamiliar bed, sometimes with video cameras and microphones pointing at you and you’re on a schedule. Ready to sleep?
Everything in this is about control. Imposing conditions, hunting down all unwanted variabilities.
Is all this really necessary?
Rigorous scientific method requires such draconian measures in many instances. Without it, nothing can emerge from the mist of variability. Nevertheless, the scalpel is not the only tool to study sleep, one could also bring … a shotgun.
To understand why we’re talking of shotguns and scalpels instead of fluffy pillows and peaceful dreams, you need to know a little about a specific chapter in another modern scientific saga: Genetics, and more precisely the Human Genome Project (HGP).
The HGP was an international consortium aiming to sequence the whole genome of a single human for the very first time. It was (and possibly still is) the largest undertaking in the history of biological science. At the time, sequencing was only possible using research-grade equipment and pretty short DNA pieces (~500 bases). To sequence the full 3 billion bases or so of the human genome, the HGP consortium employed a hierarchical method:
- First, precisely cutting and isolating the genome into tidy pieces.
- Then, cutting these pieces in small fragments at random which would be sequenced individually and reassembled by algorithms.
You saw it coming, the first part was a typical scalpel technique whereas the second was termed the shotgun method (for its shredding DNA in tiny random pieces). The first step was all about restraining the variability and then the shotgun was used in a small enough context to apply simple aligning computations.
This was a colossal endeavour with 20 universities in US, UK, France, Germany, Japan and China, employing around 5000 people, it lasted 13 years (from 1990 to 2003) and required $3.8 billion of public spending (around $ 5.2B in 2018 dollars) to finally achieve a mosaic human genome which was by construction the average human genetically speaking (derived from 13 American volunteers from Buffalo, NY male and female).
Toward the end of this project, a private company (Celera Genetics) proposed to achieve the same task within 3 years and with a $300 million budget using another method: whole genome shotgun sequencing. The idea is fairly simple: get rid of the scalpel step and do all the sequencing using the shotgun approach. The fundamental bet was that, leveraging on modern computers and advanced algorithms, enough of this less pure data would produce a result as good and as precise, for only a fraction of the price and much faster.
As it turned out, whole genome shotgun was indeed possible and accurate, becoming the standard method in genome sequencing.
We want the Adventurers program to be the shotgun method for sleep science.
For sure, Dreem can’t produce in a single night, measurements that are completely equivalent to a full polysomnography. Of course, we can’t (and honestly don’t want to) control all the parameters that surround a night. But Dreem is worn every night, by thousands of users. Through the sheer force of large statistical samples and advanced algorithms, we strongly believe we can separate the trend from the variability and achieve equal or superior precision as scalpel sleep science on many topics.
Maybe even more interestingly, this approach unlocks a different perspective: study variability itself. Dreem is not mapping a single average man, we are mapping a population, and we’re mapping its evolution through time.
Although I hope you are as thrilled as we are about this, I need to stress again that shotgun and scalpel are not opposing but complementary tools. Even in the story with the Human Genome Project, it’s important to remember that the scalpel approach, even if it proved less efficient for the precise purpose of sequencing in the end, spilled over countless invaluable results and techniques that ignited innovations which already worth hundreds of billion dollars in 2010.
Scalpel science is always the starting point of Dreem’s work (indeed we have partnership and candidate PhDs using scalpel science). We simply want to fuel sleep science with the complementary shotgun.
Adventurers are now exploring the jungle of sleep
Adventurers are Dreem users who volunteer to participate in a research journey with us.
What does it mean in practice?
From time to time, Adventurers receive propositions of missions (the name we now give to test sessions after a user coined the term). Given the mission’s objectives and requirements, they can decide to enroll. Then for a few weeks, some aspects of the normal Dreem experience will be altered and specific feedback will be asked. In the comfort of their own bed, on their own schedule, Adventurers are collectively mapping sleep.
Why do people do this?
They can have varied motivations such as benefiting of our latest release in advance or simply because they want to be able to contribute to the greater good. We have a remarkable user base who understand how hard and personal sleep issues are. Being an Adventurer allows you to act and not only wait and hope. Both personally and on behalf of Dreem, I would like to express our immense gratitude to all participants. You are amazing!
Who are the Adventurers?
The following figures illustrate how different the Adventurers program is to typical sleep studies:
In general, sleep studies are typically performed on 10 to 50 men (to avoid women’s hormonal “perturbations”…) in their twenties or thirties, for one or two nights per user. In contrast, Adventurers are a more representative sample of the general population. One mission means hundreds of volunteers from many countries, dozens of nights (for sure, it is not a perfect sample yet, our gender ratio could be better, and we are limited to western countries where Dreem is currently certified, but it is already something isn’t it?).
Yet, maybe the most important difference between the Adventurers and typical sleep studies cohorts is that we do not sort them. With the exception of very specific cases when we may discourage some users to participate in a given mission, we do not select participants. This is very different from sleep studies which either focus on healthy subjects (meaning young and not drinking too much coffee) or people with usually severe and well-defined pathologies. This is the way scalpel science works. But we believe most of us are in neither categories, and this is what we are interested in, the vast majority that lies in-between.
Challenges in world-wide remote testing for IoT and sleep science
Some of you might be saying, “Wait a minute, isn’t this just good old A/B testing? Why the fuss?”
Well… Yes and no.
Essentially, A/B testing is a form of remote experiment and so is Adventurers. Technically speaking, an objective of the Adventurers testing system is to be as easy to use as an A/B testing feature for an app or a website. Those familiar will know it’s not that easy to do and all difficulties that apply to A/B testing apply here as well. But we need to go further.
Most A/B testing has a simple design:
- two conditions, (A and B)
- one target output (e.g. a transformation rate / session duration etc.)
- between-users or between-sessions comparisons (e.g. compare users seeing website A or website B, compare sessions of website version A and sessions of website versions B)
Unlike traditional A/B testing, Adventurers mission protocols are usually more complex (and often much much more complex):
- We are not trying A vs B, but A vs B vs C vs D … or tuning a continuous parameter, or tuning several parameters together.
- We usually have several target outputs (both objective and subjective, user-based metrics).
- We usually use a combination of within-user and between-user comparison (i.e. compare for each user A, B, C etc).
- All this requires a more careful design and a more versatile test infrastructure.
In addition to the protocol’s complexity, the technical challenges are also greater than in most A/B test use-cases. Results we are interested in (like subjective restfulness, cognitive performance or measured sleep duration) come from the interplay between: the Dreem Band — the companion apps (iOS and Android, multi-language) — the users’ lifestyle, schedule etc. To gather proper data, we need to act upon these three elements, at the same time, all in sync.
Where we are heading?
It is still early days, our very first Adventurers began their very first mission around a year ago. We’ve achieved a lot already but so many exciting things are still ahead. Again, Adventurers themselves have proven invaluable sources of feedback in improving our system.
Although I wouldn’t want to spoil everything, here are some insights into what will come next:
Always more flexible
In October 2018 we released a brand new app, and a completely new set of features: Dreem Coach. We’re now leveraging upon this new app to boost further the flexibility of the Adventurers missions.
In 2019, we will use the Adventurers framework to conduct clinical trials, blending the rigor of clinical trials with the decentralization and scale permitted by our testing system.
Sharing the Adventure
We are looking at different ways of sharing the testing capability of the Adventurers system within academic and corporate partnerships.
New mission topics, broader toolkit
The first year was mostly about deep sleep. Although we still have very exciting missions on this in our pipeline, we significantly broadened the scope of trials and tests we propose to users and will continue to do so. Additional hot topics for 2019 include sleep onset, relaxation and the relation between sleep and lifestyle.
Also, quantitative testing isn’t always necessary or the most relevant. In particular, in the early stages of a feature, gathering traditional feedback is the way to go. Yet, how, when, and to whom you ask is everything. That’s why we are investing in building or integrating versatile and focused feedback gathering tools and processes. That way we can improve direction early and quickly, before optimization kicks in.
Good products and services are refined by their users. Great ones are defined by their users. Our role is to channel the crowd’s voice and crystallize it. Long live the crowd!
For readers interested in the details of how the Adventurers test platform operates and how it was built, part two of this article series: The clockwork of a nearly invisible experience is coming to the blog soon.