-
Notifications
You must be signed in to change notification settings - Fork 0
/
trd2.html
174 lines (155 loc) · 18 KB
/
trd2.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
<!DOCTYPE HTML>
<!--
Frequency by Pixelarity
pixelarity.com | [email protected]
License: pixelarity.com/license
-->
<html>
<head>
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-172833778-1"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'UA-172833778-1');
</script>
<title>mDOT - TR&D2 (Optimization)</title>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<!--[if lte IE 8]><script src="assets/js/ie/html5shiv.js"></script><![endif]-->
<link rel="stylesheet" href="assets/css/main.css" />
<!--[if lte IE 8]><link rel="stylesheet" href="assets/css/ie8.css" /><![endif]-->
</head>
<body>
<!-- Header -->
<header id="header">
<h1 id="logo" style="text-transform: none;"><a href="index.html"><img src="images/mdotLogoFadeSmall.png" alt="" height="40px;" style="vertical-align: middle;" /></a></h1>
<nav id="nav">
<ul>
<li><a href="index.html">Home</a></li>
<li>
<a href="index.html#one">About</a>
<ul>
<li><a href="vision.html">Vision</a></li>
<li><a href="innovation.html">Innovation & Impact</a></li>
<li><a href="team.html">Team</a></li>
<li><a href="EAC.html">Executive Advisors</a></li>
<li><a href="index.html#contact">Contact</a></li>
<!-- <li>
<a href="#">Submenu</a>
<ul>
<li><a href="#">Option 1</a></li>
<li><a href="#">Option 2</a></li>
<li><a href="#">Option 3</a></li>
<li><a href="#">Option 4</a></li>
</ul>
</li> -->
</ul>
</li>
<li><a href="trdhome.html">Research Projects</a>
<ul>
<li><a href="trd1.html">TR&D1 Discovery</a></li>
<li><a href="trd2.html">TR&D2 Optimization</a></li>
<li><a href="trd3.html">TR&D3 Translation</a></li>
</ul>
</li>
<li><a href="training.html">Resources</a>
<ul>
<li><a href="software.html">Software</a></li>
<li><a href="training.html">Training</a></li>
<li><a href="publications.html">Publications</a></li>
</ul>
</li>
<li><a href="index.html#two">Partnering Projects</a>
<ul>
<li><a href="cps.html">Collaborative Projects</a></li>
<li><a href="sps.html">Service Projects</a></li>
</ul>
</li>
</ul>
</nav>
</header>
<!-- Main -->
<section id="main" class="wrapper style1">
<div class="container box big bordered">
<header class="major special" style="margin-bottom: 0;">
<h2>Technology Research & Development Project 2 (TR&D2): Optimization</h2>
<p>Dynamic Optimization of Continuously Adapting mHealth Interventions via Prudent, Statistically Efficient, and Coherent Reinforcement Learning</p>
<p style="margin-top: 1em; font-size: 0.9em;">TR&D2 Lead: <a href="team.html#murphy">Susan Murphy</a></p>
<p style="margin-top: 0em; font-size: 0.9em;">Co-Investigators: <a href="team.html#kumar">Santosh Kumar</a>, <a href="team.html#marlin">Benjamin Marlin</a></p>
</header>
<!-- Content -->
<section id="content">
<a href="#" class="image main"><img src="images/bannerO.png" alt="" /></a>
<p>Mobile health (mHealth) interventions have typically used hand-crafted decision rules that map from biomarkers of an individual’s state to the selection of interventions. Recently, reinforcement learning (RL) has emerged as a promising approach for online optimization of decision rules. Continuous, passive detection of the individual’s state using mHealth biomarkers enables dynamic deployment of decision rules at the right moment, i.e., as and when events of interest are detected from sensors. RL-based optimization methods that leverage this new capability created by sensor-based biomarkers, can enable the development and optimization of temporally-precise mHealth interventions, overcoming the significant limitations of static, one-size-fits-all decision rules. Such next generation interventions have the potential to lead to greater treatment efficacy and improved long-term engagement.</p>
<p>However, there exist several critical challenges to the realization of effective, real-world RL-based interventions including the need to learn efficiently based on limited interactions with an individual while accounting for longer-term effects of intervention decisions, (i.e., to avoid habituation and ensure continued engagement), and accommodating multiple intervention components operating at different time scales and targeting different outcomes. As a result, the use of RL in mHealth interventions has mostly been limited to very few studies using basic RL methods.</p>
<p>To address these critical challenges, TR&D2 builds on more precise biomarkers of context, including <a href="trd1.html">TR&D1</a> risk and engagement scores, to develop, evaluate, and disseminate robust and data efficient RL methods and tools. These methods continually personalize the selection, adaptation and delivery timing decision rules for core intervention components so as to maximize long-term therapeutic efficacy and engagement for every individual.</p>
<!-- <h3 style="margin-top: 25px;">Innovation</h3>
<p>TR&D2 enables a broad segment of the mHealth research community to use online RL algorithms for personalizing mHealth interventions so as to maximize intervention effects and engagement. This project is innovative because the developed personalization algorithms will use models that are interpretable by mHealth intervention scientists. This, along with our track record of strong collaborations with mHealth scientists, will lead to a transformation in the role intervention scientists play in designing and evaluating these algorithms. At this time, such algorithms do not exist. This project tackles these challenges in a novel manner, combining ideas from computer science, behavioral science and statistics.</p>
-->
<h3 style="margin-top: 25px;">Research Goals</h3>
<p>TR&D2 is conducting the following innovative research to address the technological challenges described above:</p>
<ul style="padding-left: 30px;">
<li>Account for delayed treatment effects via prudent learning of decision rules. Generalize current myopic Bandit RL methods to enable learning non-myopic decision rules that account for delayed intervention effects. A particular focus is delayed effects due to intervention burden.</li>
<li>Enable efficient personalization via optimizing data sharing across users. Develop RL methods that personalize decision rules for every individual by optimally leveraging data across a population or cohort to accelerate learning.</li>
<li>Enable coherent learning of decision rules across intervention components operating at different time scales and with different objectives. Increasingly, mobile interventions include multiple components targeting different outcomes (e.g., stress, inactivity) and time scales (e.g., within day, daily). TR&D2 will develop approaches to use distal health outcomes to guide learning for these multiple components so as to minimize negative interactions.</li>
</ul>
<h3 style="margin-top: 25px;">Technological Resources</h3>
<p>TR&D2 is producing the following technological resources for the community:</p>
<ul style="padding-left: 30px;">
<li>A toolkit for online intervention optimization. This toolbox will include cloud-based modules for personalizing adaptation rules as well as smartphone modules implementing real-time intervention selection. This system component will be implemented as a cloud-based micro-service that can be flexibly integrated into other existing mobile or cloud-based platforms developed by our collaborators as well as members of the broader research community.</li>
<li>A reference tutorial for use of the online intervention optimization toolkit. Both tools will be implemented within the mDOT Center software framework and will enable a broad segment of the mHealth research community to continuously optimize mHealth intervention rules, so as to achieve optimal efficacy and engagement for individuals despite dynamic variations in the physical, behavioral, social, & environmental states.</li>
</ul>
<h3 style="margin-top: 25px;">Impact on Science & Society</h3>
<p>Research and development by TR&D2 will significantly advance RL methodology for personalizing decision rules; in particular, with regards to online algorithms that personalize interventions for each user by appropriately pooling across multiple users. Through work with <a href="cps.html">collaborative projects</a> (CPs) and <a href="sps.html">service projects</a> (SPs), TR&D2 will advance the understanding of the impact of personalization via test cases in obesity (<a href="sps.html#sp4">SP4</a>), engagement (<a href="cps.html#cp1">CP1</a>), cigarette smoking (<a href="cps.html#cp1">CP1</a>, <a href="sps.html#sp5">SP5</a>), oral health (<a href="cps.html#cp2">CP2</a>) and physical activity (<a href="cps.html#cp3">CP3</a>); this variety contains interventional components (goal setting/planning and just-in-time support) common to many mHealth interventions. TR&D2’s approaches will be used by our CPs to test whether particular aspects of state arising in dynamic behavioral theories should be intervened on in order to break links between state and adverse health outcomes. TR&D2 will inform intervention research by disseminating software systems and tools that will help to extend behavioral interventions outside of the clinical setting. This is particularly relevant for patients who have limited access to clinics. The personalization methods developed here will lay the groundwork for future projects building longer-term support, both by accommodating user non-stationarity and in maintaining longer-term user engagement. It will help establish the area of Multi-agent RL for mHealth.</p>
<h3 style="margin-top: 25px;">Recent Publications</h3>
<p>Coming soon</p>
<hr style="color: #eb4947; clear: both;" />
<h3 style="margin-top: 25px;">TR&D2 Team</h3>
<h3 style="text-decoration: underline;"><a href="http://people.seas.harvard.edu/~samurphy/" target="_blank">Susan Murphy, Ph.D.</a></h3>
<h4>TR&D2 Lead</h4>
<div style="float: left; padding: 25px;">
<p><img src="images/team/Murphy.png" alt="Susan Murphy" style="width: 150px; border-radius: 4px; border: 1px solid #ddd; padding: 5px; box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2), 0 6px 20px 0 rgba(0, 0, 0, 0.19);" /></p>
<p><img src="images/team/harvard.png" alt="Harvard" width="150px" /></p>
</div>
<p style="text-align: justify;"><a href="http://people.seas.harvard.edu/~samurphy/" target="_blank">Dr. Susan Murphy</a> is the Professor of Statistics and Computer Science, and the Radcliffe Alumnae Professor at the Radcliffe Institute, Harvard University. She directs the Statistical Reinforcement Learning Lab at Harvard University. Her research concerns clinical trial design and the development of data analytic methods for informing multi-stage decision making in health. In particular for (1) constructing individualized sequences of treatments (a.k.a., adaptive interventions) for use in informing clinical decision making and (2) constructing real-time individualized sequences of treatments (a.k.a., Just-in-Time Adaptive Interventions) delivered by mobile devices. Murphy has developed a formal model of this decision-making process and an innovative design for clinical trials called Sequential Multiple Assignment Randomized Trial (SMART) that allow researchers to optimize adaptive interventions. In 2016, she was elected a member of the National Academy of Sciences, in 2014, she was elected a member of the National Academy of Medicine, and in 2013, she was selected as a MacArthur Fellow. <strong><a href="https://scholar.google.com/citations?user=q-DPFdUAAAAJ&hl=en&oi=sra" target="_blank">Visit Google Scholar page</a></strong></p>
<hr style="color: #eb4947; clear: both;" />
<h3 style="text-decoration: underline;"><a href="https://md2k.org/santosh" target="_blank">Santosh Kumar, Ph.D.</a></h3>
<h4>Lead PI, Center Director, TR&D1, TR&D2, TR&D3</h4>
<div style="float: left; padding: 25px;">
<p><img src="images/team/Kumar_Santosh.png" alt="Santosh Kumar" style="width: 150px; border-radius: 4px; border: 1px solid #ddd; padding: 5px; box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2), 0 6px 20px 0 rgba(0, 0, 0, 0.19);" /></p>
<p><img src="images/team/memphis-logo.png" alt="University of Memphis" width="150px" /></p>
</div>
<p style="text-align: justify;"><a href="https://md2k.org/santosh" target="_blank">Dr. Santosh Kumar</a> is the Lillian and Morrie Moss Chair of Excellence Professor in the Department of Computer Science at the University of Memphis and the Director of the NIH Center of Excellence for Mobile Sensor Data-to-Knowledge (MD2K), which is headquartered at the University of Memphis. He received his Ph.D. in Computer Science and Engineering from The Ohio State University in 2006, where his dissertation won a presidential fellowship. In 2010, Popular Science magazine named him one of America’s ten most brilliant scientists under the age of 38 (called “Brilliant Ten”). In 2011, he chaired the “mHealth Evidence” meeting jointly organized by NIH, NSF, RWJF, and McKesson Foundation to establish evidence requirements for mHealth. In 2013, he was invited to meet with the NIH Director to advise him on NIH efforts in the area of mHealth and was invited to the White House to give a talk on the future of Biosensors. In 2014, he co-organized and co-chaired the NSF-NIH Workshop on Computing Challenges in Future Mobile Health (mHealth) Systems and Applications. He holds the distinction of receiving the largest grants from both NIH ($10.8 million in 2014) and NSF ($4 million In 2016) in the history of the University of Memphis. Santosh’s research seeks to define new frontiers in the discipline of mobile health (mHealth). His decade-long work has involved collecting mobile sensor data from over 100 human volunteers for 25,000+ hours in their natural environments as part of various scientific user studies. His collaborative research involves more than twenty faculty members from fifteen institutions, spanning a variety of disciplines, making his projects highly transdisciplinary. <strong><a href="https://scholar.google.com/citations?user=f0i_WNoAAAAJ&hl=en&oi=sra" target="_blank">Visit Google Scholar page</a></strong></p>
<hr style="color: #eb4947; clear: both;" />
<h3 style="text-decoration: underline;"><a href="https://groups.cs.umass.edu/marlin/" target="_blank">Benjamin Marlin, Ph.D.</a></h3>
<h4>Co-I, TR&D1, TR&D2</h4>
<div style="float: left; padding: 25px;">
<p><img src="images/team/Marlin.png" alt="Benjamin Marlin" style="width: 150px; border-radius: 4px; border: 1px solid #ddd; padding: 5px; box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2), 0 6px 20px 0 rgba(0, 0, 0, 0.19);" /></p>
<p><img src="images/team/UMass.png" alt="U Mass Amherst" width="150px" /></p>
</div>
<p style="text-align: justify;"><a href="https://groups.cs.umass.edu/marlin/" target="_blank">Dr. Benjamin Marlin</a> joined the College of Information and Computer Sciences at the University of Massachusetts Amherst in 2011. There, he co-directs the Machine Learning for Data Science lab. His current research centers on the development of customized probabilistic models and algorithms for time series with applications to the analysis of electronic health records and mobile health data. His recent work includes probabilistic models for analyzing wireless ECG data, detection of cocaine use from wireless ECG, hierarchical activity recognition from on-body sensor data with applications to smoking and eating detection, and methods for mitigating lab-to-field generalization loss in mobile health studies. Marlin is a 2014 NSF CAREER award recipient and a 2013 Yahoo! Faculty Research Engagement Program award recipient. His research has also been supported by the National Institutes of Health, the Patient-Centered Outcomes Research Institute, and the US Army Research Laboratory. Prior to joining UMass Amherst, Marlin was a fellow of the Pacific Institute for the Mathematical Sciences and the Killam Trusts at the University of British Columbia. He completed his PhD in machine learning in the Department of Computer Science at the University of Toronto. <strong><a href="https://scholar.google.com/citations?user=ey960FIAAAAJ&hl=en&oi=sra" target="_blank">Visit Google Scholar page</a></strong></p>
<hr style="color: #eb4947; clear: both;" />
</section>
</div>
</section>
<!-- Footer -->
<div id="footer">
<div class="container">
<img src="images/mdotLogoFade.png" alt="" height="100px;" style="margin-bottom: 15px; display: block; margin-left: auto; margin-right: auto;" />
<p class="copyright" style="margin-left:5%; margin-right:5%;">The mHealth Center for Discovery, Optimization & Translation of Temporally-Precise Interventions is supported by the National Institutes of Health's National Institute of Biomedical Imaging and Bioengineering through its Biomedical Technology Resource Centers Program.</p>
<p class="copyright">© <a href="https://md2k.org/" target="_blank"><span style="text-decoration: underline;">MD2K</span></a>. All rights reserved.</p>
</div>
</div>
<!-- Scripts -->
<script src="assets/js/jquery.min.js"></script>
<script src="assets/js/jquery.dropotron.min.js"></script>
<script src="assets/js/jquery.scrolly.min.js"></script>
<script src="assets/js/jquery.scrollgress.min.js"></script>
<script src="assets/js/skel.min.js"></script>
<script src="assets/js/util.js"></script>
<!--[if lte IE 8]><script src="assets/js/ie/respond.min.js"></script><![endif]-->
<script src="assets/js/main.js"></script>
</body>
</html>