Sunday, April 16, 2017

Participating in the March for Science

I plan to participate in the March for Science in Portland on April 22nd. I did not come to this decision lightly. This was different, for example, from my decision to participate in the Women’s March in January. That was a very easy decision to make based on my political views.

But science is more than just politics to me. It is of course my livelihood, as I am a faculty at Oregon Health & Science University (OHSU) and one whose work is supported by public funds. Science is also, however, the dispassionate pursuit - to the best of human ability - for discerning truth. As such, I do not want to see science subverted for political or other aims. I also want to be careful that others do not subvert the message of the march itself, which in my view is to inform people about the value of public support and taxpayer funding of science.

I am pleased that many organizations have reached similar decisions. The American Association for the Advancement of Science (AAAS) has endorsed the march, and I agree with their statement that the march is a "nonpartisan set of activities that aim to promote science education and the use of scientific evidence to inform policy." I am pleased that OHSU supports the march as well.

In the end, my concerns about the real threats to science outweighed my worries about science being subverted by politics. I consider the threats to science by the current political leadership of the US to be significant. I do not consider science to be a partisan subject. I cannot look at climate change, gun violence, or immunization-preventable diseases and state that research about them is driven by an ideological or political agenda. Yes, science is always full of disagreement and is never truly “settled,” but there are bounds of truth and there is always a need to probe even what we believe to be true further. One of the beauties of science and its dispassionate search for answers is that it is self-correcting. So when science gets something wrong, there is a good likelihood that it will be corrected by further research.

Of course, I also recognize that the public purse to fund science, or anything else that the government funds, is not unlimited. That is why we have a political process to debate and enact appropriate amounts of taxation and public spending. It is also important to remember that basic research funded by the government is not research that would be funded by private industry. In fact, industry well knows it benefits from the basic research that enables companies to develop profitable products.

I also believe that the scientific enterprise in the US is a very efficient and effective allocation of tax dollars. The National Institutes of Health (NIH) budget of about $30 billion is about 1% of overall federal spending. Federal research grants not only fund scientific research but also education and training for the next generations of scientists, clinicians, and others. NIH funding is mostly awarded through highly competitive funding opportunities that often times only have success rates of 10-15%. Despite what detractors says, the life of writing grant proposals is not a cushy one.

Even beyond the research itself, the money spent on scientific research brings money back to communities. When a faculty like myself is awarded a grant, that money not only advances research and education, but it creates jobs for people in the local community. In turn, all of those who are funded by the grant turn around and spend money in grocery stores, restaurants, and other local businesses. And of course the reality is that if my institution and state and local governments were not funded by this money, it would end up in other states. OHSU commissioned a study about five years ago showing a multiplier effect in terms of money spent on the institution having an impact back on the local economy.

Although this march is about science generally, I hope that some other points specific to biomedical research come across. As pointed out by OHSU leadership, an abrupt cut in funding, such as the 18% cut to NIH proposed by the Trump Administration, will have outsized impact due to the fact that most NIH grants are multi-year awards. This means that only a portion of a given year’s funding goes to new projects. An abrupt cut will mean that for the first year of the big cuts, very few new grants would be able to be awarded. Given how competitive the environment for funding already is, we stand to lose the momentum of both established and emerging scientists.

I also hope that another point that comes out is the misguided plan to in essence eliminate the Agency for Healthcare Research & Quality (AHRQ). While it will ostensibly be folded into NIH (AHRQ currently exists within the Department of Health and Human Services but outside NIH), the claim that its research is duplicated by other NIH entities is simply not true. AHRQ performs critical and novel research in under-researched areas of health and healthcare, such as patient safety, healthcare quality, and evidence-based medicine. As with other basic research, industry may benefit and even develop products from this research, the basic research is too far removed from their product cycle for them to want to fund it. As this is not the first time that efforts have been made to de-fund AHRQ, I have written about the value of AHRQ before.

I look forward to participating in the March for Science and advocating for the benefits of scientific research and its funding by the federal government. I hope the outpouring of support will education the downside to neglecting basic scientific research and the importance of training new scientists.

Sunday, February 19, 2017

Big Change, Little Change: OHSU Biomedical Informatics Graduate Program Renames Tracks

The Oregon Health & Science University (OHSU) Biomedical Informatics Graduate Program is renaming the two tracks of its program. While the changes to the names of the tracks are small, they reflect the big changes in the field and evolving content of the curriculum.

Since 2006, the program has had two “tracks,” which have been called Clinical Informatics (CI) and Bioinformatics & Computational Biology (BCB). These two pathways through the program have been called “tracks” because they represent two different foci within the larger field of biomedical informatics, which is the discipline that acquires, organizes, and uses data, information, and knowledge to advance health-related sciences. Historically, the differences between the tracks represented their informatics focus, in particular people, populations, and healthcare (clinical informatics) vs. cellular and molecular biology, genomics, and imaging (bioinformatics).

In recent years, however, these distinctions have blurred as “omics” science has worked its way into clinical medicine. At the same time, health, healthcare, and public health have become much more data-driven, due in no small part to the large-scale adoption of electronic health records. As such, the two tracks have begun to represent different but still distinct foci, mostly in their depth of quantitative methods (deep vs. applied) but also in coverage of other topics (e.g., system implementation, especially in complex health environments; usability; and clinical data quality and standards).

The program believes that both tracks possess a set of common competencies at a high level that reflect the essential knowledge and skills of individuals who work in biomedical informatics. The curriculum organizes these competencies into “domains,” which are groups of required and elective courses that comprise the core curriculum of each track. To reflect the evolution of the program, the program has renamed the BCB track to Bioinformatics and Computational Biomedicine (still abbreviated BCB) and the CI track to Health and Clinical Informatics (now to be abbreviated HCI). The table lists below lists the common competencies and the names of the domains for each track. Each of the domains contains required courses, individual competency courses (where students are required to select a certain number of courses from a larger list, which used to be called “k of n” courses), and elective courses.


The program will continue the overall structure of the curriculum with the “knowledge base” that represents the core curriculum of the master’s degree and the base curriculum for advanced study in the PhD program. A thesis or capstone is added to the knowledge base to qualify for the MS or MBI (latter in the HCI Track only) degrees, respectively. Additional courses are required for the PhD, ultimately culminating in a dissertation.

The materials and Web site for the program will be updated quickly to reflect the new names. The program will also be evolving course content as well as introducing new courses to reflect the foci of the new tracks. The program still fundamentally aims to train future researchers and leaders in the field of biomedical informatics.

Wednesday, February 1, 2017

A New Textbook on Health Systems Science

Many aspects of academic medicine, from the structure of the medical school curriculum to the organization of departments in Schools of Medicine, are neatly segregated into two buckets: basic science and clinical science. In the jargon of medical schools and education, basic science refers to the basic biomedical sciences that have traditionally been taught in the first two years of medical school, such as anatomy, physiology, biochemistry, and pharmacology. While plenty of clinical material has migrated into the first two years of medical school over the years, such as learning to interact professionally with patients and perform a physical examination, the main focus of the first half of medical school has historically been on basic science, culminating in the US Medical Education Licensure Examination (USMLE) Step 1 exam.

Once students finish their basic science years, they move on to the clinical sciences, where they begin rotations, also called clerkships or clinical experiences. They usually first rotate through the core medical specialties, i.e., internal medicine, surgery, pediatrics, obstetrics/gynecology, and psychiatry. This is then followed by rotations in other specialties and subspecialties, ultimately leading to graduation and the start of their residency training.

This division of medical education goes beyond just the medical school curriculum. The organizational structure in most medical schools is to group academic departments into basic science and clinical departments. These two types of departments usually have different funding models. Basic science departments are usually funded by base budgets for teaching and grants for research, with an expectation that just about all faculty have research grant funding. Clinical departments have base budgets and research programs as well, but they perform another activity, which is clinical care that provides practice opportunities (and revenues) for faculty and learning experiences for students, residents, and fellows. In many clinical departments in medical schools, research activity is modest and may be partially subsidized by the margins from clinical revenues.

The focus on these two groups of sciences takes the perspective of the physician taking care of a single patient, i.e., applying the best biomedical science through the lens of a specific clinical specialty. However, despite its primacy, there is more to the practice of medicine than taking care of single patients. Physicians and other clinicians work in a healthcare system that has other concerns, such as continually increasing costs, worries about patient safety, and questions about the quality of care delivered. As such, 21st century clinicians must be competent in more than the diagnosis and treatment of disease in individual patients. This has led to emergence of the notion of a “third science” of medicine, which focuses on how to optimally provide healthcare for patients and populations. While some describe this as “healthcare delivery science” (my preference) or “implementation science,” the emerging name, as given to a textbook in this area, is now “health systems science.”

The textbook is published by the American Medical Association (AMA), which has been supporting innovation in medical education through its Accelerating Change in Education (ACE) consortium, funded by grants to medical schools [1]. OHSU was one of the original grantees in this program to establish “medical schools of the future.” I have been pleased that one outcome of this program has been the expansion of instruction in clinical informatics for medical students, which I consider to be an essential competency for 21st century physicians [2].


The titles of the chapters of the new textbook describe the important topics covered by health systems science:
  1. Health Systems Science in Medical Education
  2. What Is Health Systems Science? Building an Integrated Vision
  3. The Health Care Delivery System
  4. Value in Health Care
  5. Patient Safety
  6. Quality Improvement
  7. Principles of Teamwork and Team Science
  8. Leadership in Health Care
  9. Clinical Informatics
  10. Population Health
  11. Socio-Ecologic Determinants of Health
  12. Health Care Policy and Economics
  13. Application of Foundational Skills to Health Systems Science
  14. The Use of Assessment to Support Learning and Improvement in Health Systems Science
  15. The Future of Health Systems Science
I am delighted myself to be the lead author of one of the chapters, not surprisingly the one on clinical informatics [3]. I hope this chapter will introduce many new generations of medical and other health professions students to the informatics field and its role in healthcare delivery. Of course, informatics plays many roles beyond healthcare delivery, such as informing the care of individual patients and facilitating all types of research, but the effective use of data and informatics is a key aspect of health systems science.

I hope that this new textbook will lead the way in emphasizing the importance of health systems science in the work of physicians and other healthcare professionals. Clinicians have long known that diagnosing and treating disease, while the centerpiece of medical practice, cannot be carried out in a vacuum outside the realm of the patient’s and larger health system’s context. The care delivered to those individual patients will be better if the clinician has the perspective of that larger system.

References

1. Skochelak, SE, Hawkins, RE, et al., Eds. (2017). Health Systems Science. New York, NY, Elsevier.
2. Hersh, WR, Gorman, PN, et al. (2014). Beyond information retrieval and EHR use: competencies in clinical informatics for medical education. Advances in Medical Education and Practice. 5: 205-212. http://www.dovepress.com/beyond-information-retrieval-and-electronic-health-record-use-competen-peer-reviewed-article-AMEP.
3. Hersh, W and Ehrenfeld, J (2017). Clinical Informatics. in Health Systems Science. S. Skochelak, R. Hawkins, L. Lawson et al. New York, NY, Elsevier: 105-116.

Sunday, January 22, 2017

Response to Request for Information (RFI): Strategic Plan for the National Library of Medicine, National Institutes of Health

Under the leadership of its new Director, Patricia Brennan, PhD, RN, the National Library of Medicine (NLM) is undertaking a strategic planning process to develop goals and priorities for the NLM going forward. This process builds on a Request for Information (RFI) in 2015 from the NLM Working Group of the Advisory Committee to the National Institutes of Health (NIH) Director (ACD) to obtain input for a report on a vision for the future of NLM in the context of NLM’s leadership transition and emerging NIH data science priorities. The report was released in 2015. I posted to this blog both the comments that I submitted for the report as well as an overview of the report after it was published.

The new RFI asks for comments on four themes:
  1. Role of NLM in advancing data science, open science, and biomedical informatics
  2. Role of NLM in advancing biomedical discovery and translational science
  3. Role of NLM in supporting the public’s health: clinical systems, public health systems and services, and personal health
  4. Role of NLM in building collections to support discovery and health in the 21st century
For each theme, respondents are asked to:
  1. Identify what you consider an audacious goal in this area – a challenge that may be daunting but would represent a huge leap forward were it to be achieved
  2. The most important thing NLM does in this area, from your perspective
  3. Research areas that are most critical for NLM to conduct or support
  4. Other comments, suggestions, or considerations, keeping in mind that the aim is to build the NLM of the future
In the remainder of this post, I will provide the comments I submitted to the RFI. I chose to limit my comments to the first of the four themes because the role of NLM is to advance the other themes – discovery, translation, and the public’s health – in the context of the first theme – namely the field of biomedical informatics, and data/open science within it.

a. Identify what you consider an audacious goal in this area – a challenge that may be daunting but would represent a huge leap forward were it to be achieved. Include input on the barriers to and benefits of achieving the goal.

I have chosen to focus my comments on the first of the four themes because the role of NLM is to advance the other themes – discovery, translation, and the public’s health – by advancing the first theme – namely the field of biomedical informatics, and data/open science within it. Therefore, the most audacious goal for all of NLM is to build and sustain the infrastructure of biomedical informatics, i.e., the people, technology, and resources to advance discovery, translation, and the public’s health.

Biomedical informatics must leverage both achievements that are new, such as digital and networking technologies, as well as goals that are enduring, such as improving individual health, healthcare, public health, and research. The NLM must promote, educate about, and fund biomedical informatics and related disciplines to the appropriate level they deserve in relation to the larger biomedical research enterprise. While research in domain-specific areas (e.g., cancer, cardiovascular, mental health) is important, biomedical informatics can provide fundamental tools to advance science in all domain-specific areas. To achieve this, we still need basic research in biomedical informatics itself, improving our knowledge and tools in many areas, including but not limited to human-computer interaction, natural language understanding, standards and interoperability, data quality, the intersection of people and organizational issues with information technology, workflow analysis, etc.

b. The most important thing NLM does in this area, from your perspective.

Although there are many institutes within NIH (e.g., NCI, NHLBI, and the Fogarty International Center) and other entities outside of NIH (e.g., AHRQ and PCORI) that fund research in informatics-related areas, NLM is the only entity that funds basic research in biomedical informatics. Most of the other institutes and entities that fund informatics support projects that are highly applied and/or domain-focused. These projects are important, but basic informatics research is also key to improving discovery, translation, and the public’s health.

The NLM is also unique in developing emerging technologies, some of which we cannot foresee now. When I was an NLM informatics postdoctoral fellow in the late 1980s, I could not have imagined the emergence of the World Wide Web, the wireless ubiquitous Internet, modern mobile devices, or the widespread adoption of electronic health records that we now have. There are likely new technologies coming down the road that few if any of us can predict that will have major impacts on health and healthcare. It is critical that the NLM and the research it supports enable these technologies to be put to optimal usage.

c. Research areas that are most critical for NLM to conduct or support.

Although it is critical for NLM to support research in biomedical informatics as applied to all areas of individual and public health and of healthcare and research, it is nearly unique in funding basic research in clinical informatics. A good deal of informatics research in the other NIH institutes is focused in basic science, e.g., genomics, bioinformatics, and computational biology. AHRQ and PCORI support clinical informatics research, but it is highly applied. Only NLM funds critical basic research in clinical informatics, and this function is vitally important as we strive to use informatics to achieve the triple aim of better health, improved healthcare, and reduced costs.

d. Other comments, suggestions, or considerations, keeping in mind that the aim is to build the NLM of the future.

Another critical function of NLM that has provided value and should be further augmented is its training programs for those who aspire to careers in informatics research. I count myself among many whose NLM fellowship training led to a successful career as a researcher, educator, and academician generally. NLM training grants have also provided support for my university to educate the next generation of informatics researchers who have gone on to become successful researchers and other leaders in the field.

A final problem is that I would like to see addressed is the name itself, "National Library of Medicine." This name does not connote all of what NLM does. Yes the NLM is a world-renowned biomedical library, and that function is critically important to continue. But NLM also provides cutting-edge research and training in informatics, and an ideal change for NLM would be a name change to something like the "National Biomedical and Health Informatics Institute," of which a robust and innovative National Library of Medicine would be a vital part.

I look forward to seeing other input to the new NLM strategic planning process and the resulting strategic plan that will set priorities going forward for this great public resource that has benefitted patients, the healthcare system, and students, faculty, and others who have worked in biomedical informatics to advance human health.

Friday, January 20, 2017

What is the Value of Those Who Create and Disseminate Knowledge?

There is an old adage, “Those who can’t do, teach.” (And Woody Allen’s further, “Those who can’t teach, teach gym.”) My usual retort is a quote from Aristotle, "Those that know, do. Those that understand, teach.”

But we seem to be entering an era where an individual’s worth is related mostly to his or her wealth. In addition, there are plenty of people, many of same mind-set, who are highly critical of academia, in particular of people whose livelihood involves creating and/or disseminating knowledge.

I am not uncritical of some aspects of the academic world in which I work, but I am even more aghast toward those who believe it to be misguided or unnecessary.

In essence, my job involves the creation and dissemination of knowledge. This takes a certain skill set and collection of talents, just like any other knowledge-oriented job. I believe that this work is important to society and worthy of its investment, even though the lion’s share of the funding of my teaching work comes from learners who pay tuition.

My job is hardly stress-free. Academia is like most pursuits in life, where a certain amount of stress and competition is good, leading to productivity and innovation. And there are times when the stress and pursuit become counterproductive.

I owe a lot to subsidized public academia that has enabled my professional success in life. I attended public schools for my entire education, from kindergarten through medical school. When I started college at the University of Illinois in 1976, tuition was $293 per semester. Not per course or per credit, but for all of the courses I took that term. Even medical school, also at University of Illinois, was relatively inexpensive for me, with tuition around $3000 per year when I started in 1980. I am not against students have some “skin in the game” in higher education, but it must be within the means of anyone who wants to pursue it. By the same token, I believe that we in academia need to be accountable in providing a skill set that enables individuals to succeed in their chosen careers.

I am extremely gratified to have an academic job that I mostly enjoy going to each day. While most higher education faculty positions have a combination of research, teaching, and service, I have found my most passion in teaching. I particularly enjoy, and have received feedback from others, that I have a knack for taking bodies of knowledge and distilling out the big themes and most salient facts. I do also enjoy research and building on the synergy of the two that characterizes optimal higher education. I make a good salary as a department chair at a public medical school. I could certainly make more money in other pursuits, but I have had plenty to live comfortably, save for retirement, send my children to college, and handle unexpected expenses.

I don’t begrudge rich people their wealth, especially those who earned it from modest beginnings and/or by producing things that truly benefit society. But wealth is hardly the only measure of a person’s contributions and value to society, and there must always be a role for those who create and disseminate knowledge.

Thursday, January 12, 2017

What is the Right Approach to Sharing Clinical Research Data?

While many people and organizations have long called for data from randomized clinical trials (RCTs) and other clinical research to be shared with other researchers for re-analysis and other re-use, the impetus for it accelerated about a year ago with two publications. One was a call by the International Committee of Medical Journal Editors (ICMJE) for de-identified data from RCTs to be shared as condition of publication [1]. The other was the publication of an editorial in the New England Journal of Medicine wondering whether those who do secondary analysis of such data were “research parasites” [2]. The latter set off a fury of debate across the spectrum, e.g. [3], from those who argued that primary researchers labored hard to devise experiments and collect their data, thus having claim to control over it, to those who argued that since most research is government-funded, the taxpayers deserve to have access to that data. (Some of those in the latter group proudly adopted the “research parasite” tag.)

Many groups and initiatives have advocated for the potential value of wider re-use of data from clinical research. The cancer genomics community has long seen the value of a data commons to facilitate sharing among researchers [4]. Recent US federal research initiatives, such as the Precision Medicine Initiative [5] and the 21st Century Cures program [6] envision an important role for large repositories of data to accompany patients in cutting-edge research. There are a number of large-scale efforts in clinical data collection that are beginning to accumulate substantial amounts of data, such as the National Patient-Centered Clinical Research Network (PCORNet) and the Observational Health Data Sciences and Informatics (OHDSI) initiative.

As with many contentious debates, there are valid points on both sides. The case for requiring publication of data is strong. As most research is taxpayer-funded, it only seems fair that those who paid are entitled to all the data for which they paid. Likewise, all of the subjects were real people who potentially took risks to participate in the research, and their data should be used for discovery of knowledge to the fullest extent possible. And finally, new discoveries may emerge from re-analysis of data. This was actually the case that prompted the Longo “ esearch parasites” editorial, which was praising the “right way” to do secondary analysis, including working with the original researchers. The paper that the editorial described had discovered that the lack of expression of a gene (CDX2) was associated with benefit from adjuvant chemotherapy [7].

Some researchers, however, are pushing back. They argue that those who carry out the work of designing, implementing, and evaluating experiments certainly have some exclusive rights to the data generated by their work. Some also question whether the cost is a good expenditure of limited research dollars, especially since the demand for such data sets may be modest and the benefit is not clear. One group of 282 researchers in 33 countries, the International Consortium of Investigators for Fairness in Trial Data Sharing, notes that there are risks, such as misleading or inaccurate analyses as well as efforts aimed at discrediting or undermining the original research [8]. They also express concern about the costs, given that there are over 27,000 RCTs performed each year. As such, this group calls for an embargo on reuse of data for two years plus another half-year for each year of the length of the RCT. Even those who support data sharing point out the requirement for proper curation, wide availability to all researchers, and appropriate credit to and involvement of those who originally obtained the data [9].

There are a number of challenges to more widespread dissemination of RCT data for re-use. A number of pharmaceutical companies have begun making such data available over the last few years. Their experience has shown that the costs are not insignificant (estimated to be about $30,000-$50,000 per RCT) and a scientific review process is essential [10]. Another analysis found that the time to re-analyze data sets can be long, and so far the number of publications have been few [11]. An additional study found that identifiable data sets were only explicitly visible from 12% of all clinical research funded by the National Institutes of Health in 2011 [12]. This means that from 2011 alone, there are possibly more than 200,000 data sets that could be made publicly available, indicating some type of prioritization might be required.

There are also a number of informatics-related issues to be addressed. These not only include adherence to standards and interoperability [13], but also attention to workflows, integration with other data, such as that from electronic health records (EHRs), and consumer/patient engagement [14]. Clearly the trialists who generate the data must be given incentives for their data to be re-used [15]. My own work assessing the caveats of re-using EHR data is somewhat applicable here too, in that even RCT data may not have the breadth of data or cover sufficient periods of time for additional analyses [16].

There is definitely great potential for re-use of RCT and other clinical research data to advanced research and ultimately health and clinical care for the population. However, it must be done in ways that represent an appropriate use of resources and result in data that truly advances research, clinical care, and ultimately individual health.

References
1. Taichman, DB, Backus, J, et al. (2016). Sharing clinical trial data: a proposal from the International Committee of Medical Journal Editors. New England Journal of Medicine. 374: 384-386.
2. Longo, DL and Drazen, JM (2016). Data sharing. New England Journal of Medicine. 374: 276-277.
3. Berger, B, Gaasterland, T, et al. (2016). ISCB’s initial reaction to The New England Journal of Medicine Editorial on data sharing. PLoS Computational Biology. 12(3): e1004816. http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004816.
4. Grossman, RL, Heath, AP, et al. (2016). Toward a shared vision for cancer genomic data. New England Journal of Medicine. 379: 1109-1112.
5. Collins, FS and Varmus, H (2015). A new initiative on precision medicine. New England Journal of Medicine. 372: 793-795.
6. Kesselheim, AS and Avorn, J (2017). New "21st Century Cures" legislation: speed and ease vs science. Journal of the American Medical Association. Epub ahead of print.
7. Dalerba, P, Sahoo, D, et al. (2016). CDX2 as a prognostic biomarker in stage II and stage III colon cancer. New England Journal of Medicine. 374: 211-222.
8. Anonymous (2016). Toward fairness in data sharing. New England Journal of Medicine. 375: 405-407.
9. Merson, L, Gaye, O, et al. (2016). Avoiding data dumpsters — toward equitable and useful data sharing. New England Journal of Medicine. 374: 2414-2415.
10. Rockhold, F, Nisen, P, et al. (2016). Data sharing at a crossroads. New England Journal of Medicine. 375: 1115-1117.
11. Strom, BL, Buyse, ME, et al. (2016). Data sharing — is the juice worth the squeeze? New England Journal of Medicine. 375: 1608-1609.
12. Read, KB, Sheehan, JR, et al. (2015). Sizing the problem of improving discovery and access to NIH-funded data: a preliminary study. PLoS ONE. 10(7): e0132735. http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0132735.
13. Kush, R and Goldman, M (2016). Fostering responsible data sharing through standards. New England Journal of Medicine. 370: 2163-2165.
14. Tenenbaum, JD, Avillach, P, et al. (2016). An informatics research agenda to support precision medicine: seven key areas. Journal of the American Medical Informatics Association. 23: 791-795.
15. Lo, B and DeMets, DL (2016). Incentives for clinical trialists to share data. New England Journal of Medicine. 375: 1112-1115.
16. Hersh, WR, Weiner, MG, et al. (2013). Caveats for the use of operational electronic health record data in comparative effectiveness research. Medical Care. 51(Suppl 3): S30-S37.