{"id":2506,"date":"2014-12-15T08:47:23","date_gmt":"2014-12-14T21:47:23","guid":{"rendered":"http:\/\/blog.panicola.com\/?p=2506"},"modified":"2014-12-15T08:47:23","modified_gmt":"2014-12-14T21:47:23","slug":"amplio-surgeon-score-cards","status":"publish","type":"post","link":"https:\/\/blog.panicola.com\/?p=2506","title":{"rendered":"Amplio &#8211; surgeon score cards"},"content":{"rendered":"<p>https:\/\/medium.com\/backchannel\/should-surgeons-keep-score-8b3f890a7d4c<\/p>\n<div class=\"section-inner layoutSingleColumn\">\n<h2 class=\"graf--h2\" data-align=\"center\">Making the Cut<\/h2>\n<h4 class=\"graf--h4\" data-align=\"center\"><a id=\"1f21\"><\/a><em class=\"markup--em markup--h4-em\">Which surgeon you get matters\u200a\u2014\u200aa lot. But how do we know who the good ones are?<\/em><\/h4>\n<figure class=\"graf--figure\"><a id=\"8910\"><\/a><\/p>\n<div class=\"aspectRatioPlaceholder is-locked\">\n<div class=\"aspect-ratio-fill\"><\/div>\n<p><img decoding=\"async\" class=\"graf-image\" src=\"https:\/\/d262ilb51hltx0.cloudfront.net\/max\/1200\/1*yRvWHC88tupF7_FvGWSChg.png\" alt=\"\" data-image-id=\"1*yRvWHC88tupF7_FvGWSChg.png\" data-width=\"1200\" data-height=\"22\" data-action=\"zoom\" data-action-value=\"1*yRvWHC88tupF7_FvGWSChg.png\" \/><\/div>\n<\/figure>\n<blockquote class=\"graf--pullquote pullquote graf--startsWithDoubleQuote\"><p><a id=\"dced\"><\/a>\u201cYou can think of surgery as not really that different than golf.\u201d Peter Scardino is the chief of surgery at Memorial Sloan Kettering Cancer Center (MSK). He has performed more than 4,000 open radical prostatectomies. \u201cVery good athletes and intelligent people can be wildly different in their ability to drive or chip or putt. I think the same thing\u2019s true in the operating room.\u201d<\/p><\/blockquote>\n<p class=\"graf--p\"><a id=\"1d12\"><\/a>The difference is that golfers keep score. Andrew Vickers, a biostatistician at MSK, would hear cancer surgeons at the hospital having heated debates about, say, how often they took out a patient\u2019s whole kidney versus just a part of it. \u201cWait a minute,\u201d he remembers thinking. \u201cDon\u2019t you <em class=\"markup--em markup--p-em\">know <\/em>this?\u201d<\/p>\n<p class=\"graf--p graf--startsWithDoubleQuote\"><a id=\"dc4e\"><\/a>\u201cHow come they didn\u2019t know this already?\u201d<\/p>\n<p class=\"graf--p\"><a id=\"f81c\"><\/a>In the summer of 2009, he and Scardino teamed up to begin work on a software project, called Amplio (from the Latin for \u201cto improve\u201d), to give surgeons detailed feedback about their performance. The program\u2014still in its early stages but already starting to be shared with other hospitals\u200a\u2014\u200astarted with a simple premise: the only way a surgeon is going to get better is if he knows where he stands.<\/p>\n<p class=\"graf--p\"><a id=\"2cb9\"><\/a>Vickers likes to put it this way. His brother-in-law is a bond salesman, and you can ask him, How\u2019d you do last week?, and he\u2019ll tell you not just his own numbers, but the numbers for his whole group.<\/p>\n<p class=\"graf--p\"><a id=\"4f41\"><\/a>Why should it be any different when lives are in the balance?<\/p>\n<figure class=\"graf--figure\"><a id=\"dc75\"><\/a><\/p>\n<div class=\"aspectRatioPlaceholder is-locked\">\n<div class=\"aspect-ratio-fill\"><\/div>\n<p><img decoding=\"async\" class=\"graf-image\" src=\"https:\/\/d262ilb51hltx0.cloudfront.net\/max\/1200\/1*d97UzsZ2OyFVRGj2cQKyeQ.png\" alt=\"\" data-image-id=\"1*d97UzsZ2OyFVRGj2cQKyeQ.png\" data-width=\"1200\" data-height=\"22\" data-action=\"zoom\" data-action-value=\"1*d97UzsZ2OyFVRGj2cQKyeQ.png\" \/><\/div>\n<\/figure>\n<figure class=\"graf--figure postField--outsetLeftImage\"><a id=\"7986\"><\/a><\/p>\n<div class=\"aspectRatioPlaceholder is-locked\">\n<div class=\"aspect-ratio-fill\"><\/div>\n<p><img decoding=\"async\" class=\"graf-image\" src=\"https:\/\/d262ilb51hltx0.cloudfront.net\/max\/1200\/1*wMWz0sG3HFsXpPDDk95Fkw.png\" alt=\"\" data-image-id=\"1*wMWz0sG3HFsXpPDDk95Fkw.png\" data-width=\"1200\" data-height=\"1026\" data-action=\"zoom\" data-action-value=\"1*wMWz0sG3HFsXpPDDk95Fkw.png\" \/><\/div><figcaption class=\"imageCaption\">Andrew Vickers<\/figcaption><\/figure>\n<p class=\"graf--p\"><a id=\"6dee\"><\/a>The central technique of <a class=\"markup--anchor markup--p-anchor\" href=\"http:\/\/www.mskcc.org\/amplio-system\" target=\"_blank\" rel=\"nofollow\" data-href=\"http:\/\/www.mskcc.org\/amplio-system\">Amplio<\/a>, using outcome data to determine which surgeons were more successful, and why, takes on a powerful taboo. Perhaps the longest-standing impediment to research into surgical outcomes\u200a\u2014\u200athe reason that surgeons, unlike bond salesmen (or pilots or athletes), are so much in the dark about their own performance\u200a\u2014\u200aare the surgeons themselves.<\/p>\n<p class=\"graf--p graf--startsWithDoubleQuote\"><a id=\"3c68\"><\/a>\u201cSurgeons basically deeply believe that if I\u2019m a well-trained surgeon, if I\u2019ve gone through a good residency program, a fellowship program, and I\u2019m board-certified, I can do an operation just as well as you can,\u201d Scardino says. \u201cAnd the difference between our results is really because I\u2019m willing to take on the challenging patients.\u201d<\/p>\n<p class=\"graf--p\"><a id=\"f10b\"><\/a>It is, maybe, a vestige of the old myth that anyone ordained to cut into healthy flesh is thereby made a minor god. It\u2019s the belief that there are no differences in skill, and that even if there <em class=\"markup--em markup--p-em\">were<\/em> differences, surgery is so complicated and multifaceted, and so much determined by the patient you happen to be operating on, that no one would ever be able to tell.<\/p>\n<p class=\"graf--p\"><a id=\"14d8\"><\/a>Vickers said to me that after several years of hearing this, he became so frustrated that he sat down with his ten\u00ad-year-\u00adold daughter and conducted a little experiment. He searched YouTube for \u201cradical prostatectomy\u201d and found two clips, one from a highly respected surgeon and one from a surgeon who was rumored to be less skilled. He showed his daughter a 15\u00adsecond clip of each and asked, \u201cWhich one is better?\u201d<\/p>\n<p class=\"graf--p graf--startsWithDoubleQuote\"><a id=\"3f9f\"><\/a>\u201cThat one,\u201d she replied right away.<\/p>\n<p class=\"graf--p\"><a id=\"4be7\"><\/a>When Vickers asked her why, \u201cShe looked at me, like, <em class=\"markup--em markup--p-em\">can\u2019t you tell the difference?<\/em> You can just see.\u201d<\/p>\n<figure class=\"graf--figure graf--iframe\"><a id=\"1206\"><\/a><\/p>\n<div class=\"iframeContainer\"><iframe loading=\"lazy\" src=\"https:\/\/medium.com\/media\/5376a71d620c8cce024660e424d2aa7b?maxWidth=640\" width=\"640\" height=\"480\" frameborder=\"0\" data-width=\"640\" data-height=\"480\" data-media-id=\"5376a71d620c8cce024660e424d2aa7b\"><\/iframe><\/div><figcaption class=\"imageCaption\">Would you want to be cut by this surgeon?<\/figcaption><\/figure>\n<figure class=\"graf--figure graf--iframe\"><a id=\"80a8\"><\/a><\/p>\n<div class=\"iframeContainer\"><iframe loading=\"lazy\" src=\"https:\/\/medium.com\/media\/5e5dfc4a2ef7fae2be257446dfe87c44?maxWidth=640\" width=\"640\" height=\"480\" frameborder=\"0\" data-width=\"640\" data-height=\"480\" data-media-id=\"5e5dfc4a2ef7fae2be257446dfe87c44\"><\/iframe><\/div><figcaption class=\"imageCaption\">Or this one?<\/figcaption><\/figure>\n<p class=\"graf--p\"><a id=\"634a\"><\/a><a class=\"markup--anchor markup--p-anchor\" href=\"http:\/\/www.nejm.org\/doi\/full\/10.1056\/NEJMsa1300625\" target=\"_blank\" rel=\"nofollow\" data-href=\"http:\/\/www.nejm.org\/doi\/full\/10.1056\/NEJMsa1300625\">A remarkable paper<\/a> published last year in the <em class=\"markup--em markup--p-em\">New England Journal of Medicine<\/em> showed that maybe Vickers\u2019s daughter was onto something.<\/p>\n<p class=\"graf--p\"><a id=\"8a4c\"><\/a>In the study, run by John Birkmeyer, a surgeon who at the time was at the University of Michigan, bariatric surgeons were recruited from around the state of Michigan to submit videos of themselves doing a gastric bypass operation. The videos were sent to another pool of bariatric surgeons to be given a series of 1-to-5 rating on factors such as \u201crespect for tissue,\u201d \u201ctime and motion,\u201d \u201ceconomy of movement\u201d and \u201cflow of operation.\u201d<\/p>\n<p class=\"graf--p\"><a id=\"8bbb\"><\/a>The study\u2019s key finding was that not only could you reliably determine a surgeon\u2019s skill by watching them on video\u200a\u2014\u200askill was nowhere near as nebulous as had been assumed\u200a\u2014\u200abut that those ratings were highly correlated with outcomes: \u201cAs compared with patients treated by surgeons with high skill ratings, patients treated by surgeons with low skill ratings were at least twice as likely to die, have complications, undergo reoperation, and be readmitted after hospital discharge,\u201d Birkmeyer and his colleagues wrote in the paper.<\/p>\n<p class=\"graf--p\"><a id=\"f56a\"><\/a>You can actually watch a couple of these videos yourself [<em class=\"markup--em markup--p-em\">see above<\/em>]. Along with the overall study results, Birkmeyer published two short clips: one from a highly rated surgeon and one from a low-rated surgeon. The difference is astonishing.<\/p>\n<p class=\"graf--p\"><a id=\"8a1f\"><\/a>You see the higher-rated surgeon first. It\u2019s what you always imagined surgery might look like. The robot hands move with purpose\u200a\u2014\u200aquick, deliberate strokes. There\u2019s no wasted motion. When they grip or sew or staple tissue, it\u2019s with a mix of command and gentle respect. The surgeon seems to know exactly what to do next. The way they\u2019ve set things up makes it feel roomy in there, and tidy.<\/p>\n<p class=\"graf--p\"><a id=\"332c\"><\/a>Watching the lower-rated surgeon, by contrast, is like watching the hidden camera footage of a nanny hitting your kid: it looks like abuse. The surgeon\u2019s view is all muddled, they\u2019re groping aimlessly at flesh, desperate to find purchase somewhere, or an orientation, as if their instruments are being thrashed around in the undertow of the patient\u2019s guts. It\u2019s like watching middle schoolers play soccer: the game seems to make no sense, to have no plot or direction or purpose or boundary. It\u2019s not, in other words, like, \u201cThis one\u2019s hands are a bit shaky,\u201d it\u2019s more like, \u201cDoes this one have any clue what they\u2019re doing?\u201d<\/p>\n<p class=\"graf--p\"><a id=\"2714\"><\/a>It\u2019s funny: in other disciplines we reserve the word \u201csurgical\u201d for feats that took a special poise, a kind of deftness under pressure. But the thing we maybe forget is that not all surgery is worthy of the name.<\/p>\n<p class=\"graf--p\"><a id=\"483a\"><\/a>Vickers is best known for showing exactly how much variety there is, plotting, in 2007, the so-called \u201clearning curve\u201d for surgery: a graph that tracks, on one axis, the number of cases a surgeon has under his belt, and on the other, his recurrence rates (the rate at which his patients\u2019 cancer comes back).<\/p>\n<figure class=\"graf--figure\"><a id=\"6116\"><\/a><\/p>\n<div class=\"aspectRatioPlaceholder is-locked\">\n<div class=\"aspect-ratio-fill\"><\/div>\n<p><img decoding=\"async\" class=\"graf-image\" src=\"https:\/\/d262ilb51hltx0.cloudfront.net\/max\/899\/1*rYQ3mvv3B3N45vRXOQzXSQ.png\" alt=\"\" data-image-id=\"1*rYQ3mvv3B3N45vRXOQzXSQ.png\" data-width=\"899\" data-height=\"719\" data-action=\"zoom\" data-action-value=\"1*rYQ3mvv3B3N45vRXOQzXSQ.png\" \/><\/div><figcaption class=\"imageCaption\">As surgeons get more experience, their patients do better. This \u201clearning curve\u201d shows patients\u2019 5 year cancer-free rates rise with procedure volume.<\/figcaption><\/figure>\n<p class=\"graf--p\"><a id=\"ebe7\"><\/a>He showed that in incidents of prostate cancer that haven\u2019t spread beyond the prostate\u200a\u2014\u200aso-called \u2018organ-confined\u2019 cases\u200a\u2014\u200athe recurrence rates for a novice surgeon were 10 to 15%. For an experienced surgeon, they were less than 1%. With recurrence rates so low for the most experienced surgeons, Vickers was able to conclude that in organ-confined cancer cases, the <em class=\"markup--em markup--p-em\">only<\/em>reason a patient would recur is \u201cbecause the surgeon screwed up.\u201d<\/p>\n<p class=\"graf--p\"><a id=\"edc1\"><\/a>There\u2019s a large literature, going back to a famous paper in 1979, finding that hospitals with higher volumes of a given surgical procedure have better outcomes. In the \u201979 study it was reported that for some kinds of surgery, hospitals that saw 200 or more cases per year had death rates that were 25% to 41% lower than hospitals with lower volumes. If every case were treated at a high-volume hospital, you would avoid more than a third of the deaths associated with the procedure.<\/p>\n<p class=\"graf--p\"><a id=\"e466\"><\/a>But what wasn\u2019t clear was <em class=\"markup--em markup--p-em\">why<\/em> higher volumes led to better outcomes. And for decades, researchers penned more than 300 studies restating the same basic relationship, without getting any closer to explaining it. Did low-volume hospitals end up with the riskiest patients? Did high-volume hospitals have fancier equipment? Or better operating room teams? A better overall staff? An editorial as late as 2003 summarized the literature with the title, \u201cThe Volume\u2013Outcome Conundrum.\u201d<\/p>\n<p class=\"graf--p\"><a id=\"5e51\"><\/a>A 2003 paper by Birkmeyer, <a class=\"markup--anchor markup--p-anchor\" href=\"http:\/\/www.nejm.org\/doi\/full\/10.1056\/NEJMsa035205\" target=\"_blank\" rel=\"nofollow\" data-href=\"http:\/\/www.nejm.org\/doi\/full\/10.1056\/NEJMsa035205\">\u201cSurgeon volume and operative mortality in the United States,\u201d<\/a> was the first to offer definitive evidence that the biggest factor determining the outcome of many surgical procedures\u200a\u2014\u200athe hidden element that explained most of the variation among hospitals\u200a\u2014\u200awas the procedure volume not of the hospital, but of the individual surgeons.<\/p>\n<p class=\"graf--p graf--startsWithDoubleQuote\"><a id=\"7360\"><\/a>\u201cIn general I don\u2019t think anyone was surprised that there was a learning curve,\u201d Vickers says. \u201cI think they were surprised at what a big difference it made.\u201d Surprised, maybe, but not moved to action. \u201cYou may think that everyone would drop what they were doing,\u201d he says, \u201cand try and work out what it is that some surgeons are doing that the other ones aren\u2019t\u2026 But things move a lot more slowly than that.\u201d<\/p>\n<p class=\"graf--p\"><a id=\"e51e\"><\/a>Tired of waiting, Vickers started sharing some initial ideas with Scardino about the program that would become Amplio. It would give surgeons detailed feedback about their performance. It would show you not just your own results, but the results for everyone in your service. If another surgeon was doing particularly well, you could find out what accounted for the difference; if your own numbers dropped, you\u2019d know to make an adjustment. Vickers explains that they wanted to \u201cstop doing studies showing surgeons had different outcomes.\u201d<\/p>\n<p class=\"graf--p graf--startsWithDoubleQuote\"><a id=\"2b31\"><\/a>\u201cLet\u2019s do something about it,\u201d he told Scardino.<\/p>\n<figure class=\"graf--figure\"><a id=\"7841\"><\/a><\/p>\n<div class=\"aspectRatioPlaceholder is-locked\">\n<div class=\"aspect-ratio-fill\"><\/div>\n<p><img decoding=\"async\" class=\"graf-image\" src=\"https:\/\/d262ilb51hltx0.cloudfront.net\/max\/1200\/1*d97UzsZ2OyFVRGj2cQKyeQ.png\" alt=\"\" data-image-id=\"1*d97UzsZ2OyFVRGj2cQKyeQ.png\" data-width=\"1200\" data-height=\"22\" data-action=\"zoom\" data-action-value=\"1*d97UzsZ2OyFVRGj2cQKyeQ.png\" \/><\/div>\n<\/figure>\n<figure class=\"graf--figure postField--outsetLeftImage\"><a id=\"38bd\"><\/a><\/p>\n<div class=\"aspectRatioPlaceholder is-locked\">\n<div class=\"aspect-ratio-fill\"><\/div>\n<p><img decoding=\"async\" class=\"graf-image\" src=\"https:\/\/d262ilb51hltx0.cloudfront.net\/max\/1200\/1*2A4VPpQlSn3GXPSrxiehFg.jpeg\" alt=\"\" data-image-id=\"1*2A4VPpQlSn3GXPSrxiehFg.jpeg\" data-width=\"1200\" data-height=\"1026\" data-action=\"zoom\" data-action-value=\"1*2A4VPpQlSn3GXPSrxiehFg.jpeg\" \/><\/div><figcaption class=\"imageCaption\">Dr. Scardino<\/figcaption><\/figure>\n<p class=\"graf--p\"><a id=\"6e71\"><\/a>The first time I heard about Amplio was on the third floor of the Chrysler Building, in a room they called the Innovation Lab\u200a\u2014\u200athe very room you\u2019d point to if the Martians ever asked you what a 125-year old bureaucracy looks like. As I arrived, the receptionist was trying to straighten up a small mess of papers, post-its, cookies, and coffee stirrers. \u201cThe last crowd had a wild time,\u201d she said. Every surface in the room was gray or off-white, the color of questionable eggs. It smelled like hospital-grade hand soap.<\/p>\n<p class=\"graf--p\"><a id=\"ce65\"><\/a>The people who filed in, though, and introduced themselves to each other (this was a summit of sorts, a \u201cCollaboration Meeting\u201d where different research groups from around MSK shared their works in progress) looked straight out of a well-funded biotech startup. There was a Fulbright scholar; a double-major in biology and philosophy; a couple of epidemiologists; a mathematician; a master\u2019s in biostats and predictive analytics. There were Harvards, Cals, and Columbias, bright-eyed and sharply dressed.<\/p>\n<p class=\"graf--p\"><a id=\"8012\"><\/a>Vickers was one of the speakers. He\u2019s in his forties but he looks younger, less like an academic than a seasoned ski instructor, a consequence, maybe, of the long wavy hair, or the well-worn smile lines around his eyes, or this expression he has that\u2019s like a mix of relaxed and impish. He leans back when he talks, and he talks well, and you get the sense that he <em class=\"markup--em markup--p-em\">knows<\/em> he talks well. He\u2019s British, from north London, educated first at Cambridge and then, for his PhD in clinical medicine, at Oxford.<\/p>\n<p class=\"graf--p\"><a id=\"be5e\"><\/a>The first big task with Amplio, he said, was to get the data. In order for surgeons to improve, they have to know how well they\u2019re doing. In order to know how well they\u2019re doing, they have to know how well their <em class=\"markup--em markup--p-em\">patients<\/em> are doing. And this turns out to be trickier than you\u2019d think. You need an apparatus that not only keeps meticulous records, but keeps them consistently, and throughout the entire life cycle of the patient.<\/p>\n<p class=\"graf--p\"><a id=\"5753\"><\/a>That is, you need data on the patient <em class=\"markup--em markup--p-em\">before<\/em> the operation: How old are they? What medications are they allergic to? Have they been in surgery before? You need data on what happened <em class=\"markup--em markup--p-em\">during<\/em> the operation: where\u2019d you make your incisions? how much blood was lost? how long did it take?<\/p>\n<p class=\"graf--p\"><a id=\"5087\"><\/a>And finally, you need data on what happened to the patient <em class=\"markup--em markup--p-em\">after<\/em> the operation\u200a\u2014\u200ain some cases years after. In many hospitals, followup is sporadic at best. So before the Amplio team did anything fancy, they had to devise a better way to collect data from patients. They had to do stuff like find out whether it was better to give the patient a survey before or after a consultation with their surgeon? And what kinds of questions worked best? And who were they supposed to hand the iPad to when they were done?<\/p>\n<p class=\"graf--p\"><a id=\"0c80\"><\/a>Only when all these questions were answered, and a stream of regular data was being saved for every procedure, could Amplio start presenting something for surgeons to use.<\/p>\n<figure class=\"graf--figure\"><a id=\"5c62\"><\/a><\/p>\n<div class=\"aspectRatioPlaceholder is-locked\">\n<div class=\"aspect-ratio-fill\"><\/div>\n<p><img decoding=\"async\" class=\"graf-image\" src=\"https:\/\/d262ilb51hltx0.cloudfront.net\/max\/900\/1*U_AiCNkqyxrBMSgLd55UTw.jpeg\" alt=\"\" data-image-id=\"1*U_AiCNkqyxrBMSgLd55UTw.jpeg\" data-width=\"900\" data-height=\"642\" data-action=\"zoom\" data-action-value=\"1*U_AiCNkqyxrBMSgLd55UTw.jpeg\" \/><\/div><figcaption class=\"imageCaption\">A screen in Amplio shows how a surgeon\u2019s patients are doing against their colleagues\u2019<\/figcaption><\/figure>\n<p class=\"graf--p\"><a id=\"817e\"><\/a>After years of setup, Amplio now is in a state where it can begin to affect procedures. The way it works is that a surgeon logs into a screen that shows where they stand on a series of plots. On each plot there\u2019s a single red dot sitting amid some blue dots. The red dot shows your outcomes; the blue dots show the outcomes for each of the other surgeons in your group.<\/p>\n<p class=\"graf--p\"><a id=\"9a23\"><\/a>You can slice and dice different things you\u2019re interested in to make different kinds of plots. One plot might show the average amount of blood lost during the operation against the average length of the hospital stay after it. Another plot might show a prostate patient\u2019s recurrence rates against his continence or erectile function.<\/p>\n<p class=\"graf--p\"><a id=\"f03e\"><\/a>There\u2019s something powerful about having outcomes graphed so starkly. Vickers says that there was a surgeon who saw that they were so far into the wrong corner of that plot\u200a\u2014\u200apatients weren\u2019t recovering well, and the cancer was coming back\u200a\u2014\u200athat they decided to stop doing the procedure. The men spared poor outcomes by this decision will never know that Amplio saved them.<\/p>\n<p class=\"graf--p\"><a id=\"ef66\"><\/a>It\u2019s like an analytics dashboard, or a leaderboard, or a report card, or\u2026 well, it\u2019s like a lot of things that have existed in a lot of other fields for a long time. And it kind of makes you wonder, why has it taken so long for a tool like this to come to surgeons?<\/p>\n<p class=\"graf--p\"><a id=\"1829\"><\/a>The answer is that Amplio has cleverly avoided the pitfalls of some previous efforts. For instance, in 1989, New York state began publicly reporting the mortality rates of cardiovascular surgeons. Because the data was \u201crisk-adjusted\u201d\u2014an unfavorable outcome would be considered less bad, or not counted at all, if the patient was at risk to begin with\u200a\u2014\u200asurgeons started pretending their patients were a lot worse off than they were. In some cases, they avoided patients who looked like goners. \u201cThe sickest patients weren\u2019t being treated,\u201d Vickers says. One investigation into why mortality in New York had dropped for a certain procedure, the coronary artery bypass graft, concluded that it was just because New York hospitals were sending the highest-risk patients to Ohio.<\/p>\n<p class=\"graf--p\"><a id=\"25c0\"><\/a>Vickers wanted to resist such gaming. But the answer is not to quit adjusting for patient risk. After all, if a given report says that your patients have 60% fewer complications than mine, does that mean that you\u2019re a 60% better surgeon? It depends on the patients we see. It turns out that maybe the best way to prevent gaming is just to keep the results confidential. That sounds counter to a patient\u2019s interests, but it\u2019s been shown that patients actually make little use of objective outcomes data when it\u2019s available, that in fact they\u2019re much more likely to choose a surgeon or hospital based on reputation or raw proximity.<\/p>\n<\/div>\n<div class=\"section-inner u-fillWidth\">\n<figure class=\"graf--figure postField--fillWidthImage\"><a id=\"5ec0\"><\/a><\/p>\n<div class=\"aspectRatioPlaceholder is-locked\">\n<div class=\"aspect-ratio-fill\"><\/div>\n<p><img decoding=\"async\" class=\"graf-image\" src=\"https:\/\/d262ilb51hltx0.cloudfront.net\/max\/2000\/1*6tOey6wC_ERNsxMxFbaXjg.jpeg\" alt=\"\" data-image-id=\"1*6tOey6wC_ERNsxMxFbaXjg.jpeg\" data-width=\"1500\" data-height=\"1000\" \/><\/div>\n<\/figure>\n<\/div>\n<div class=\"section-inner layoutSingleColumn\">\n<p class=\"graf--p\"><a id=\"583b\"><\/a>With Amplio, since patients, and the hospital, and even your boss are blinded from knowing whose results belong to whom, there\u2019s no incentive to fudge risk factors or insist that a risk factor\u2019s weight be changed, unless you think it\u2019s actually good for the analysis.<\/p>\n<p class=\"graf--p\"><a id=\"e637\"><\/a>That\u2019s why Amplio\u2019s interface for slicing and dicing the data in multiple ways matters, too. Feedback systems in the past that have given surgeons a single-dimensional report\u200a\u2014\u200asay, they only track recurrence rates\u200a\u2014\u200ahave failed by creating a perverse incentive to optimize along just that one dimension, at the expense of all the others. Another reminder that feedback is, like surgery itself, fraught with complication: if you do it wrong, it can be worse than useless.<\/p>\n<p class=\"graf--p\"><a id=\"dbd1\"><\/a>Every member of the Amplio team I spoke to stressed this point over and over again, that the system had been painstakingly built from the \u201cbottom up\u201d\u200a\u2014\u200atuned via detailed conversations with surgeons (\u201cAre you accounting for BMI? What if we change the definition of blood loss?\u201d)\u200a\u2014\u200aso that the numbers it reported would be accurate, and risk-adjusted, and multidimensional, and credible. Because only then would they be actionable.<\/p>\n<p class=\"graf--p\"><a id=\"7a50\"><\/a>Karim Touijer, a surgeon at MSK who has used Amplio, explains the system\u2019s chief benefit is the fact that you can vividly see how you\u2019re doing, and that someone else is doing better. \u201cWhen you set a standard,\u201d he says, \u201cthe majority of people will improve or meet that standard. You tend to shrink the outliers. If I\u2019m an outlier, if my performance leaves something to be desired, then I can go to my colleagues and say what is it that you\u2019re doing to get these results?\u201d Touijer sees this as the gradual standardization of surgery: you find the best performers, figure out what makes them good, and spread the word. He said that already within his group, because the conversations are more tied to outcomes, they\u2019re talking about technique in a more objective way.<\/p>\n<p class=\"graf--p\"><a id=\"6b4a\"><\/a>In fact, he says, as a result of Amplio he and his team have devised the first randomized clinical trial that is solely dedicated to surgical maneuvers.<\/p>\n<p class=\"graf--p\"><a id=\"4ab6\"><\/a>Touijer specializes in the radical prostatectomy, considered one of the most complex and delicate operations in all of surgical practice. The procedure\u200a\u2014\u200ain which a patient\u2019s cancerous prostate is entirely removed\u200a\u2014\u200ais highly sensitive to an individual surgeon\u2019s skill. The reason is that the cancer ends up being very close to the nerves that control sexual and urinary function. It\u2019s an operation unlike, say, kidney cancer, where you can easily go widely around the cancer. If you operate too far around the prostate, you could easily damage the rectum, the bladder, the nerves responsible for erection, or the sphincter responsible for urinary control. \u201cIt turns out that radical prostatectomy is very, very intimately influenced by surgical technique,\u201d Touijer says. \u201cOne millimeter on one side or less than a millimeter on the other can change the outcome.\u201d<\/p>\n<figure class=\"graf--figure postField--outsetLeftImage\"><a id=\"0b0d\"><\/a><\/p>\n<div class=\"aspectRatioPlaceholder is-locked\">\n<div class=\"aspect-ratio-fill\"><\/div>\n<p><img decoding=\"async\" class=\"graf-image\" src=\"https:\/\/d262ilb51hltx0.cloudfront.net\/max\/560\/1*xSmzfBH3k9dlJOwllLhnIw.png\" alt=\"\" data-image-id=\"1*xSmzfBH3k9dlJOwllLhnIw.png\" data-width=\"560\" data-height=\"734\" data-action=\"zoom\" data-action-value=\"1*xSmzfBH3k9dlJOwllLhnIw.png\" \/><\/div><figcaption class=\"imageCaption\">Option B in the first A\/B test for surgery: \u201cA second bite is taken deeply into the fascia of the lateral pelvic fascia\u201d<\/figcaption><\/figure>\n<p class=\"graf--p\"><a id=\"79b2\"><\/a>There\u2019s a moment during the procedure where the surgeon has to decide whether to make a particular stitch. Some surgeons do it, some don\u2019t; we don\u2019t yet know which way is better. In the randomized trial, if the surgeon doesn\u2019t have a compelling reason to pick one of the two alternatives, he lets the computer decide randomly for him. With enough patients, it should be possible to isolate the effect of that one decision, and to find out whether the extra stitch leads to better outcomes. The beauty is, since the outcomes data was already being tracked, and the patients were already going to have the surgery, the trial costs almost nothing.<\/p>\n<p class=\"graf--p\"><a id=\"f0d0\"><\/a>If you\u2019ve worked on the web, this model of rapid, cheap experimentation probably sounds familiar: what Touijer is describing is the first A\/B test for surgery. As it turns out this particular test didn\u2019t yield significant results. But several other tests are in the works, and some may improve some specific surgical techniques\u2014improving the odds for all patients.<\/p>\n<p class=\"graf--p\"><a id=\"48c8\"><\/a>In <em class=\"markup--em markup--p-em\">Better<\/em>, Atul Gawande argues that when we think of improving medicine, we always imagine making new advances, discovering the gene responsible for a disease, and so on\u200a\u2014\u200aand forget that we can simply take what we already know how to do, and figure out how to do it better. In a word, iterate.<\/p>\n<p class=\"graf--p graf--startsWithDoubleQuote\"><a id=\"4eed\"><\/a>\u201cBut to do that,\u201d Scardino says, \u201cwe have to measure it, we have to <em class=\"markup--em markup--p-em\">know<\/em> what the results are.\u201d<\/p>\n<p class=\"graf--p\"><a id=\"b718\"><\/a>Scardino describes how when laparascopy was first becoming an option for radical prostatectomy, there was a lot of hype. \u201cThe company and many doctors who were doing it immediately claimed that it was safer, had better results, was more likely to cure the cancer and less likely to have permanent urinary or sexual problems.\u201d But, he says, the data to support it were weak, and biased. \u201cWe could see in Amplio early on that as people started doing robotic surgery, the results were clearly worse.\u201d It took time for them to hit par with the traditional open procedure; it took time for them to get better.<\/p>\n<figure class=\"graf--figure\"><a id=\"d7bb\"><\/a><\/p>\n<div class=\"aspectRatioPlaceholder is-locked\">\n<div class=\"aspect-ratio-fill\"><\/div>\n<p><img decoding=\"async\" class=\"graf-image\" src=\"https:\/\/d262ilb51hltx0.cloudfront.net\/max\/1200\/1*d97UzsZ2OyFVRGj2cQKyeQ.png\" alt=\"\" data-image-id=\"1*d97UzsZ2OyFVRGj2cQKyeQ.png\" data-width=\"1200\" data-height=\"22\" data-action=\"zoom\" data-action-value=\"1*d97UzsZ2OyFVRGj2cQKyeQ.png\" \/><\/div>\n<\/figure>\n<p class=\"graf--p\"><a id=\"a6cd\"><\/a>After a pilot among prostate surgeons, Amplio spread quickly to other services within MSK, including for kidney cancer, bladder cancer and colorectal cancer. Vickers\u2019s team has been working with other hospitals\u200a\u2014\u200aincluding Columbia in New York, the Barbara Ann Karmanos Cancer Institute in Michigan, and the MD Anderson Cancer Center in Texas\u200a\u2014\u200ato slowly begin integrating with their systems. But it\u2019s still early days: even within their own hospital, surgeons were wary of Amplio. It took many conversations, and assurances, to convince them that the data were being collected for their benefit\u200a\u2014\u200anot to \u201cname and shame\u201d bad performers.<\/p>\n<p class=\"graf--p\"><a id=\"e9a9\"><\/a>We know what happens when performance feedback goes awry\u200a\u2014\u200asimilar efforts to \u201cgrade\u201d American schoolteachers, for instance, have perhaps generated more controversy than results. To do performance feedback well requires patience, and tact, and an earnest imperative to improve everyone\u2019s results, not just to find the negative outliers. But Vickers believes that enough surgeons have signed on that the taboo has been broken at MSK. And results are bound to flow from that.<\/p>\n<p class=\"graf--p\"><a id=\"e9b7\"><\/a>It\u2019s all about trust. Remember the Birkmeyer study that compared surgeons using videos? It was only possible because Birkmeyer had built up relationships by way of a previous outcomes experiment in Michigan that meticulously protected data. \u201cThat\u2019s a question that we get really frequently,\u201d Birkmeyer told me when we spoke about the paper. \u201cHow on earth did we ever pull that study off?\u201d The key, he says, is that years of research with these surgeons had slowly built goodwill. When it came time to make a big ask, \u201cthe surgeons were at a place where they could trust that we weren\u2019t gonna screw them.\u201d<\/p>\n<p class=\"graf--p\"><a id=\"f86b\"><\/a>Amplio will no doubt have to be able to say the same thing, if it\u2019s to spread beyond the country\u2019s best research cancer centers into the average regional hospital.<\/p>\n<figure class=\"graf--figure\"><a id=\"c3aa\"><\/a><\/p>\n<div class=\"aspectRatioPlaceholder is-locked\">\n<div class=\"aspect-ratio-fill\"><\/div>\n<p><img decoding=\"async\" class=\"graf-image\" src=\"https:\/\/d262ilb51hltx0.cloudfront.net\/max\/1200\/1*92DnbKLxYxeJUV5TZf38UQ.png\" alt=\"\" data-image-id=\"1*92DnbKLxYxeJUV5TZf38UQ.png\" data-width=\"1200\" data-height=\"22\" data-action=\"zoom\" data-action-value=\"1*92DnbKLxYxeJUV5TZf38UQ.png\" \/><\/div>\n<\/figure>\n<p class=\"graf--p\"><a id=\"3792\"><\/a>In 1914, a surgeon at Mass General got so fed up with the administration, and their refusal to measure outcomes, that he created his own private hospital, \u201cthe End Result Hospital,\u201d where detailed records were to be kept of every patient\u2019s \u201cend results.\u201d He published the first five years of his hospital\u2019s cases in a book that became one of the founding documents of evidence-based medicine.<\/p>\n<p class=\"graf--p graf--startsWithDoubleQuote\"><a id=\"9c95\"><\/a>\u201cThe Idea is so simple as to seem childlike,\u201d he wrote, \u201cbut we find it ignored in all Charitable Hospitals, and very largely in Private Hospitals. It is simply to follow the natural series of questions which any one asks in an individual case: What was the matter? Did they find it out beforehand? Did the patient get entirely well? If not\u200a\u2014\u200awhy not? Was it the fault of the surgeon, the disease, or the patient? What can we do to prevent similar failures in the future?\u201d<\/p>\n<p class=\"graf--p\"><a id=\"a83f\"><\/a>It might finally be time for that simple, \u201cchildlike\u201d concept to reach fruition. It\u2019s like Vickers said to me one night in early November, as we were discussing Amplio, \u201cHaving been in health research for twenty years, there\u2019s always that great quote of Martin Luther King: The arc of history is long, but it bends towards justice.\u201d<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>https:\/\/medium.com\/backchannel\/should-surgeons-keep-score-8b3f890a7d4c Making the Cut Which surgeon you get matters\u200a\u2014\u200aa lot. But how do we know who the good ones are? \u201cYou can think of surgery as not really that different than golf.\u201d Peter Scardino is the chief of surgery at Memorial Sloan Kettering Cancer Center (MSK). He has performed more than 4,000 open radical prostatectomies. &hellip; <a href=\"https:\/\/blog.panicola.com\/?p=2506\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Amplio &#8211; surgeon score cards<\/span> <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5,35,8,26,33,9,3],"tags":[],"class_list":["post-2506","post","type-post","status-publish","format-standard","hentry","category-data-saving-lives","category-ef","category-entrepreneurship","category-facts-data-points","category-health-market-quality","category-healthcare","category-rapid-learning-health-systems"],"_links":{"self":[{"href":"https:\/\/blog.panicola.com\/index.php?rest_route=\/wp\/v2\/posts\/2506","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.panicola.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.panicola.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.panicola.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.panicola.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2506"}],"version-history":[{"count":1,"href":"https:\/\/blog.panicola.com\/index.php?rest_route=\/wp\/v2\/posts\/2506\/revisions"}],"predecessor-version":[{"id":2507,"href":"https:\/\/blog.panicola.com\/index.php?rest_route=\/wp\/v2\/posts\/2506\/revisions\/2507"}],"wp:attachment":[{"href":"https:\/\/blog.panicola.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2506"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.panicola.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2506"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.panicola.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2506"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}