> Furthermore, simplified metrics frequently distort incentives. If graduation rates are the metric by which funding is determined, then a school might do whatever it takes to bolster them. Although some of these efforts might add value to students’ learning, it’s also possible to game the system in ways that are counterproductive to actual education.
I thought this part of the article particularly interesting. The problem is not just the access to data, but the obsession to optimize the numbers shown. The solution is to understand that:
whenever a metric becomes a target, it ceases to be a good measure
Which is known as Goodhart's Law [0]. The problem with transparency and any kind of data is that you should be very careful when choosing your metrics and targets, as statistics can be deceiving. Unfortunately, this is probably too complicated to institutionalize in large groups of people who might not get the point, so ultimately a balance has to be chosen between transparency and confidentiality.
In the case of medical records, I believe doctors should have the option to write two notes: One accessible to the patients and the other for medical staff, without the option of sharing it. I fear the consequences of their work being so transparent could ultimately lead to bad communication and in the worst case leading to the wrong treatment.
It's amazing to me how often Goodhart's Law can be observed in the wild, and I've come to believe it is an important enough concept that it should probably be something (along with logical fallacies and other concepts related to critical thinking) that should be prioritized above a lot of other things taught in high schools.
So when you tell execs goodhart's law, they ask, ok what can we do about it when managing large organizations? Be it a 200 person department of a 30'000 person organization or even more?
The usual argument is you need a metric and a check metric that can catch gaming the first. Eg unit sales and mean price per sale — the latter preventing a race to the bottom on price point.
You need at least a dozen to prevent gaming of the first. It's not even that hard, but you do need to be aware that you need many distinct metrics and you will bring at least one down at least somewhat, and hopefully by repeated iteration you will end up with everything better.
And what prevents that when it becomes known there are n+1 metrics to game? And your doing this in a large organization, so it is statistically inevitable that people will want to game the metrics, bad actor or not?
The best approach is really hard. (Which is why proxy metrics in the first place.)
The best approach is to have a -very- clear set of goals, and to accurately measure progress to those goals, not to proxy metrics or goals.
Yeah, I know, easier said than done.
Usually a business has multiple goals, which are complementary. Enumerating, and measuring these make the system harder to game. Or more accurately make it harder to get better metrics without also getting closer to the actual goals.
If you want sales,then measure sales. If you want growth measure that. If you want customer satisfaction then measure that. If you want happy employees then measure that. List -everything-, measure everything. Understand the tensions, improvement in one area can drop the metrics in another. So work to keep improving them all.
All of this is -hard-. So most places don't bother. Find a simple measure (say stock price or turnover or profit) and ignore sustainability etc.
i dont understand how this relates to transparency. nothing about giving patients the right to access their own medical records involves metrics in any way.
ok, i kind of understand that, but it is still very different than a metric.
metrics become problematic because they are reductive and they are tied to compensation, promotion, or other rewards.
here we are talking about a raw dump of all data, which is the opposite of a reductive metric, and there is no compensation or reward tied to providing the data.
In the case of medical records, I believe doctors should have the option to write two notes: One accessible to the patients and the other for medical staff, without the option of sharing it. I fear the consequences of their work being so transparent could ultimately lead to bad communication and in the worst case leading to the wrong treatment.
[0]: https://en.wikipedia.org/wiki/Goodhart's_law