Finding Metrics for Success
May 2013
A few weeks ago I came across a post by Cal Newport on his blog Study Hacks. He described his process for identifying metrics for success in his field. The majority of advice I see is based on anecdotal evidence or post-hoc analysis of a single successful career. Issues with that approach should be obvious; with a single data point, it is impossible to determine what factors actually led to the success.
Think about it as a machine learning problem. If you want to build a classifier that determines whether a person will be successful based on certain quantifiable metrics, many data points are required. In addition, only training on positive examples will also likely lead to a poor classifier. A dataset containing multiple examples is needed, and the set needs both positive and negative examples. Building this kind of dataset is what Cal recommended.
For me, I am interested in obtaining a research-focused tenure track position—preferably, but not necessarily, in the form of a professor at a university—so I identified people who had recently obtained such a position. I then went back and looked at other individuals who graduated around the same time and had the same thesis advisor. Once these individuals have been identified, it is just a matter following their career path since graduation.
Even by taking this more scientific approach, conclusions can be difficult to draw. Just because a person does not have the career I want, it does not make them a failure. Did they have different goals? Were they presented with different opportunities? Are they actually successful, but I just do not realize it?
Not everyone in my field wants an academic position. Some go on to make big money in the financial sector. Others obtain high-level positions in companies like Yahoo, Google, or IBM. One of the difficulties of measuring the success of a person who leaves academia is that their career becomes much more difficult to track; typically their publication record stops and they do not apply for research funding.
I attempted to limit my analysis to people who appeared to follow an academic track, but suddenly stopped. Maybe they spent 3 or 4 years as a post-doc before moving on to industry. Maybe they obtained a professorship at a research university, but left for a lower-tier university after 4 or 5 years—strongly implying they were refused tenure. While I am still drawing conclusions based on assumptions, it is the best I can do without actually tracking down these people and interviewing them.
Conventional wisdom is that you need to consistently publish in quality venues. All of the people I considered successful did this, but so did most of the unsuccessful people. It appears this is a necessary, but insufficient requirement for a successful career.
People who put a little more thought into it will also say that the publications need to be highly cited. After looking into it a bit, this metric is a bit murky. Several of the people that I noted as unsuccessful were highly cited. Some of the successful people had far fewer citations than I expected. Many of their most highly cited papers were actually older system papers from graduate school where they were the eighth author; even though the work was highly cited, it would be unlikely that the work would actually be associated with them. My conclusion is that being highly cited is helpful, but the most important thing is that your work is not ignored.
Surprisingly, research funding was also a bit of a mixed bag. While you are not going to get tenure without at least some type of funding, it also does not necessarily guarantee success. In the end, if you publish, get cited, and get funded, you are probably doing a good job.
On top of those three things, I did notice two other qualities that seemed to be universal in the successful subjects and absent in the unsuccessful subjects. I also rarely hear these two qualities mentioned. In my field, successful people publish in a variety of areas and collaborate with other researchers.
When I say they publish in a variety of areas, I don’t mean completely different subjects. I mean they do not stay in their very specific niche. For instance, my thesis was very much on the feature side of speech recognition. Moving forward, I should be looking at work in acoustic modeling and even language modeling. Working in a variety of areas also provides the opportunity to work with a variety of people.
Based on these discovered metrics, I have already started to implement some changes to my approach. Instead of focusing solely on my own research, I have started working with others on their research. Instead of guarding my own research projects, I am looking to involve others. No need to be greedy.