Our ability to provide comprehensive software suites to our users hinges on our ability to hire staff experienced in both scientific data analysis and software engineering... In the absence of such people, much larger teams containing both astronomers and industry programmers under formal project management need to be formed. - Economou et al. 2004However with the data reduction systems and the telescopes themselves becoming more and more automated replacements for those hero programmers are becoming hard to find because of the lack of experienced developers with an appropriate astronomical background, and the fact that it's not really a respectable profession...
We have found it extremely hard to hire good people to work on astronomical software. There is no career path within the universities for software specialists, despite the fact that there's no logical distinction between building hard- and soft-ware instruments. Smart and sensible graduate students, desirous of a career in astronomy, simply don't choose to specialise in the software required to reduce modern observational datasets. - Lupton et al. 2001Which of course is the reason we're having to replace those hero programmers in the first place, without some sort of established career path the astronomical software community is suffering from 'leakage' around the edges. My own situation is typical, I'm generally described by faculty as a "worrying senior" fellow.
Despite industry-led criticisms of the hero programmer paradigm, such software-scientists are a required. Building complicated bespoke systems to do science takes domain knowledge, not just of software engineering, but also of the underlying science behind what you want to accomplish. Simply put scientists, and the institutions the employ them, can't afford to support the large structured software teams that would be necessary if those hero programmers didn't consitently punch above their weight. Scientists also generally aren't that keen to get involved in the software design process made necessary by more formal processes that larger teams would entail.
Unfortunately historically those same scientists have been reluctant to provide the necessary support and career advancement that would be required to keep people like me around, sometimes through a misguided belief that software is easy and robust software can be produced by any wet-behind-the-ears graduate student.
While there is of course a huge oversupply of hopeful candidates for any long term posting in astronomy, but if you talk to software-scientists at those watering holes where we usually congregate, like the recent ADASS conference in Quebec, you'll find more than the expected amount of doom-and-gloom going around. My situation isn't unique, I'm not the only worrying senior programmer living contract-to-contract...
Of course up until recently, despite our complaints, it's been other people worrying how senior we've become, not us. Most of the programmers that have managed to stick around inside academia for any length of time, and there are many that just come and go, are usually fairly good at what they do. That means they knew they could go out and get a 'real job', probably paying more than they were earning in academia, when or if it came to it...
Unfortunately, amougst other things, the current economic turmoil has taken away our comfort blanket and left us very much out in the cold. Although, perhaps, with a better winter coat and a set of decent boots than many these days. None the less, its not a situation that's going to encourage people to specialise in software.
I don't see any of this changing in the near future. In fact I see the situation getting worse, the current generation of students are further away from the software, and underlying hardware, than I've ever seen. A culture of black boxes is very much in evidence. But you have to ask, what happens when the black boxes break?
I definitely sympathize with your predicament. I did my PhD in Astronomy, and the majority of my effort was spent writing software to reduce Subaru Data for weak-lensing analysis. I decided not to pursue the standard professorial route, and applied to a few "Scientific Programmer" positions.. several never responded to me (perhaps they had someone in mind already), and the ones that did paid absolute beans. While there are "computational astronomy" positions, for the N-body crowd it seems, the positions for senior developers for data-related software are few and far between. I decided to say to hell with it, and now I work for Apple. :)
ReplyDeleteBy the way, I think I actually met you once at IfA in Honolulu, where I did my PhD with Nick Kaiser. NIce blog... I found it while looking for some PDL information and decided to poke around since I remembered your name.