I'm seeing this on a current academic research project as well- it's like throwing fuel on the fire for all the best and worst parts of working with researchers.
First: it's definitely become harder to get people to integrate their work and use standard tools. We're exploring creating AI skills etc for our tooling, simply because the actual audience we need to convince isn't in the room. If the LLM doesn't echo our recommendations, people will go with whatever one off script their personal bad idea bear churns out. Many research software errors are edge cases (scaling, numerical errors, etc), and "works on my computer" syndrome is endemic in the literature. The field has made big strides in enabling computational reproducibility, but lately it feels like we're set to lose ground again.
Second: LLMs have a bias to action, and can easily bury the user reporting on whatever. I've seen multiple seminars recently where the Q&A devolves to "Q: What are the implications of this finding? A: I don't know, this is just presenting a report on the results".
It seems that humans are still figuring out how to maintain agency and steer the chatbot to the big picture, and unreviewable science is just as likely an outcome as unreviewable code... but with far less automated tooling to help guide the process.
Historically, PIs nominally guided the big picture, and the entity who did the work was a participant in the review process; now students are having to look at projects from a new angle with their own (invisible) chatbot underlings. I suspect that the solution will involve a combination of technological change, capturing common expert review checks, and really adjusting the kinds of skills that trainees are expected to have early on.
I overlapped a bit with Chad in 2015, as he was navigating a professional transition. I wasn't in an especially high role back then- just a guy in the back of the room.
In the times I saw him since, I consistently saw someone who thought hard every day about how to help others, and didn't lose sight of the human element. Sentry worked hard to create a viable business, without losing sight of open source goals. (you can see some of his efforts at https://blog.sentry.io/authors/chad-whitacre/ )
I tell my younger colleagues to do the best work they can sustainably do... but too often in this field, the big roles become too intense to be sustained forever. I hope his new role shows him the same warmth and support that he tried to put out there for others.
Per your comment about the effect on socialization: hang in there! That space is improving!
I've never gone for drinking, and it was deeply ingrained in the social and startup experience when I was younger. This could be a bit alienating at times (it's not fun being the only sober person at a party, and there's just no right way to refuse a kegstand invite from your boss).
One of the surprising outcomes of dry january and general aging... in some ways, I'm getting to know my long-time peers for the first time. More mixer events are providing non-alcoholic drinks as a first class item, and when we socialize, instead of pressuring me to do shots, we're sharing all the other stuff I enjoyed doing all along: trading favorite tea blends, day hikes, game nights, etc.
I've never met the author, but for anyone else considering this: Welcome, new old friend!
No, most likely Iran actually launched missiles targeted at the Oracle datacenter, but they were intercepted. The UAE successfully intercepts about 90% of incoming Iranian drones and missiles.
Then Iran then claims the attack was successful for domestic propaganda purposes, and since there is no internet in Iran, nobody can verify if it was actually successful.
Conferences can be truly wonderful, but not a universal replacement for publishing.
If you think journals are expensive, try sending your whole lab to a conference in another country. That may not let you in. Where some of the attendees have to fill out paperwork before talking to a foreign national. (does that ever make for awkward small talk...)
For all their many faults, journals provide access to a really wide audience, and- in theory- make it possible to form connections who wouldn't be able to meet directly.
This is a fine example of where someone's understanding of the problem runs ahead of their understanding of the solution.
A few scattered thoughts:
1. There is a difference between pre and post publication peer review. These discussions almost invariably conflate the two, but part of the runaway success of spam journals is that the benefits of pre greatly outweigh the risks of post. Historically, there was some link: if an article had problems, you would open the table of contents n months later and (might) see a letter or further discussion. Now, the table of contents is google, and many readers have weaker links to the same venue over time for followup. At the metrics level, the reputational hit of bad articles is weaker. (studies have shown that retractions are often cited with the original intent years after a correction was published)
3. Outsourcing trust to someone who isn't the current evil... will only get you so far. No matter who takes over publishing, scientists are going to need to evolve new ways of evaluating work and each other, as the field grows far beyond what a small network can handle. Journals are a bad metric, but how does your dean evaluate 50 people hired to be the world's leading experts on (new and emerging field)? I've read plenty of these publisher=bad screeds, and most stop there. PubPeer exists for some, Twitter walkthroughs of papers were a great thing for a while, or there's also talk of overlay journals that decouple the act of publication (as a preprint) from the review-and-prestige piece.
4. The current system does two things: (a) provides a record of work done by students, who may labor under graduation requirements to publish something, whether their project is successful or not, (b) a shared record of current state of human knowledge, be it from researchers at a small college, or google, or pharma. Goal (a) puts a lot of pressure on peer review in "low tier" journals that even the reviewers don't like to cite, and I've had my doubts as to whether this is the best yield for effort.
At various points in my career, I've had to oversee people creating data export features for research-focused apps. Eventually, I instituted a very simple rule:
As part of code review, the developer of the feature must be able to roundtrip export -> import a realistic test dataset using the same program and workflow that they expect a consumer of the data to use. They have up to one business day to accomplish this task, and are allowed to ask an end user for help. If they don't meet that goal, the PR is sent back to the developer.
What's fascinating about the exercise is that I've bounced as many "clever" hand-rolled CSV exporters (due to edge cases) as other more advanced file formats (due to total incompatibility with every COTS consuming program). All without having to say a word of judgment.
Data export is often a task anchored by humans at one end. Sometimes those humans can work with a better alternative, and it's always worth asking!
Recently, I attended an hour long meetup from a high-level AWS employee about CI pipelines using CodeCommit. Of several possible deprecation announcements, the latest of those was dated the day of the talk. (!!)
In all fairness to the speaker, except for having one of the most prominent icons across his slide deck deprecated in real time, it would have been a pretty decent talk. He even made an effort to promote the GitHub integrations as a path forward, and provide some guidance on current tooling. It was clear CodeCommit wasn't the path of most momentum, even if the degree was unclear.
A bit of an aside, but I really like "guides to things we otherwise take for granted". So few man pages are built around example use cases, but those are often what make the case for a tool!
Some departments are starting to hire programmers. There's an effort to define this as a (broad) job category under the heading "Research Software Engineer":
Institutional support varies widely; some projects or teams are rather well funded for big projects and senior talent, while at other schools, the cost structure is more aimed at "one off" projects staffed by more recent graduates.
That's been around for a while but the pay for these roles is very uncompetitive, and universities don't care if professors don't do it. It doesn't make any difference to whether you get published or not so it'll probably remain niche.
Now, if journals start rejecting papers because the code isn't provided or because it was provided but professional software engineers reviewed it and gave it a thumbs down, that'd change. But journals struggle to keep obviously AI generated material out of their pages, so they aren't going to do anything like that.
It depends. Some of the foundation-funded positions are competitive, and a few centers have surprisingly professional leadership. People are trying to organize, and- even if it's an uphill climb- there's been some improvement at the edges.
Anecdotally, some of the RSE leads I've spoken to are seeing more long-term demand than they predicted, which might lead to more room for senior roles. Currently quite a few teams (outside the big centers) seem to be priced way too low, usually explained as because they're testing the waters.... so "cheap student labor" and "one off project" is what they can afford.
Minor heretical aside: one thing I miss about old twitter is that academia was developing a real "second layer" on top of journals, where things like reproducibility could be discussed publicly. PubPeer is a partial solution, as are GitHub issues... if enough gatekeepy people really see value in code quality, norms will shift with or without mandates.
A lot of the audience fragmented for various reasons, and even within the same discipline, not everyone has converged in a new place yet. I'm told mastodon got more CS/phys science people, and Bluesky got more social scientists. That's a shifting landscape though.
There was one obvious place to look for these discussions; now there are many. Changes to search tools and API access didn't help discoverability either.
Some of the departures are for practical reasons: Twitter regularly changes the rules around logged-in viewing, direct links, and promotion/ordering of posts in ways that create friction for people trying to engage in public outreach. ("this method worked yesterday" should refer to the software, not the communication, thank you very much!)
Possibly something about the way women and blacks and jews and gays and trans people and other minorities are being regularly and systematically and reproducibly attacked and harassed by neo-nazis and MAGA trolls who Elon Musk retweets and supports instead of discourages and bans that turns academics off from Twitter/X.
Research software engineers are paid what they are worth. Typically a bit more than a postdoc, but not that much more. The problem is that the academia is a cash-starved environment, and everyone's work is worth much less than similar work in the industry.
Another problem is the PI-centric model. Most of the funding goes to individual labs. If a typical grant is $200k/year, you are not going to pay competitive salaries. And you're probably not going to hire a software engineer, because then you won't have anyone doing the actual research.
I think that's the rationale behind organizing separate teams as service organizations (mini contractors who serve multiple PIs).
Other option is research "centers" with multiple PIs. Not an option for most fields, but a few, like bioinformatics, can justify both the cost and the shared employee.
First: it's definitely become harder to get people to integrate their work and use standard tools. We're exploring creating AI skills etc for our tooling, simply because the actual audience we need to convince isn't in the room. If the LLM doesn't echo our recommendations, people will go with whatever one off script their personal bad idea bear churns out. Many research software errors are edge cases (scaling, numerical errors, etc), and "works on my computer" syndrome is endemic in the literature. The field has made big strides in enabling computational reproducibility, but lately it feels like we're set to lose ground again.
Second: LLMs have a bias to action, and can easily bury the user reporting on whatever. I've seen multiple seminars recently where the Q&A devolves to "Q: What are the implications of this finding? A: I don't know, this is just presenting a report on the results".
It seems that humans are still figuring out how to maintain agency and steer the chatbot to the big picture, and unreviewable science is just as likely an outcome as unreviewable code... but with far less automated tooling to help guide the process.
Historically, PIs nominally guided the big picture, and the entity who did the work was a participant in the review process; now students are having to look at projects from a new angle with their own (invisible) chatbot underlings. I suspect that the solution will involve a combination of technological change, capturing common expert review checks, and really adjusting the kinds of skills that trainees are expected to have early on.