AI researchers' challenges: atomic analogies and strained institutions

Raw Text

We need to heal AI research norms, not build a super project. The reflections and reverberations that we're feeling in the AI community from Oppenheimer's quest for the atomic bomb.

Nathan Lambert

Share

The pressures on the AI community, especially the so-called "top" labs, to compare themselves to the Manhattan Engineering District (a.k.a. the Manhattan Project) largely contribute to making an already tense moment in time for AI and those distributing large language models even more confusing. Some of the analogies people use really miss for direct technical reasons, but it is important to understand the implications of how the quest for the atomic bomb changed scientific society well before the deep learning revolution.

When the Oppenheimer movie first came out, people quickly rushed to make comparisons, saying we need a Manhattan Project for AI and that the risks or control policies for the two technologies will be similar. There are many reasons this is not the case.

While the atomic bomb was a clear target, and folks like Oppenheimer knew it immediately when learning of the scientific breakthrough in splitting the atom, AI is extremely unknown. While quantum mechanics were new, the theory was developing in line with this technology. With AI, the end goals are extremely amorphous and unknown. The nature of the AI risk and concern among the longer-term contingent of the community is entirely grounded in uncertainty and forecasting emergent behavior. My biggest criticism around AI safety is the lack of specific metrics that they would classify as concerning -- whenever a lab decides on what their metrics are, it should be shared publicly. The engineering-first approach of the Manhattan Project, with a specific goal, does not work when we don't have a clear target for AI development . The calls for investment and centralization should look extremely different than the development of atomic weapons, even if the magnitude of perceived world change can be relatively similar.

Monitoring all developments in AI is much harder than it was for atomic weapons. Powerful AI will be a substantial data transfer and computing expenditure that is not entirely different from expected internet processing. Atomic materials literally emit a physical, radioactive signature. This fact alone will make it so the compliance actions around building trust with AI will need entirely different levers, even if the goals are similar. One of my biggest lessons from a  workshop  with policymakers and government officials is that the building of trust and communication is crucial to mitigating risk, but it's primarily based on the specific technology at hand. Showcasing that you are not doing certain things with AI is extremely hard and leaves almost no physical paper trail that would be noteworthy (though, the digital footprint once a sufficiently powerful AI is built, would likely be seen).

At the time of the Manhattan Project, as documented in the film, scientists held substantial political sway. A lot of interesting details (and spoilers for movie-goers) aside, the public hearings that Oppenheimer endured marked a transition away from scientists being directly embedded in government decision-making structures around security (the biggest credibility drop for scientists since, is probably the COVID communication failures). Modern political institutions have weaker forms of this reality. These political institutions and power structures are balanced with the academic institutions, which aren't mentioned as much in the context of Oppenheimer, but are extremely relevant today.

Within this context of fragile trust, the internet has emerged. The science of AI is heavily driven by preprints posted on Arxiv, which rapidly increases the transmission rate of information. While nuclear physics was accessible to a relative few, the AI moment is happening at an extremely broad scale. Double down on this confusion is the fact that insiders were already distrustful of the academic AI institutions providing the most common credential -- paper acceptances. Research credibility in AI was on the down according the most individuals, as participation increased and the burdened peer-review cycle broke down. In the post-ChatGPT world, the incentive structures here are increasingly strained: major sponsors of these conferences, big tech companies, are often pulling back from participation right when public interest climaxes. It's to be determined where this great pressure actually cracks AI research norms, but it's likely correlated with how scientists increasingly are not driving the conversation around AI, while algorithms are.

The last driving theme of the movie is the call to action that many of the scientists felt. The call to action, that many initially resisted, was due to the obvious immensity of the work. The film shows many initial moral qualms eventually subdue to join the project. In AI, there are multiple forms of this and clear differences in the source of the motivation.

Given the level of capital investment allocated for generative AI, the ride is now one we will ride to grand success or "failure" that pivots into another use of the technology. We're riding the self-fulfilling prophecy of investment-building companies. In the atomic era, there were existential considerations that we have totally flipped the script on. For many at the Manhattan Project, it was  succeed or else we all die  and now many AI researchers are driving for success  even though we may hurt ourselves . Emotionally, this feeling is much harder to pin down -- humans are not well suited for adversaries that are not compartmentalizable like a warring nation (e.g. our constant lag on climate change). The emotional complexity of the situation is a big part of why  people are getting so burned out , and there's a lot of work to be done before that's clear.

Many of the scientists participating in the Manhattan Project reported it being the best time of their lives. People love it when they can transparently understand the motivations and likely outcomes of their work. AI is far from that, and it is worth continuing to distill down towards.

Interconnects is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe

Note: I recommend you to see the movie, or even   read the book it's based on if this sort of context is compelling to you and or your work.

Research distribution in AI

The algorithms I'm mentioning that control research distribution are the social media algorithms. Years ago, during my Ph.D., things like ML Twitter were driven by relatively small follower lists and graduate students telling the story and results of their papers. This process has almost entirely been super-seeded by a group of accounts that share every impactful paper immediately after it shows up on Arxiv (a remarkable service for discovery in the community). This separation from the authors tends to create dramatically higher viewership than one graduate student would get before, at the cost of narrative control.

A recent example, that I still may talk about further is this paper " Scaling RLAIF ". I could tell when I first saw it that it was  probably  not exactly what the title said. Work in RLAIF is so desperately needed, but this one paper doesn't make it clear how to scale RLAIF. It's a useful, small study on evaluating RLAIF with the primary metrics being reward model scores and AI labeler alignment, not exactly easy things to understand and connect to literature.

A few research cycles into this dynamic, authors now get implicit feedback on how to frame and share their papers. Previously, embellishing a paper a little bit could be a useful tactic for acceptance in a conference as the reviewers could check the "high impact" button in the review. This is not a new tactic for academics, yet the awareness of it and the reckoning of it will come on. ICML was recently hit with a  viral blog post  about how it awarded its best paper award to a paper that succeeded by not sharing the full extent of the literature. While super frustrating, this is a side story to me in the trends threatening the research ecosystem. Best-paper vibes are now based on how your paper handles the critique and evaluation of the general public online.

1

However you sketch it, embellishment fuels the algorithm for scientific distribution. I've heard of multiple high-profile scientists trying to formulate projects for  the viral paper . We've seen so many examples of this with folks making unscientific claims about the apparent evaluation drift of a new AI-driven chat product, or some odd phenomena with no measurements or causality (AI Snake Oil is great for these  type s of  posts ).

Given how research projects are initiated, through long and slow funding cycles from organizations that care about reach, the ability to get your work to an audience does matter. There are a select few authors who get the benefit of having a following: they can immediately share their work with the world in a calibrated and accessible manner. This is the most modern form of internet dynamics driving increased power to fewer people. I don't like that this is the case, but it's something that HuggingFace taught me to do --  communicate and champion your work because no one else will . It's a reality that most of the scientists that I resonate with don't love, but we can only play with the cards we are given and ride the waves.

Power concentration, algorithmic distribution, dramatically increased participation, or rapid progress alone could be enough to destabilize an academic community. We get to have them all in AI.

There's one x-factor that subtly manipulates the algorithms and narratives driving AI research:  incomplete participation . Many companies are consolidating projects and restricting what can be shared. Both of these make it harder to follow scientific progress.

The restriction of sharing is the obvious one, fewer researchers are sharing their new ideas. There are regular plots showing which institutions publish the most papers at any given ML conference, and removing Google DeepMind from this would be like removing MIT and Stanford, or something similarly large.

This closure is normal when technology becomes technologically feasible -- competitive pressures are real -- but it seems like AI research was industrialized much more than previous technological revolutions. I've heard that at Google, many people have their projects absorbed into  Gemini  based on the  far-outness  of their researcher (anything related to transformers is absorbed, evergreen research / new methods, maybe not). This is leading to a big gap in the literature where Google used to validate the results and include academics in their projects.

The true validation for most scientific papers or ideas in AI is the repetitive trial of it in new papers. When multiple people come out with similar ideas at once, it validates a new method. I had this once in my Ph.D. where my DeepMind internship  work on exploration in offline RL  was released within a week of  extremely similar work from Berkeley . When many people were releasing many papers, it made it a little easier to estimate what was actually happening. Now, even if we assume the same amount of noise in a paper's reporting, we'll have more measurement errors in terms of trying to understand what it means. When a Llama X paper is released or Anthropic talks about RLHF, a small unclear statement could drive a lot of investment in incorrect directions. This was not always the case and makes reproduction have a different flavor.

So here we are, in the bizarre situation where people in closed labs are trying to track how long it takes for a new idea to trickle down into the open-source or academic communities. For example, I'd been specifically pushed on if I knew how HuggingFace came up with its  continuous batching implementation in TGI  and there are similar stories for  speculative decoding . The estimate was that it took 0-18 months for most of these ideas to get out.

We have a lot of cultural changes to get through with this new land of pulling back of information, while our distribution and credentialing mechanisms are in crisis. The best thing to do is not make assumptions about who did the work or what a paper says by its title and communication. The execution is always in the details.

Here’s the sort of thing that we need to overcome — I don’t know how audience capture and algorithms change brilliant and skilled peoples’ brain space to get to this point, but here we are (I recommend reading the thread for more context, and FYI Emily is leading the organization of another NLP conference next year, as of now). It’ll take a lot of work, but we can repair things.

Housekeeping:

Interconnects referrals: If you’re one of my lovely loyal fans, I’ll give you free paid posts if you use a referral link you find on the Interconnects Leaderboard . Sharing really helps me continue to publish and engage with fresh ideas.

Student discounts: If you’re a student and want a large paid discount, check out the About page .

Elsewhere:

Good summary slides on the issues facing RLHF, from Louis C ., the leading goose in RLHF.

Noahpinion piece paints an interesting theory around education and what generative AI will do. It’s a good read and may be true, but this assumes that access to education is the primary gating factor.

Noahpinion Is it time for the Revenge of the Normies? On the app formerly known as Twitter, I’m known for occasionally going on rants about how it’s good to be normal and average and middle-class. To some degree this is because I believe that the only successful society is an egalitarian one where people don’t have to be exceptional in order to live good and comfortable and … Read more 2 days ago · 250 likes · 119 comments · Noah Smith

1

quote from Reddit : "The field is so unserious that it’s frankly absurd how much importance is placed on these things. I for one have made a lucrative career on the back of my award that definitely wouldn’t have happened to the same extent otherwise."

Share

Previous

Single Line Text

We need to heal AI research norms, not build a super project. The reflections and reverberations that we're feeling in the AI community from Oppenheimer's quest for the atomic bomb. Nathan Lambert. Share. The pressures on the AI community, especially the so-called "top" labs, to compare themselves to the Manhattan Engineering District (a.k.a. the Manhattan Project) largely contribute to making an already tense moment in time for AI and those distributing large language models even more confusing. Some of the analogies people use really miss for direct technical reasons, but it is important to understand the implications of how the quest for the atomic bomb changed scientific society well before the deep learning revolution. When the Oppenheimer movie first came out, people quickly rushed to make comparisons, saying we need a Manhattan Project for AI and that the risks or control policies for the two technologies will be similar. There are many reasons this is not the case. While the atomic bomb was a clear target, and folks like Oppenheimer knew it immediately when learning of the scientific breakthrough in splitting the atom, AI is extremely unknown. While quantum mechanics were new, the theory was developing in line with this technology. With AI, the end goals are extremely amorphous and unknown. The nature of the AI risk and concern among the longer-term contingent of the community is entirely grounded in uncertainty and forecasting emergent behavior. My biggest criticism around AI safety is the lack of specific metrics that they would classify as concerning -- whenever a lab decides on what their metrics are, it should be shared publicly. The engineering-first approach of the Manhattan Project, with a specific goal, does not work when we don't have a clear target for AI development . The calls for investment and centralization should look extremely different than the development of atomic weapons, even if the magnitude of perceived world change can be relatively similar. Monitoring all developments in AI is much harder than it was for atomic weapons. Powerful AI will be a substantial data transfer and computing expenditure that is not entirely different from expected internet processing. Atomic materials literally emit a physical, radioactive signature. This fact alone will make it so the compliance actions around building trust with AI will need entirely different levers, even if the goals are similar. One of my biggest lessons from a  workshop  with policymakers and government officials is that the building of trust and communication is crucial to mitigating risk, but it's primarily based on the specific technology at hand. Showcasing that you are not doing certain things with AI is extremely hard and leaves almost no physical paper trail that would be noteworthy (though, the digital footprint once a sufficiently powerful AI is built, would likely be seen). At the time of the Manhattan Project, as documented in the film, scientists held substantial political sway. A lot of interesting details (and spoilers for movie-goers) aside, the public hearings that Oppenheimer endured marked a transition away from scientists being directly embedded in government decision-making structures around security (the biggest credibility drop for scientists since, is probably the COVID communication failures). Modern political institutions have weaker forms of this reality. These political institutions and power structures are balanced with the academic institutions, which aren't mentioned as much in the context of Oppenheimer, but are extremely relevant today. Within this context of fragile trust, the internet has emerged. The science of AI is heavily driven by preprints posted on Arxiv, which rapidly increases the transmission rate of information. While nuclear physics was accessible to a relative few, the AI moment is happening at an extremely broad scale. Double down on this confusion is the fact that insiders were already distrustful of the academic AI institutions providing the most common credential -- paper acceptances. Research credibility in AI was on the down according the most individuals, as participation increased and the burdened peer-review cycle broke down. In the post-ChatGPT world, the incentive structures here are increasingly strained: major sponsors of these conferences, big tech companies, are often pulling back from participation right when public interest climaxes. It's to be determined where this great pressure actually cracks AI research norms, but it's likely correlated with how scientists increasingly are not driving the conversation around AI, while algorithms are. The last driving theme of the movie is the call to action that many of the scientists felt. The call to action, that many initially resisted, was due to the obvious immensity of the work. The film shows many initial moral qualms eventually subdue to join the project. In AI, there are multiple forms of this and clear differences in the source of the motivation. Given the level of capital investment allocated for generative AI, the ride is now one we will ride to grand success or "failure" that pivots into another use of the technology. We're riding the self-fulfilling prophecy of investment-building companies. In the atomic era, there were existential considerations that we have totally flipped the script on. For many at the Manhattan Project, it was  succeed or else we all die  and now many AI researchers are driving for success  even though we may hurt ourselves . Emotionally, this feeling is much harder to pin down -- humans are not well suited for adversaries that are not compartmentalizable like a warring nation (e.g. our constant lag on climate change). The emotional complexity of the situation is a big part of why  people are getting so burned out , and there's a lot of work to be done before that's clear. Many of the scientists participating in the Manhattan Project reported it being the best time of their lives. People love it when they can transparently understand the motivations and likely outcomes of their work. AI is far from that, and it is worth continuing to distill down towards. Interconnects is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. Subscribe. Note: I recommend you to see the movie, or even   read the book it's based on if this sort of context is compelling to you and or your work. Research distribution in AI. The algorithms I'm mentioning that control research distribution are the social media algorithms. Years ago, during my Ph.D., things like ML Twitter were driven by relatively small follower lists and graduate students telling the story and results of their papers. This process has almost entirely been super-seeded by a group of accounts that share every impactful paper immediately after it shows up on Arxiv (a remarkable service for discovery in the community). This separation from the authors tends to create dramatically higher viewership than one graduate student would get before, at the cost of narrative control. A recent example, that I still may talk about further is this paper " Scaling RLAIF ". I could tell when I first saw it that it was  probably  not exactly what the title said. Work in RLAIF is so desperately needed, but this one paper doesn't make it clear how to scale RLAIF. It's a useful, small study on evaluating RLAIF with the primary metrics being reward model scores and AI labeler alignment, not exactly easy things to understand and connect to literature. A few research cycles into this dynamic, authors now get implicit feedback on how to frame and share their papers. Previously, embellishing a paper a little bit could be a useful tactic for acceptance in a conference as the reviewers could check the "high impact" button in the review. This is not a new tactic for academics, yet the awareness of it and the reckoning of it will come on. ICML was recently hit with a  viral blog post  about how it awarded its best paper award to a paper that succeeded by not sharing the full extent of the literature. While super frustrating, this is a side story to me in the trends threatening the research ecosystem. Best-paper vibes are now based on how your paper handles the critique and evaluation of the general public online. 1. However you sketch it, embellishment fuels the algorithm for scientific distribution. I've heard of multiple high-profile scientists trying to formulate projects for  the viral paper . We've seen so many examples of this with folks making unscientific claims about the apparent evaluation drift of a new AI-driven chat product, or some odd phenomena with no measurements or causality (AI Snake Oil is great for these  type s of  posts ). Given how research projects are initiated, through long and slow funding cycles from organizations that care about reach, the ability to get your work to an audience does matter. There are a select few authors who get the benefit of having a following: they can immediately share their work with the world in a calibrated and accessible manner. This is the most modern form of internet dynamics driving increased power to fewer people. I don't like that this is the case, but it's something that HuggingFace taught me to do --  communicate and champion your work because no one else will . It's a reality that most of the scientists that I resonate with don't love, but we can only play with the cards we are given and ride the waves. Power concentration, algorithmic distribution, dramatically increased participation, or rapid progress alone could be enough to destabilize an academic community. We get to have them all in AI. There's one x-factor that subtly manipulates the algorithms and narratives driving AI research:  incomplete participation . Many companies are consolidating projects and restricting what can be shared. Both of these make it harder to follow scientific progress. The restriction of sharing is the obvious one, fewer researchers are sharing their new ideas. There are regular plots showing which institutions publish the most papers at any given ML conference, and removing Google DeepMind from this would be like removing MIT and Stanford, or something similarly large. This closure is normal when technology becomes technologically feasible -- competitive pressures are real -- but it seems like AI research was industrialized much more than previous technological revolutions. I've heard that at Google, many people have their projects absorbed into  Gemini  based on the  far-outness  of their researcher (anything related to transformers is absorbed, evergreen research / new methods, maybe not). This is leading to a big gap in the literature where Google used to validate the results and include academics in their projects. The true validation for most scientific papers or ideas in AI is the repetitive trial of it in new papers. When multiple people come out with similar ideas at once, it validates a new method. I had this once in my Ph.D. where my DeepMind internship  work on exploration in offline RL  was released within a week of  extremely similar work from Berkeley . When many people were releasing many papers, it made it a little easier to estimate what was actually happening. Now, even if we assume the same amount of noise in a paper's reporting, we'll have more measurement errors in terms of trying to understand what it means. When a Llama X paper is released or Anthropic talks about RLHF, a small unclear statement could drive a lot of investment in incorrect directions. This was not always the case and makes reproduction have a different flavor. So here we are, in the bizarre situation where people in closed labs are trying to track how long it takes for a new idea to trickle down into the open-source or academic communities. For example, I'd been specifically pushed on if I knew how HuggingFace came up with its  continuous batching implementation in TGI  and there are similar stories for  speculative decoding . The estimate was that it took 0-18 months for most of these ideas to get out. We have a lot of cultural changes to get through with this new land of pulling back of information, while our distribution and credentialing mechanisms are in crisis. The best thing to do is not make assumptions about who did the work or what a paper says by its title and communication. The execution is always in the details. Here’s the sort of thing that we need to overcome — I don’t know how audience capture and algorithms change brilliant and skilled peoples’ brain space to get to this point, but here we are (I recommend reading the thread for more context, and FYI Emily is leading the organization of another NLP conference next year, as of now). It’ll take a lot of work, but we can repair things. Housekeeping: Interconnects referrals: If you’re one of my lovely loyal fans, I’ll give you free paid posts if you use a referral link you find on the Interconnects Leaderboard . Sharing really helps me continue to publish and engage with fresh ideas. Student discounts: If you’re a student and want a large paid discount, check out the About page . Elsewhere: Good summary slides on the issues facing RLHF, from Louis C ., the leading goose in RLHF. Noahpinion piece paints an interesting theory around education and what generative AI will do. It’s a good read and may be true, but this assumes that access to education is the primary gating factor. Noahpinion Is it time for the Revenge of the Normies? On the app formerly known as Twitter, I’m known for occasionally going on rants about how it’s good to be normal and average and middle-class. To some degree this is because I believe that the only successful society is an egalitarian one where people don’t have to be exceptional in order to live good and comfortable and … Read more 2 days ago · 250 likes · 119 comments · Noah Smith. 1. quote from Reddit : "The field is so unserious that it’s frankly absurd how much importance is placed on these things. I for one have made a lucrative career on the back of my award that definitely wouldn’t have happened to the same extent otherwise." Share. Previous.