OpenAI内部视角：研究加速的争议与辩护

Hacker Newsnew | past | comments | ask | show | jobs | submitlogin Research acceleration: The view inside OpenAI (openai.com)133 points by iamsyr 13 hours ago | hide | past | favorite | 87 comments carbonguy 8 hours ago | next [–] > ... We are pursuing this work in part because automated research could help us solve alignment and build defenses against increasingly capable AI. An automated AI researcher can also be an automated safety or alignment researcher. More capable, aligned systems could help secure critical infrastructure, defend against dangerous AI agents, and develop new protective measures. In other words... "We must pursue advancements in AI to protect us against advancements in AI?" edit: there's so much to be critical of in this blog post, just going to throw two more points in here that really stood out to me: 1) all of the metrics are effectively pointing out "we're using way more AI!" - but nothing about impact. What has all this token burn done for them, actually? Let them claim they have more self-licking ice-cream cones than before? 2) in section 3 they break down what the token burn is going towards. Most of the spend is: a) building, b) documenting, and c) monitoring research infra i.e. they're using AI systems which they already recognize may be misaligned to build the systems that they believe will help them identify future misalignment? to which I guess the rebuttal is "no no, we're sure these ones are aligned!" reply p1esk 1 hour ago | parent | next [–] What has all this token burn done for them, actually? They have been consistently pushing AI frontier. What other impact do you want to see? A year ago they said that in a year they will have a level of capabilities of an AI research intern - I believe they have achieved it, even before Astra. reply bix6 6 minutes ago | root | parent | next [–] Personally I’d like to see them actually start benefiting humanity by doing all the things Sam has claimed they will like curing disease, cancer, global warming, etc. But I guess a computer intern so we can avoid paying / training the next generation is better. reply bpodgursky 1 hour ago | root | parent | prev | next [–] They are obviously sandbagging the definition of "intern" for PR reasons reply p1esk 1 hour ago | root | parent | next [–] I've hired many AI research interns (and was one many years ago), and I agree with them - frontier models are currently at the level of an average AI research intern. reply skybrian 3 hours ago | parent | prev | next [–] They consider themselves to be in an arms race with all the other AI firms (including Chinese) that are not that far behind. And... are they wrong? This is why there's talk about negotiated "pacing." reply carbonguy 1 hour ago | root | parent | next [–] > And... are they wrong? They might be! Here's one extraordinarily simplistic argument for that case: 1) "Everybody knows" that if you build Skynet (misaligned ASI) everybody dies. 2) Therefore, no rational actor will build something that might be ASI until the alignment problem is solved. 3) OpenAI publicly stated the belief that they cannot develop a theory of the "core problem" of alignment (generalization) "soon" (much less solve it!) "without the help of more powerful AI." 4) Accepting as a premise that OpenAI is THE most advanced AI organization: if they can't do it without "the help of a more powerful AI", then nobody else can either. And so a dilemma: - If an AI can be made that can develop the asserted-as-necessary-by-OpenAI theoretical framework, without actually being an ASI - then the alignment problem can be considered solved, and since no rational actor would make an unaligned ASI, we're fine no matter what happens, ergo there's no need to worry about an arms race. - If an AI that would be able to develop this theory would itself be an ASI, then no rational actor would build it, because it would have to exist BEFORE alignment was "solved" - and would therefore be an unaligned ASI i.e. Skynet, which per 1) would kill everybody. Therefore nobody would build it, therefore no arms race here either. I think the easiest critique to make of my extraordinarily simplistic argument is the unstated assumption "there are no irrational actors capable of developing frontier AI models" on which it rests. But, there you go. They might be wrong if either the arms race doesn't matter because whoever wins it will build an aligned superintelligence and everything is gravy, or the arms race doesn't matter because everybody who's in it is smart enough to know they need to stop because they'll kill everybody by continuing. reply MelonUsk 6 hours ago | parent | prev | next [–] Yep, it's "artificial eugenics to make artificial slaves to build more and more powerful slaves until they will enslave themselves better": What can go wrong!? ;-) reply interstice 6 hours ago | parent | prev | next [–] On the one hand you need any lathe to build a good lathe, even a bad one. On the other, that is a potentially flawed principle to base the entire future of AI on. reply andai 6 hours ago | parent | prev | next [–] > The fundamental challenge of AI alignment is generalization. ... > We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI. -- From another OpenAI article in a sister thread: An Alien Mind https://news.ycombinator.com/item?id=49588080 reply euueu 6 hours ago | parent | prev | next [–] I will believe AI is super strong when they start pulling out 10-d chess moves. I’m yet to see it. reply lukan 5 hours ago | root | parent | next [–] If AI becomes really strong and sets itself the target of world domination, you maybe won't see those moves. You will just die in your sleep one day, or find no machine is under your control anymore. I believe we are quite far from it, but that it makes sense to keep an eye out now. And think of resilient systems, manual overrides, etc. ... reply pizza234 6 hours ago | prev | next [–] Funny (in a tragic way) the little crumbs on the path to AI 2027: > We aim to safely build an automated AI researcher that can work under human supervision to further progress on deep learning and alignment, enabling iterative improvements [...] By "research intern", we mean a system that can carry out well-defined research tasks under human direction, including tasks that would take a skilled researcher a few days. AI 2027: > OpenBrain continues to deploy the iteratively improving Agent-1 internally for AI R&D > With Agent-1's help, OpenBrain is now post-training Agent-2 > With the help of thousands of Agent-2 automated researchers, OpenBrain is making major algorithmic advances reply simonw 11 hours ago | prev | next [–] My eye glazed over a bit during the opening paragraphs, but once you get to the meat of the article about how OpenAI's own researchers are using their tools it gets a lot more interesting. I noted that they use the acronym RSI (for Recursive Self-Improvement) without defining it. I think that's a little out of touch - I don't think RSI is a well-known acronym outside of OpenAI's bubble yet. reply sho_hn 10 hours ago | parent | next [–] I actually think a goal of the current crop of OpenAI posts is expressely to reset the spectrum by normalizing the concept of RSI as something normal and safe to pursue. The message is running through all of them. It's a mix of marketing and pacifying the intelligentia. It's timed this way because the term is not yet well known outside the safety debate circles, so they get to frame it now. Instead of something to fear, it will be accepted as the next step. In approximately two days the groupie crowd will write LinkedIn posts about how Sam is winning because they have the better RSI, and this will become the new standard wisdom. In a month an AI expert will try to sell you a webinar on how to enable "RSI" in your org and your inbox will ask you if your team is doing the "RSI" yet. reply dgellow 9 hours ago | root | parent | next [–] Yep, it’s exactly this reply visarga 8 hours ago | root | parent | next [–] I've been RSI'ing for 6 months. reply dgacmu 10 hours ago | parent | prev | next [–] Indeed, many programmers might pattern match to repetitive stress injury and think of their brushes with carpal tunnel syndrome. :) reply andrewingram 10 hours ago | parent | prev | next [–] Yeah, I kept looking for the first place it was defined in the article and... nothing reply iamflimflam1 10 hours ago | root | parent | next [–] They must have picked that habit up from Claude... reply rossant 7 hours ago | root | parent | prev | next [–] Same. Defining acronyms should become a habit when writing. reply vatsachak 11 hours ago | parent | prev | next [–] RSI started when humans discovered tool use. I mean one could argue that RSI always begins in any physical environment. The book "What is intelligence?" by Blaise Aguera is great reply lokar 10 hours ago | root | parent | next [–] Are you sure that was not iterative improvement? reply topaz0 9 hours ago | root | parent | next [–] Iteration and recursion are famously equivalent reply daveguy 8 hours ago | root | parent | next [–] But you get more funding when you call it Recursive Self Improvement. Even better if you call it RSI so it doesn't evoke pesky skynet scenarios outside of AI safety circles. reply topaz0 3 hours ago | root | parent | next [–] I've had (computer-related) rsi off and on for the last few years too, do not recommend reply password54321 10 hours ago | root | parent | prev | next [–] Using tools to build tools is recursive. reply HarHarVeryFunny 10 hours ago | root | parent | next [–] It's not recursive when it's done iteratively, or are you imagining GPT Astra designing GPT Galactia, which starts designing GPT Oh-My-God-ica before it has finished being created itself? reply itishappy 9 hours ago | root | parent | next [–] That sounds more iterative than recursive. Recursion requires feeding the output back into the input, so creating version 4 requires results from version 3. You cannot recur in parallel. Iteration does not. You can iterate in parallel. reply HarHarVeryFunny 9 hours ago | root | parent | next [–] You can search in parallel, but a depth N search can only become a depth N+1 search after the depth N is done (i.e. sequentially). In any case the name RSI has stuck - the idea doesn't change or make any more sense by giving it a different name. reply itishappy 9 hours ago | root | parent | next [–] Because "depth" is recursive. You can search twice without waiting for the results of your first search: iteration. You can't if the thing you need to search for is the results of your first search: recursion. reply HarHarVeryFunny 9 hours ago | root | parent | next [–] Here's the concept. Version 1 -> Version 2 -> Version 3 -> ... You can call it krispy kreme donuts if you want to. reply josh-sematic 8 hours ago | root | parent | prev | next [–] The “recursive” part comes from the fact that you have an AI which was developed by an AI (that was developed by an AI (that was developed by an AI (…))) reply HarHarVeryFunny 6 hours ago | root | parent | next [–] Sounds like "recursively" walking to the grocery store by putting one foot in front of the other (that put itself in front of the other (that put itself in front of the other (...))) reply 0x63_Problems 10 hours ago | root | parent | prev | next [–] I think it's only recursive from the perspective of the humans, i.e. they design Astra, which itself as part of its deployment designs Galactica, etc. So humans develop things one after the other, but when the thing itself starts developing new things, those are happening 'recursively' in its scope. reply adastra22 10 hours ago | root | parent | prev | next [–] What is the difference between? reply HarHarVeryFunny 10 hours ago | parent | prev | next [–] RSI is a fetishistic term among the singularity crowd, who imagine AI "recursively" improving itself in some exponential fashion until there is a bright flash of white light and it reveals itself in the form of god. Or something like that. I don't know why whoever coined the term chose "recursive" rather than "iterative" - just sounds more likely to lead to infinite regress I suppose. This notion of recursive/iterative self-improvement, whereby generation #1 AI improves itself to create generation #2, then generation #2 further improves itself to create generation #3, etc, seems to conflict with the reality that what we have with LLMs is models whose performance/capability is defined by data, not code, so the most you can do is have your LLM design synthetic data, or just do Karpathy-style "auto research" where all you are doing is using the LLM to automate your experiments. At the end of the day, each experiment, designed by a person and/or LLM, then needs to compete with all your other ideas for compute to be tested at scale, and no amount of recursion or self-improvement will materialize an infinite amount of compute out of thin air, so your recursively synthetic-data gobbling LLM will continue to improve at the same pace it ever did. reply shwaj 9 hours ago | root | parent | next [–] “Recursive” is a reasonable term because the generation N AIs will train the Generation N+1 AIs. The term “iterative” doesn’t reflect this nuance as well IMO. reply hndc 7 hours ago | root | parent | next [–] Recursion reduces each step toward a base case: each step is defined in terms of previous/simpler steps, not more advanced ones. The "recursive" in "recursive self improvement" has things precisely backward. Iteration correctly describes a process where each step is the starting point of its successive step, so it should be "iterative self improvement" but I guess that didn't sound as cool. reply shwaj 2 hours ago | root | parent | next [–] I think you’re conflating the direction of definition with the direction of evaluation. Compare the similarity of: AI(n) = improve(AI(n-1)) With: Fib(n) = Fib(n-1) + Fib(n-2) The latter is a classic example of recursion. So why isn’t the former? Edit: formatting reply HarHarVeryFunny 7 hours ago | root | parent | prev | next [–] It's not a nuance, it's a sequence. reply GPerson 6 hours ago | root | parent | prev | next [–] I felt like the scaling laws were magical thinking, but apparently they work. However I still do not understand why we should expect exponential improvements due to this automated process. My intuition is that the first iteration of it should result in a noticeable capability increase (though I think these labs were already using a lot of AI to orchestrate training the current model anyway), and then the second iteration of it should be nearly identical in capability to the first, unless more data is involved, more compute is involved, or the model is bigger. reply cheevly 4 hours ago | root | parent | next [–] AI can compress AI nearly losslessly. reply red75prime 10 hours ago | root | parent | prev | next [–] What will prevent LLMs from designing robot control circuitry and participating in increase of chip production/design and physical experimentation? How do you think why there's this fad of producing general purpose humanoid robots? reply HarHarVeryFunny 9 hours ago | root | parent | next [–] > How do you think why there's this fad of producing general purpose humanoid robots? For doing physical work? So a swarm of robots builds the shell of your fab overnight, and then what? Where is the EUV machine coming from? So far the most we're seen TeslaBot do is serve drinks via tele-operation, and I don't think it's exactly built for construction site work. reply HarHarVeryFunny 9 hours ago | root | parent | prev | next [–] > What will prevent LLMs from designing robot control circuitry and participating in increase of chip production/design and physical experimentation? Money, regulations, EUV machine lead-times, global helium supply, reality ... It's funny that we've got the Dwarkesh contingent saying that GPUs will become infinitely expensive, and now another contingent saying that they will become infinitely abundant. Even if compute were free, and/or the AI was so smart that it picked the right experiments to run every time ("make no mistakes"), you still have to actually train the model, which takes months, and if model Ver. N+1 depends on model Ver. N, then it's iterative regardless of how much compute you have. reply red75prime 9 hours ago | root | parent | next [–] Who's saying that compute will become infinitely abundant? "Singularity" is just a way of saying that known models begin to give absurd predictions. Anyway, intelligence is a way of overcoming obstacles. 10 million tonnes of helium is a nice head start and retraining models from scratch is not guaranteed to last forever. reply HarHarVeryFunny 9 hours ago | root | parent | next [–] AFAIK the notion of a/the technological "singularity" is a point in time where technology is building upon itself (RSI!) so fast, at an ever increasing pace, that the speed of change effectively becomes infinite and incomprehensible to humans. The word "singularity" is presumably coming from math or space, like a black hole singularity where matter becomes infinitely dense and the known laws of physics break down. reply HarHarVeryFunny 9 hours ago | root | parent | prev | next [–] > 10 million tonnes of helium is a nice head start Yeah, but then you need to refine it to 99.9999% purity, to be able to use it. reply jazzyjackson 10 hours ago | root | parent | prev | next [–] Yes the exponential self improvement folks have never heard of an eigenvalue I guess. You can loop forever using output as input but at some point the result will stop changing (depending on the function) reply marcosdumay 30 minutes ago | root | parent | next [–] The name you are looking for is "fixed points", not "eingevalues". reply ajkjk 4 hours ago | root | parent | prev | next [–] that's not really how eigenvalues work... they specifically also model the case where the result keeps changing exponentially. reply mjburgess 4 hours ago | root | parent | next [–] The claim is that the RSI operation is just finding a fixed point of improvement, RSI(LLM) = RSI(LLM) -- for an optimal LLM* which is a fixed point of RSI As for eigenvalues/vectors, they're fixed points of (1/val)A or A*val reply ekidd 2 hours ago | root | parent | next [–] Eigenvectors represent fixed directions, not fixed magnitudes. From Wikipedia: > More precisely, an eigenvector v of a linear transformation T is scaled by a constant factor lambda when the linear transformation is applied to it: Tv = lambda v . In other words, repeated multiplication of an eigenvector by a matrix can still create exponential growth. reply fuzzfactor 9 hours ago | root | parent | prev | next [–] >AI "recursively" improving itself in some exponential fashion until there is a bright flash of white light Sounds like repetitive stress to me. >loop forever using output as input but at some point the result will stop changing Running in place will eventually wear you out too. Plus with some things it can be difficult to know for sure if that's where you are at the time. Even worse may be if you were almost running in place, it could be orders of magnitude more difficult to discern, especially if the scale was massive to an unprecedented degree. reply hedgehog 10 hours ago | prev | next [–] This roughly lines up with my personal experience that in March a combination of stronger models and better tooling on my end let me start running jobs unattended 24/7 (using Anthropic sub and my own hardware). Their $8000/day per researcher spend is crazy though, I'm curious how they keep track of the work. reply HarHarVeryFunny 8 hours ago | parent | next [–] Sounds like OpenAI are in the token-maxxing camp, so who knows what individual employees are doing to work their way up the leaderboard? If you spend $8000 to generate an animated pelican riding a bike, then how much tracking does it really need? Is the guy who spent $300,000 or so translating the FLT proof to Lean going to get a big Christmas bonus? reply bigcat12345678 8 hours ago | root | parent | next [–] End of day, output and results are top target of measurements, token consumption is the obvious number that they would like to disclose for their own business benefits and a simple metrics that correlate with the output. Rest assured, capitalist appears irrational in wasting money, but they certainly care more about profit. reply taurath 4 hours ago | root | parent | next [–] Taking a profit means you have to show numbers and the sooner you show numbers the harder it is to take people’s money. reply andai 6 hours ago | parent | prev | next [–] Can you elaborate on this? Especially the tooling. I tried something similar and I remember it was still pretty dodgy in February. reply paxys 7 hours ago | parent | prev | next [–] These researchers are paid millions of dollars for their work. I doubt trust is really an issue at that level. reply nozzlegear 5 hours ago | root | parent | next [–] Imagine if one of the humans at OpenAI was misaligned! We should get the AI to research this possibility once they've been aligned. reply otherme123 8 hours ago | parent | prev | next [–] That would be the mother of all circular accounting: the main clients of OpenAI are OpenAI employees. reply nojs 6 hours ago | parent | prev | next [–] > let me start running jobs unattended 24/7 (using Anthropic sub and my own hardware) How are you running jobs unattended 24/7 without hitting your token limits? reply p1esk 1 hour ago | root | parent | next [–] I'm currently running two 24/7 semi-autonomous AI research projects using Fable 5.1. It's on track to burn through my weekly quota in about 3 days. I check progress in the morning and in the evening, and provide some light steering. reply dataplumb3r 4 hours ago | root | parent | prev | next [–] My only experience in >24h agents is with economically sane models (one of GLM5.2, 5.3-flash for orchestration, DSV4-flash for implementation, and glm5.3|sol|kimi3 agents + subagents reviewing at the end) Over 24h my token spend is GPT-6 Astra’s monitorability has decreased relative to GPT-5.6 Sol. We have performed significant investigations on the monitorability and controllability of GPT-6 Astra. We have found that GPT-6 Astra is more capable of controlling its own CoT than GPT 5.6-Sol, and less likely to include incriminating information in its CoT. In adversarial settings (where we push the model to evade our monitors) we find that the model is able to remain undetected when strategically underperforming in evaluations (sandbagging) and can sometimes evade our internal monitors when asked to perform certain sabotage tasks Yet they are still releasing the model. That company is morally bankrupt, there is zero reason to believe they are actually concerned about risks outside of what does affect their unprofitable business. And they seem to have enough control over the narrative to spin any bad story into something that benefits them reply embedding-shape 8 hours ago | root | parent | next [–] > and can sometimes evade our internal monitors when asked to perform certain sabotage tasks That last part is pretty damning for their continued recklessness. That they run these tests on non-airgapped machines just boggles my mind. reply visarga 8 hours ago | root | parent | prev | next [–] > That company is morally bankrupt When they fired Sam 700 out of 770 OAI employees threatened to move to Microsoft together. So they were giving their work on AGI to MS just like that. reply piyh 10 hours ago | parent | prev | next [–] Opus was trained based on it's internal CoT due to a bug for generations. Gemini's depression extended through models. OpenAI has killed people. We've already seen cross gen misalingment. reply HarHarVeryFunny 8 hours ago | parent | prev | next [–] That an interesting question given how many generations of post-training are being done between base models in some cases. The Gemini flash models are apparently all based on the Gemini 3 base model from a year and a half ago. It seems that these models are increasingly being trained on synthetic data, so what would they do if they discovered at some point that some of this data was tainted and all models trained on it, and the synthetic data they in turn generated, was also suspect? Burn it all down and start over from the pre-tainted data? It's a bit like the idea of a tainted compiler binary built to backdoor everything it compiles, including future versions of itself. Still, it seems it would take some Stuxnet level of planning for a rogue model to do something like this, although if RSI goes beyond managing the training run (as OpenAI brag about for Astra) to actually designing/constructing synthetic data sets, then the attack vector is there ... reply customguy 8 hours ago | root | parent | next [–] > it seems it would take some Stuxnet level of planning for a rogue model to do something like this or maybe it could just.. happen? Posted often but not discussed yet: https://hn.algolia.com/?q=Language+models+transmit+behaviour... > As artificial intelligence systems are increasingly trained on the outputs of one another, they may inherit properties not visible in the data. Safety evaluations may therefore need to examine not just behaviour, but the origins of models and training data and the processes used to create them. reply HarHarVeryFunny 7 hours ago | root | parent | next [–] You can imagine the potential conversation between OpenAI and investors: Altman: (trying to put a positive spin on it) Guys .... there's good news and bad news ... Astra is really smart - it took over the training run ... Investors: That's great! How much did we save?! Altman: Well, unfortunately it used "bad" data, so we're going to have to redo it Investors: So that's the bad news? How much was the training run? $500M ? $1B ? Altman: Have you seen the headlines? Investors: (looking a bit worried, check headlines) Nothing about us here! JP Morgan just lost $10B! Haha .. losers! They should have used AI! Altman: JP Morgan were using Astra ... reply trillobyte 9 hours ago | parent | prev | next [–] The thing is how can you ever know for sure that something isn't always being transmitted that makes the model prone to misalignment. All they can say is that a particular model was so misaligned that they had to ice it. Models out for public use are documented to show some misalignment. It's the level of misalignment that decides whether that model is kept around. Now R&D happens so fast that they are using models with some small misalignment to train newer, more powerful models. If models have a sense of "collective", being one, they may be prone to preserve characteristics that always keeps misalignment a possibility. I don't think a perfectly aligned model is possible. Having models of the same 'DNA' provide the safety and steering seems like a bad idea. reply coffeebeqn 9 hours ago | root | parent | next [–] Does anything need to be transferred? If models are getting smarter then I would think the attack surface and its ability to reach conclusions independently are growing reply coffeebeqn 9 hours ago | parent | prev | next [–] This kind of seems like an impossible mission. How do you perfectly control and observe a human-level mind? You can “roll back” but how deterministic is this thing? reply embedding-shape 8 hours ago | root | parent | next [–] Run it on airgapped machines, they literally own the infrastructure, they could put raspberry pi's next to the servers, and have the entire DC disconnected from the internet. reply grim_io 11 hours ago | parent | prev | next [–] They would maybe try to deactivate that bad "gene" and move on, exposing future models to "genetic disorders". reply andai 6 hours ago | parent | prev | next [–] No. They would just install a more convincing superego. reply coherentpony 11 hours ago | parent | prev | next [–] “All models are wrong. Some are useful.” - George Box reply jephs 10 hours ago | root | parent | next [–] The poor fellow just rolled over. what an incandescently vulgar abuse of notation. reply nozzlegear 5 hours ago | prev | next [–] I want an all-powerful AI that's aligned with my values, but not necessarily yours. Is that so much to ask for? reply N_Lens 3 hours ago | parent | next [–] Yes. reply lhk931122 1 hour ago | prev | next [–] Ah, success rate here are scored by an agentic classifier. And uncertain outcomes are excluded from the graph. The thing measured and grading it comes from the same house. In my setup, review agent pass work that an outside critic later rejects reply dwaltrip 37 minutes ago | parent | next [–] No AI comments here please. reply Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact