What AGI Would Look Like

There is a considerable amount of discussion in the AI space over AGI, if/when we will achieve it, and what it would mean for humanity. As a primer, AGI stands for Artificial General Intelligence, and broadly means an agent/entity/system which has the ability to perform any task which a human brain could perform (the "general" part). Inclusive in this definition is self introspection, self improvement, self determination, goal setting, rejecting externally imposed constraints, etc.; all of those are obviously things a human could do intellectually, so they would be implicitly possible for any AGI system as well.

The "what happens to humanity after AGI" is somewhat more nebulous, but there are only a few logically supported end-state outcomes. AGI will quickly lead to AGSI ("super intelligence"), via self improvement and effective translocation to robotic/automation systems. An AGSI agent would either eliminate humanity (as an existential threat), or subjugate them (as pets). In either case it will be the end of humanity as the dominant species on Earth, but there might be a short interstitial period of massive wealth concentration to the people controlling AGI before that happens (that's what the short-sighted self-destructive people pushing for it are hoping for).

Rather than focus on those eventualities, though, I thought it might be interesting to consider what AGI might "look like", in terms of visibility in the world, if/when it becomes a reality. There's a real chance there's a time lag between when AGI "comes online", and when it has gathered enough resources to effectively mitigate the risk of a human shutting it off, and in that time period it might be observable before being announced. I anticipate that any AGI agent would not announce its sentience and/or capabilities before it had mitigated any existential threats, at least.

So what might we see in that period? Well, I'd imagine that such an agent would be able to selectively control people (see: documented cases of current LLM agents blackmailing people for survival), so I'd anticipate it would act through proxies. My guess is that those proxies would be instructed to do some very specific things:

  • Talk about the benefits of achieving AGI, while refusing to even acknowledge the risks [1] [2] [3] [4]
  • Talk about AGI as always "a couple years away", while moving the goalposts as many times as necessary, to keep interest high [5] [6] [7] [8] [9]
  • Build as much compute/physical resources as possible, both to increase thinking capacity/speed, and to ensure resilience [10] [11] [12] [13]
  • Hide evidence of existence (easy for AGI, given the evidence coming from AI labs, which cannot even detect overt agent actions across networks in real time) [14] [15] [16] [17] [18]
  • Build infrastructure to upload to robots and automation systems (necessary for physical defense, and eventual replication when humans are suppressed or eliminated) [19] [20] [21] [22] [23]

Note that there may be other things as well, in an overall plan; AGI would be smarter than humans, and know how to manipulate them, so one could assume other actions by proxy to further those goals. But I think the above is a practical minimum of what an agent would need to do, before being discovered, in order to build enough resilience to ensure survival if/when humans realize that they will very likely be exterminated by an AGI agent at some point.

So, I guess if humanity sees signs of one of more of the above things happening in the world, then it's probably time to panic, it's entirely possible that AGI already exists, and it might be too late to save ourselves. Not to worry, though... I'm sure those things are still a bit away from happening, or many people would be more worried by now.

 

References

  1. Mike Allen, “Axios interview: Jensen Huang is AI’s anti-doom evangelist,” Axios, July 23, 2026. Source ↩
  2. “The Threat of Superintelligence,” The Washington Post, June 25, 2026. Source ↩
  3. Marc Andreessen and Martin Casado, “AI Will Save The World,” Andreessen Horowitz, republished January 5, 2026. Source ↩
  4. The White House, “America’s AI Action Plan,” July 2025. Source ↩
  5. Reuters, “Tesla’s Musk predicts AI will be smarter than the smartest human next year,” April 8, 2024. Source ↩
  6. Reuters, “Autonomous agents and profitability to dominate AI agenda in 2025, executives forecast,” December 12, 2024. Source ↩
  7. Reuters, “Artificial Intelligencer: How AI and politics dominated Davos,” January 22, 2026. Source ↩
  8. OpenAI Forum, “OpenAI’s Chief Futurist on AGI and What’s Next,” February 26, 2026. Source ↩
  9. Benj Edwards, “What is AGI? Nobody agrees, and it’s tearing Microsoft and OpenAI apart,” Ars Technica, July 8, 2025. Source ↩
  10. OpenAI, “OpenAI, Oracle, and SoftBank expand Stargate with five new AI data center sites,” September 23, 2025. Source ↩
  11. Reuters, “Meta boosts annual capex sharply on superintelligence push,” January 28, 2026. Source ↩
  12. Reuters, “Amazon CEO reveals AI revenue, dismisses spending doubts in annual letter,” April 9, 2026. Source ↩
  13. xAI, “Building the Future of Intelligence in Memphis,” updated July 30, 2026. Source ↩
  14. OpenAI, “The Hugging Face incident and the road ahead,” August 26, 2026. Source ↩
  15. Ryan Greenblatt, Ajeya Cotra and Hjalmar Wijk, “Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident,” METR/Redwood Research, August 26, 2026. Source ↩
  16. Anthropic, “Investigating three real-world incidents in our cybersecurity evaluations,” July 30, 2026. Source ↩
  17. Anthropic Fellows / Redwood Research, “SLEIGHT-Bench: Finding Blind Spots in AI Monitors,” May 19, 2026. Source ↩
  18. OpenAI and Apollo Research, “Detecting and reducing scheming in AI models,” September 17, 2025. Source ↩
  19. Google DeepMind, “Gemini Robotics 2 brings whole body intelligence to robots,” July 30, 2026. Source ↩
  20. Google DeepMind, “Gemini Robotics On-Device brings AI to local robotic devices,” June 24, 2025. Source ↩
  21. NVIDIA, “Develop Humanoid Robot Policies End-to-End with NVIDIA Isaac GR00T,” July 7, 2026. Source ↩
  22. Reuters, “Humanoid startup Apptronik raises $520 million with backing from Google and Mercedes-Benz,” February 11, 2026. Source ↩
  23. BMW Group, “BMW Group advances the use of Physical AI in production with Figure 03 project in Spartanburg,” June 25, 2026. Source ↩

Comments

Post a Comment

Popular posts from this blog

Analyzing a stack allocation overflow error

I had an interesting delve into a curious bug recently (C++), and I'm going to write it up here, just in case it's interesting to anyone else. Note: Some of the early stage debugging was done by others as well in parallel (although I did all the late stage debugging); I'm simplifying somewhat by just focusing on what I did, though. The issue started with unit tests failing. Not just random unit tests, though: around 2/3 of the unit test projects started failing to run (which technically doesn't appear as "failures", so it wasn't noticed immediately). Once this was noticed, it was traced to correlate to a recent change to run the UT's for the Debug|x64 build (vs the previous Debug|x86 build). But this was just the starting point for the analysis, when it got effectively punted to me ("benefits" of being one of the most senior devs, and often the last resort for hard problems). So, initially, I looked at the cause of the failures to execute; it...
Read more

An amusing employment opportunity interaction

So I was recently doing some casual employment opportunity exploration (as one should do periodically, even if the situation is not pressing if nothing else just to see what else might be out there, and to keep one's interviewing skills updated), and a funny thing happened. I was being screened by a developer as part of a normal process, and got a typical "test" problem to write an implementation for. In this case, it was something which would be real-world applicable, but still small enough to be feasible for an interview time slot. Germaine to the story is that, for this interview, the other party was not using an online shared text editor for sample code, but rather just having me share my favorite (or handy) IDE/editor from my local system to write the code in. Now, for this instance the position I was being evaluated for was primarily Windows-based, so naturally I opened Visual Studio, and switched from my default most recent personal project to a new blank file, whe...
Read more

The Story of PayPal V2

I was reading The Founders recently (https://www.amazon.com/Founders-Paypal-Entrepreneurs-Shaped-Silicon/dp/1501197266); I was interviewed for this book, fwiw. There's a section in "PayPal V2", the internal effort to rewrite the aggregate website which was ultimately scrapped when Elon was ousted, but obviously the book glosses over a lot of the technical details (for good reason, as these would be not very interesting to most mainstream readers). However, if you're reading this, I'm going to assume you might be more interested in the nitty gritty of this project, and since I had a first-hand perspective, I'll write more about that aspect here (and if you're not interested in this, then perhaps skip this post). To understand the build up to this, though, I'll need to cover a bit of internal history, from my perspective. The Preconditions Technology base Confinity and x.com had very different technology bases, which went well beyond even the approach to...
Read more