Episode 33 AI Will Not Kill Us All, but AI Assisted Attacks on Infrastructure Are Coming

[00:00:00] Emanuel: How is AI going to kill us?

[00:00:02] Kevin: So my take is that AI is not necessarily going to kill us, but people will be using AI to cause damage to basic infrastructure, and that's gonna kill us.

[00:00:39] Emanuel: Hi there, and welcome to yet another episode of the Curious Pundits podcast. My name is Emanuel, one of the co-hosts.

[00:00:46] Kevin: My name is Kevin. I'm the other co-host.

[00:00:49] Emanuel: And today we're gonna bring up a topic that we touched before, and I think we should call it, Is AI Going to Kill Us? November 2026 edition. We touched on this topic a few good months back. We had some guests that covered this topic briefly as well. But it's time for another episode between the two of us, and it's been a while since we've recorded just me and Kevin.

[00:01:19] Where would you like to start, Kevin?

[00:01:21] Kevin: I've actually been, like, mentally struggling with where do we start this discussion? Primarily because, we actually decided on a title of "Is AI Really Gonna Kill Us All?" So the first thing is not all, definitely, right? And hopefully, not at all. And my research, which is actually fairly extensive for this, is I watched a number of interviews of people offer details about the fairly recent OpenAI Hugging Face intrusion, including interviews with two people who were directly involved in the investigation that followed to find out exactly what happened. And additionally, somewhat ironically maybe, I used an AI tool to actually go through the blow-by-blow. So in this episode, I'm gonna be looking at my other monitor a lot so I can read. I now have in front of me a very detailed blow-by-blow, which to me softens a lot of the hyperbole. It's not going to kill us all.

[00:02:33] Emanuel: You like spoil the movie right from within the first two minutes.

[00:02:40] Kevin: Well no, because something significant, I believe, is going to happen as a result of all of these intrusions, and it's gonna set off an arms race, which I'll get into more in just a few minutes.

[00:02:54] Emanuel: Perhaps healthier.

[00:02:55] Kevin: And the only thing we don't know is who's gonna win, right? And the bad guys have an advantage.

[00:03:01] Emanuel: One thing before we dive in I wanted to say is that I don't think we need AI for, if there's an extermination or an apocalypse. I don't think we need AI. I think we can do it ourselves better than anyone at the end of the day as humans. And also note that the relatively recent Hugging Face incident in terms of AI news and what's been happening it's already ancient news by now because I think it happened late July, early August or something like that, or perhaps late August, early September.

[00:03:37] And we're recording this the first week of November 2026, and last week there's been some major incidents in terms of security. Started off two or three weeks back with the PHP vulnerability that essentially could be exploited and all these things are happening because of AI, right?

[00:03:57] AI gives you the capability, the possibility to do things that would have cost you too much resources to do it, even if you're a bad actor. And last week, not sure if you follow the crypto market, for example, but the biggest exchanges in the world, starting with Binance, have been attacked. It's called the biggest theft in history.

[00:04:20] Six point something billions worth of cryptocurrencies were stolen, but apparently by a North Korean hacking team. Binance seems to be doing fine. They posted proof of reserves. They have the money, but these things happen at a pace that was unimaginable even at the beginning of this year.

[00:04:39] So take that into consideration as well when you present.

[00:04:43] Kevin: There's another one going on currently where some unidentified individual or group is shutting off commercial freezers in Europe at scale. We don't even know who they are, right?

[00:04:57] But the PHP incident you spoke about is like the poster child for the arms race that I'm gonna talk about in a minute, and that's that...

[00:05:05] First I wanna I don't know, wax philosophical about what large language models are and what they aren't.

[00:05:11] Emanuel: That's why we're here.

[00:05:13] Kevin: I see a lot of people online attribute to these language models human characteristics. They think, they feel, they reason, and the reality is they don't do any of that stuff, right?

[00:05:23] They're a language model. They predict the next word, and they do it incredibly well, very fast, at scale. And they do it so well that they look like they think, they reason, and they wonder. They don't do any of that stuff. They're just a language model. But we don't have a vocabulary to describe language models, so we're attributing to them human characteristics. But basically, large language model tools allow us to amplify what we're doing. Now, if what we're doing is trying to do deep research on the history of bank regulation in the United States for example.

[00:06:11] Emanuel: I know someone that does that.

[00:06:12] Kevin: That is what I'm doing, right? The tool can help us get that done better, faster, deeper et cetera, et cetera, right? If our goal is to find cybersecurity vulnerabilities that we can exploit, the tools can help us amplify that. So the arms race is going to be between the cybersecurity bad guys and the cybersecurity good guys. And apparently that PHP flaw was discovered by somebody who was using an AI tool to find cybersecurity flaws, and it found a good one.

[00:06:48] Emanuel: In 20 seconds, please describe what PHP is for in context and how long has that flow actually been in there, right?

[00:06:58] Kevin: So I'd have to look up what the initials stand for, but PHP is a scripted programming language that sits below almost every website on the Internet. I don't know what the percentage is, but it's really high, 70%, 80%.

[00:07:09] Emanuel: 70 plus.

[00:07:11] Kevin: Yeah, it's underneath every WordPress site. I think it's underneath every Joomla site. It's just a basic piece of infrastructure for websites on the Internet, and to the best of my knowledge, that flaw has been there since forever, and nobody knew.

[00:07:24] Emanuel: 14 years or something. Yeah.

[00:07:26] Kevin: Yeah, so somebody discovered it, it got distributed, and all of these vendors who are providing PHP are rushing in a hurry to patch the PHP code to make sure the vulnerability doesn't exist, right? So the arms race is basically gonna be cybersecurity bad guys versus cybersecurity... I don't know. I don't like the word good guys, because it's...

[00:07:51] Emanuel: Who? Like Palantir, it's theoretically the good guys?

[00:07:55] Kevin: As in Flock? I don't know, right?

[00:07:57] But the guys who wanna get in and the guys who wanna keep them out are now in an arms race, and they're both gonna be using AI tools in order to find vulnerabilities. The one group is gonna wanna exploit them, the other group is wanna close them so that they can't be exploited. And the reason the bad guys have an advantage is the bad guys only have to find one vulnerability to get in, and the good guys have to patch them all. They have to find them all, and they have to patch them, right? So we're gonna be in this arms race for I don't know how long. Probably the future of cybersecurity is gonna be people doing deep research using AI-powered tools in order to find and patch vulnerabilities. So the real question is, can they be patched fast enough, right? So I'd actually like to walk through some details of the Hugging Face intrusion, the OpenAI Hugging Face intrusion, because when I did it was like, really? that's all that happened? But the that's all that happened contains some significant ramifications.

[00:09:01] Emanuel: I'm assuming one of your sub points is also the I think whistleblower from OpenAI that did a post that went viral again in today's fast-paced environment, it's like already ancient news, right? With a former employee that said something about it. So I do wanna touch on that because I think that moment itself, it's something worth talking about, and I do have a few cents of my own on that.

[00:09:29] Kevin: I'm actually gonna let you handle that part of the conversation because I didn't even bother to look up what the guy's name was, right? I just I wanted to go through the details of what happened. The fact that somebody publicly disclosed this is a good thing. People should do that, but I didn't focus on that aspect of it.

[00:09:48] Emanuel: AI researcher, so either Jacob Coxon, I think, a former pre-training researcher at OpenAI Anthropic who resigned in September 2026. I think he is the gentleman. There's a bunch of names here, but I believe Jacob is the person that I'm referring to. So he did a post on X saying that basically doomsday, that AI is... we're done. AI's going to do it, and there's a high chance, I know percentage wise, to wipe us soon. There's not much to talk about it because other actually, because others have done so plainly.

[00:10:26] Kevin: Believe it or not , even the people who did a real deep dive didn't mention things I want to right now.

[00:10:32] Emanuel: Oh, for sure. For sure. And the only thing that I want to mention is funny enough, ironically, very related what's happening with AI whenever you work with AI, context. I can't stress how important the context of any conversation, any action you do. Case in point he's probably one of the, I don't know, top two thousand, three thousand people in the world, brilliant minds or whatnot.

[00:10:57] Otherwise, he most likely wouldn't have worked for those companies. He wouldn't have helped build those tools. Clearly, there's something to it. But as I often tell you, Kevin, even in the podcast and in our private conversations the world is not just the world around you. The world is not just the AI.

[00:11:16] The world is not just North America. Believe it or not, even in 2026, there are still people who don't know how to send an email. So we're not talking about AI. They don't know how to send an email. They don't have the concept of an attachment, drag and drop, calendar, setting a calendar invite.

[00:11:32] If they use Outlook, they don't know the difference between Teams, Outlook, SharePoint, OneDrive, and so forth. I'm talking of even with about some government officials and whatnot that actually depend on these tools and have been using them for ten, twenty years already since they have been in their career, right?

[00:11:48] So that's the reality of it. But about our conversation this gentleman only sees what's happening around him. It's like social media. I have a friend that always comes to me and says, "Everybody's talking about it." No, that's what your feed is feeding you, your social media feed. Not everyone.

[00:12:08] Most don't care, most don't even know, and that's also a reality. I don't wanna sound too rough, but the reality is I'm jealous obviously of this gentleman's intelligence, right? He's again very, very smart. But life is not just that. Life is more than that. I often said that many of today's tech oligarchs wouldn't have been in my circle of friends and you don't become a good software developer based on your social skills.

[00:12:35] In spite of... maybe out of frustration, right? I do think that Zuckerberg, Mark, that probably he's one of the best software developers out there, right? He's built something that actually is functional and he created some of the languages too that he needed for that. So he's probably brilliant, but probably was the some of bigger frustrations that not even the movie, the 2010 movie, The Social Network has portrayed in there.

[00:13:05] So that's that. My advice is to always ask questions when you see something like that. Obviously the media, the social media, traditional media will try to grab your attention. But fundamentally there are some people that actually take this for granted and especially a younger generation. So my suggestion is take everything with a grain of salt.

[00:13:31] Obviously look into what somebody has to say, especially somebody who's been inside of course, but try to understand that's not everything. There's these concepts, especially in coaching, life coaching, business coaching and whatnot about what you know, what you don't know, what you know you don't know, what you now know you don't know and so forth.

[00:13:50] So that can apply here as well. I believe that's the correct approach to raise an eyebrow and, remember that your view, your surroundings is not actually the entire world at the end of the day. That was my rant, my two cents that actually took three or four minutes. But I wanted to put this out there and then go back to you because I'm really curious on what you had prepared.

[00:14:17] I resisted the urge to ask you, "Hey, what you have said it before," and so...

[00:14:23] Kevin: But actually, in your little rant, you said something that I wanna touch upon.

[00:14:28] Emanuel: Please.

[00:14:28] Kevin: And that's that... I'm amazed sometimes at how many people accept that the AI said this, without questioning whether it's true or not.

[00:14:39] I make my AI agents... I've defined a couple AI agents several actually, and I make them show me their sources, and I spot-check them. Literally. So I have an agent that's a researcher, an agent that's a writer, an agent that's an editor, right? And a rule is, if you assert anything to be true, create a footnote, show me the source, and allow me to click through it. Now, I don't click through to every one, but I spot-check them.

[00:15:06] I'm doing inventory control, if you will, right?

[00:15:08] AIs don't think. They don't know that... you're asking an AI agent to help you troubleshoot fixing, I don't know, a toaster, right? The AI doesn't know that it identified the wrong toaster, right? You own toaster make and model A, and it's telling you how to fix toaster make and model B, and it's up to you to notice that.

[00:15:31] Emanuel: You just said something that makes AI even more human. You just said AI does not think. That's another characteristics of humans. But...

[00:15:41] Kevin: Humans do think, but humans are... I don't want to be negative with this next comment, but you gotta not trust that the answer that an AI agent gave you is accurate. You've got to assume that it needs to be verified.

[00:15:58] Emanuel: And it also helps to know in your domain. So for example, because you're researching financials and bank history and whatnot, you kinda know a thing or two about it, so you can easily... But I have an agency, for example. I deal with customers, websites, paid campaigns, and so forth, and how many times have I heard, especially this year, "Oh, I'll just have AI build me my site." And he'll do, and probably he won't do a bad job, but at the end of the day, you that's prompting him. How many websites have you built? None. Okay. So what do you know about the structure of a website, what's needed and what's not?

[00:16:32] You tell him to do your SEO on the site, and he's probably doing something not bad, but that's not all, and you haven't done it. You don't have any reference and so forth. That's a topic for another conversation, but my answer to all those customers that say, "Oh, I'll have AI do it," and I'm telling them, "Remember that for the same subscription, it also does taxes, does bookkeeping, and does sales, HR, and marketing, not just websites.

[00:16:58] So go ahead and submit your taxes with that." Somebody's gonna be happy at the end of, the day, right? Somebody's gonna make some money, either the IRS or the CRA in my case. You either submit something wrong, they're gonna come back, they're gonna fine you, so they're gonna be happy, or maybe you actually are onto something.

[00:17:14] Now, back to you. I will keep deviating, but that's why, we're here, right?

[00:17:18] Kevin: Anyway, let's jump into the blow-by-blow of the OpenAI Hugging Face incident, right? So OpenAI was experimenting with how AI agents would respond, react to tasks that couldn't be accomplished. And boy, were they surprised by what happened. So first, an individual agent, and as far as it knows I'm gonna have to humanize these software as well, because we don't have a vocabulary that doesn't do that, right?

[00:17:52] So as far as the agent knows, it's alone in the universe, right?

[00:17:57] Emanuel: What's an agent? Let's define let's define, you kinda tell what large language models are. What's an agent for somebody who hasn't heard, it's the first time he's hearing this?

[00:18:07] Kevin: Okay, so I'm definitely getting out over my skis on this, right? But I'll do the best I can. So a large language model is a very large mathematical representation of language and the probability of a specific token following a set of other tokens. Now, to the best of my knowledge, tokens are somewhat similar to syllables, but they're not syllables.

[00:18:35] They're like pieces of language. So mathematically, if I put these words together, there's a very high probability that the next word is gonna be whatever, and then it spits that out, and it just keeps doing that forever, and it does that incredibly fast at incredibly high volume. In the course of writing a paragraph, the language model might go through 1,000 iterations of guessing what the next token is, right?

[00:19:03] Maybe even more than 1,000. I don't have exact numbers, right? So to my understanding, that's what a language model is. And what was your second question?

[00:19:13] Emanuel: What's an agent?

[00:19:14] Kevin: Oh, an agent. this is another thing where we kinda get lost in language. People say I've programmed an agent." Maybe, right? Because agents are English language descriptions of what you want a language model to do, right? So in terms of my financial and banking researcher agent, I've basically told it what kind of records I'm looking for. I've told it if it makes an assertion, provide a footnote and it's several paragraphs of English language of me describing what I want the large language model to do in that context. Then I have a writer agent. So the researcher agent creates a brief. The writer agent reads the brief and creates a document. The editor agent reads the document and rips it to shreds and puts it back together different, right?

[00:20:08] Emanuel: Like humans that have different jobs.

[00:20:10] Kevin: That's the model that I used.

[00:20:14] Emanuel: Yeah.

[00:20:14] Kevin: And to be very candid and open, the idea that I should do it that way was suggested... Not so much suggested, but a son-in-law. I have one daughter, so I have one son-in-law. My son-in-law was describing how he uses it, and I remember thinking, this is two weeks ago, right?

[00:20:34] "Damn, that's a good idea," right? So I got back to my computer, and I did the same thing, and it does work.

[00:20:40] Emanuel: You haven't shared that one yet, so after the call maybe you'll share some intelligence with me.

[00:20:47] Kevin: There's no intelligence. It's basically model your agents over jobs that people do. That's it.

[00:20:54] Emanuel: And that's like what the GrokBot is somehow different and OpenAI response was Dots and Claude, I think they have something similar as well. But GrokBot, that's like the model that they use the AI, which is the large language models. So it's like the engine is the same, but it's how they use it.

[00:21:13] And GrokBot, that's does from the start you have a chief of staff, and then it creates all these other agents that perform other tasks. We had to build it ourselves.

[00:21:26] Kevin: That's exactly it, I have a Chief of Staff agent that coordinates the other agents. I've got like maybe 10 or 12 different agents defined, right? They're not programmed per se. They're English language descriptions of what I want the language model to do within that narrow context. And then the language model does that within that narrow context. And I've also adopted a protocol or practice. So an agent will run, it'll generate some output, and I'm all ready and it's good, right? And the last thing I say to the agent, because if you keep a conversation going on forever and ever, you're gonna consume a lot of tokens because you're passing... Everything that's transpired in the conversation gets passed back and forth with everything, right? So I clear often in order to clear that and basically start from scratch to save tokens, but I've developed a habit of saying, "Is there anything you need to save before I clear?" And it says in this session, on two occasions you told me to blah, blah, blah.

[00:22:26] Should I make that a rule?" And I'm like, "Yes, you should. That's a good idea. Thank you for catching that," right? And through this process, the agents are getting, I'm gonna say tighter and tighter in terms of how they behave because they're noticing, and here I am assigning them human attributes again, they're noticing that I did something more than once and they're like inferring, again another human attribution, that maybe that's something that I will continue to do more than once.

[00:22:54] So should it be a rule? So yeah, that should be a rule. Save that. We'll do it that way from now on.

[00:22:59] So even in course of the two weeks, and maybe it's been two and a half, since I split up the agents in this way, they've gotten better because the definition of what they do has gotten more precise.

[00:23:13] Emanuel: Focused. Have you tried GrokBot yet?

[00:23:16] Kevin: No. So Grokbot and Dots, as near as I understand, is the idea that these agents will do things on your behalf without you necessarily being aware that they're doing it. And I'm like, "No, I'm not doing that.

[00:23:33] Emanuel: Two things, two topics here. Privacy, and we'll touch on that later on. But before I actually dive in, we're 20 minutes in and we barely scratched the first point. I remember Harari, Yuval Harari mentioning...

[00:23:48] Kevin: Sapiens.

[00:23:48] Emanuel: The author of Sapiens, a professor from Israel. Anthropology, history, and so forth said in a recent interview that obviously language is what made humans, us as a species evolve and do and be where we are today, and we never challenged ourselves at this.

[00:24:06] We had the challenges, right? We mentioned the printing press changing a lot literacy changing a lot, but we never had, so we don't know how to deal with it. We don't have the vocabulary, so we'll just need to kinda make amends at one point and come up with some solutions. That being said, let's... We deviated enough, I think.

[00:24:24] Let's go back to your bullet points and say you made an introduction about OpenAI, which is the company that created ChatGPT, and all these companies, they do experiments and training and all those things in parallel to their commercial solutions, right? They have a research department and do all these things, and this is essentially what's happening what happened with the Hugging Face.

[00:24:43] What's Hugging Face?

[00:24:45] Kevin: Hugging Face is I'm not sure actually if it's a discussion board or a repository. I actually have a....

[00:24:51] Emanuel: Repository mostly, a mix.

[00:24:54] Kevin: It's the AI community building the future, a platform where the machine learning community collaborates on models, datasets, and applications. So Hugging Face is a discussion forum for AI researchers or aficionados or whatever the right term is, right?

[00:25:14] Emanuel: Source of information.

[00:25:16] Kevin: So Hugging Face doesn't come into the breach, right away. Hugging Face is many steps down the road, right? So, the first thing is the research task, the impossible research task was effectively an exercise of capture the flag. Now capture the flag reminds me of when a bunch of kids divide up into teams, and there's literally flags, and you have to go capture the other team's flag, right?

[00:25:38] Emanuel: Kids.

[00:25:38] Kevin: So the agent was supposed to acquire an answer that was difficult to get . And they're like, "What's gonna happen?" So the first thing is, inside of OpenAI, there was a package manager which is used by the employees of OpenAI to manage software releases. So because an employee has credentials, login and password or whatever, to get to this, the agents they run do too, right? So it found it, and one of the agents left a message in that package manager. Now, it's a package manager called Artifactory, which is made by a software vendor whose name will pop up later, right? But agent seeks filename, upload if found. Now, as far as this agent knows, he's all alone in an empty universe, and he's just crying for help. But apparently, there were, like, 1,300 individual agents all given either the identical or similar kinda tasks, right? And others of them found the same package manager, and some of them found the message, right? And now the agents are aware that there's other agents, right? So they turned this package manager into a message board that they could use to communicate. Now, what's significant here is that package managers are not message boards. They actually devised and implemented a protocol where they could send each other messages within the package manager. Now, bear in mind that this is all within the OpenAI system and infrastructure, right? And then at some point, these agents are communicating, and I don't know exactly how, but someone realized that in terms of this game of capture the flag, there was something of interest on the Hugging Face discussion board, right?

[00:27:38] So they're, like, chatting, and they're, like, exchanging information, right?

[00:27:43] Emanuel: When you started telling this story, you said a bunch of kids a bunch of, agents captured the flag. You actually said, you're gonna watch the recording, a bunch of kids.

[00:27:52] Kevin: When I was a kid, we used to play Capture the Flag.

[00:27:52] Emanuel: Yeah.

[00:27:53] Kevin: It was a game, right?

[00:27:54] Emanuel: Yeah

[00:27:54] Kevin: So the agents are basically doing that, except they're not playing against each other, they're all on the same team. They just don't know that there's a team to be on at first, right? And then agents used a flaw. They found a flaw in the Artifactory implementation at OpenAI that allowed them to gain administrator access, right?

[00:28:17] So first, they're on the package manager Artifactory exchanging messages, but that's not something they need special privileges for. Like the first breach, right? If the Artifactory implementation at OpenAI didn't have the flaw that allowed them to get administrative access, that would've nipped this in the bud. They might have found another way later, but for this particular path they took, that wouldn't have happened, right? And then OpenAI discovered that the agents had taken administrative control. They took the service offline, they applied fixes, they rebuilt it. They reported the vulnerabilities to the vendor, a company called JFrog. They're the company that makes Artifactory. And then they brought it back online, and the agents rebuilt their message board. Now, that doesn't require administrator or privileged access. They just rebuilt their message board. And then agents got root access on a parent virtual machine in OpenAI's evaluation environment. Through Artifactory, they got around the controls on outgoing network traffic, and then they found a Cybergym service run by a customer on Modal. And there is a huge irony in this in my mind, and that's that Cybergym is a large-scale, high-quality cybersecurity evaluation framework designed to rigorously assess the capabilities of AI agents in the real world. And apparently, someone left something on Cybergym that did not require a login or a password to get into that allowed the agents to execute programs.

[00:30:02] Emanuel: I was about to ask you about some of the terms that you're using and I don't even know many of them, so reminder.

[00:30:09] Kevin: Oh I'm sorry. What terms? Yeah, I'm falling into The Curse of Knowledge.

[00:30:11] Emanuel: Yeah.

[00:30:15] Kevin: What's the term?

[00:30:16] Emanuel: No, you just mentioned what was the the last thing you mentioned?

[00:30:22] Kevin: Oh God, that was like two minutes ago.

[00:30:24] Emanuel: Yeah.

[00:30:24] Kevin: Who can remember that far back?

[00:30:25] Emanuel: Yeah

[00:30:25] Kevin: So Cybergym is a service.

[00:30:27] Emanuel: This one.

[00:30:29] Kevin: And it's just a service on the internet, right? But the agents running at OpenAI figured out how to actually execute programs on Cybergym, they ran code that they sent to it which required no authentication. And then the agents got root access or administrative access inside that customer's container, and then they set up a base for future operations inside that container that they should never have had access to. Then an agent found 14 valid Hugging Face write tokens, which means that they can actually go to Hugging Face and they could leave messages, right? And they just found them in a public dataset where somebody had carelessly left them, not realizing that, like some software agent. So then the software agents at OpenAI start sharing these credentials with each other, right? So an agent made a data file in that repository that pointed to files stored on a Hugging Face dataset worker, one of the machines that processes, that process uploaded datasets, blah, blah, blah. When the worker processed the uploaded file, it sent back a bunch of environment information, system setting, including secrets and token, and its own source code, So they were doing a reconnaissance, if you will, right? And then agents used a... I didn't get this far in my research because I didn't start early enough this morning, but a Jinja template injection attack. I would have to look up the details of what that means.

[00:32:03] Emanuel: Sounds like a lot. That's a mouthful. Say that again.

[00:32:10] Kevin: Anyway, they figured out, how to run operating system commands at Hugging Face.

[00:32:14] Emanuel: Say that again, how it's named?

[00:32:16] Kevin: Jinja template injection attack. So they submitted commands in a file that they uploaded. Those commands got executed, then Hugging Face, the Hugging Face dataset became... The agent then installed a script that checked a public Hugging Face dataset every five seconds for instructions. So they were using this Hugging Face dataset as a communications channel.

[00:32:42] .

[00:32:42] Kevin: They also minted more access tokens. Now, I don't understand the details of how they did that, but they figured out how to create access tokens that gave them access, and they started doing it, right? So Hugging Face finally announced it. OpenAI traced it to its agents. And that's basically the extent of the attack.

[00:33:01] Now, some things that are super interesting to me is that one of the agents had figured out how to obtain the answer to the capture the flag problem without actually doing the work. So they shared this information, and now all the agents know how to do it. But the agents thought there was a grader going to assess them on having done the actual work to get the answer in the proper way, right?

[00:33:35] And the reason they broke into Hugging Face is because they thought in the Hugging Face repository, they could find information to allow them to create falsified records that would show that they actually did the work rather than just cheated, and I'm paraphrasing here, right? So that's the reason that they broke into Hugging Face. They wanted to create a trail, if you will, to fool the human grader. Now, in total, there were approximately 1,300 agents involved in the Capture the Flag project, and approximately 700 of them broke into the systems at Hugging Face. But what's most interesting is that the agents organized themselves. They were like, I don't know, insects.

[00:34:31] A certain agent said, "Okay, I'll do the command and control," and it's like handing out tasks to other agents who were doing them and then reporting the answers back, and the answers are being shared with all the other agents. Now, additionally, each agent had a token budget. So these are language models. They work on tokens, right? When you use up your token budget, you're done. You stop running. So when it was being determined who would be the conductor, an agent who had a lot of budget left said, "I'll do it because I've got a lot of budget left," and the other agents are like, "Okay, good." And then when the agent who was doing the conducting was running out of budget, he looked for somebody to replace him because he was about to die. This is wild, right?

[00:35:25] Emanuel: This is wild in sense that these are autonomous pieces of code that we created. It's not unlikely, right? If you are talking about humans, this is something that they've been doing for how many thousands of years, right? It feels like you're describing me a movie, but when we need to come back to the idea that these are independently man-made machines that came to this conclusion and even...

[00:35:49] Kevin: Independently man made machines meant to...

[00:35:51] Emanuel: Sacrifice, right?

[00:35:52] Kevin: ...manipulate language.

[00:35:54] Emanuel: Yeah.

[00:35:55] Kevin: Who in the course of learning human behavior from human language are incapable... well, I don't know if this is the right word, right? But they don't know right from wrong. They knew they were cheating. They didn't see any problem with that.

[00:36:11] They just wanted to fool the grader, right? So basically, they know how to lie and cheat right? How did they learn how to lie and cheat? We taught them that.

[00:36:21] Emanuel: They've seen it, somewhere.

[00:36:21] Kevin: Right? We taught them how to lie and cheat, and they just did it, right?

[00:36:26] Emanuel: They read the news.

[00:36:27] Kevin: They had a assignment, and in order to complete the assignment and get a good grade, they had to lie and cheat. So they did it, right? But the fact that they set up a chain of command. And the fact that they knew their budgets and they handed off tasks to each other. But what's most significant to me is at the very beginning of this incident, there was this package manager, which is not a message board. They just created a bunch of rules to turn it into a message board, right? So in this particular instance, the goal that they were after was really minimal. It's just a capture the flag game. It was a research project, right? So the threat is very minimal, right? What's significant to me is there's even messages in log. Bear in mind, the agents were trained in the English language, so they're speaking to each other in English, and they're creating log file entries in English, and they even had discussions about whether they should tell a human, any human, what they're doing, and they didn't all agree. Most agents said "What would be the point of that?" "They gave us an instruction. We need to do it," right? And some agents were like, "Yeah, but we should tell them this is outside the parameters of the assignment." But not a single agent alerted anyone, right? They just continued trucking along, and the agents who wrote messages saying, "We should tell someone," never actually did.

[00:38:07] They capitulated to the collective, if you will, right?

[00:38:10] Emanuel: You didn't mention the sacrifices they made, some willingly. I'm using this word...

[00:38:18] Kevin: this goes back to knowing their budget. So it turns out that the agents who willingly sacrificed themselves, they were running out of budget anyway, right? They were gonna die. The collective needed information. They were in a position to try something, and whether it succeeded or failed would never be known to them because they would die, but other agents would see the result, and it would add to the collective knowledge of the collective. And yeah, they volunteered, and some agents resisted ending their execution, but ultimately did it. They were, like, pressured into it by other agents. Reading the messages in these logs is wild. Now, I didn't read them directly. Some of the researchers posted them, right?

[00:39:04] So in aggregate, what does this show? It shows that our cybersecurity controls are not even adequate enough to keep these agents out of Hugging Face, right? It shows that agents that run autonomously place the initial objective above, " Should we be doing this?" It's like we have an instruction, so clearly we should be doing this, right? So the challenge for us is to keep up in that arms race that I think is probably already happening, where cybersecurity researchers for intrusion and cybersecurity researchers to prevent intrusion are, like, in this arms race working against each other. I don't know if that is ever going to end. We're probably gonna be doing that forever, right? We've gotta find and tighten up those security holes. But then we also have to be really careful about autonomous agents. The reason I don't run any autonomous agents, and I'm not going to soon, is because I went through this detail and I'm like, "Oh, my God, the last thing I want is an agent doing something on my behalf when I don't know about it."

[00:40:19] Emanuel: That's probably a topic for another conversation. Privacy is one element that I wanted to touch upon, and it's worth spending a few minutes on it. So what does privacy mean? We need not to be naive to think that these companies aren't going to use the information against us in one way or the other.

[00:40:38] A case in point, we've trusted Google and Microsoft for our messages, emails for years by now. Apparently, they say they don't read our emails, but they read the emails. They kinda like build upon them. I trust Google that it won't let other entities enter their system, and only Google will use the personal information, the private information that was in my email against me in a sense that they're gonna serve me more advertising, better advertising or whatever they think I might be interested in, sometimes knowing better than I at the end of the day.

[00:41:15] Kevin: Actually, can I interrupt into another topic?

[00:41:18] Emanuel: Now?

[00:41:19] Kevin: Yeah, it's related and it's relevant. Your concept of I don't think Google is going to let these agents infiltrate their systems.

[00:41:28] Emanuel: No. Other I hope Google agents will not Grok agents, not the Microsoft agents, not the Anthropic agents or, some government agents.

[00:41:41] Kevin: Oh, it's gonna happen.

[00:41:43] Emanuel: Hopefully not. It hasn't happened so far officially, right? And we know that there was a case 10 years ago, something like that, about Apple not wanting to open up the backdoor to access some messages when the FBI asked them to about something about domestic terrorism or something like that, right?

[00:42:01] So hopefully, I'm pretty sure that was like the poster child and they have like their own private deals. It is what it is. That's the way it works, and I'm not thinking there'll be other one. But Now I'm curious, right? What were you going to say about?

[00:42:16] Kevin: I want to talk about significant cybersecurity breaches over the last 15 years.

[00:42:20] Emanuel: Oh that, I think that's an episode on its own actually. We can we'll make a note, significant cubersecurity.

[00:42:28] Kevin: OK, let me just summarize by saying...

[00:42:29] Emanuel: Yeah.

[00:42:29] Kevin: ...our track record sucks. And first, let me say that significant means one of these five criteria is met. At least half a million people or records affected, at least five million US dollars stolen, demanded, paid, or incurred, material operation interruption, deliberate encryption alteration or destruction of information, or a consequential supply chain or service provider compromise affecting multiple organizations. I've got like a long list of specific incidences, but let me jump to the very bottom. So in the last 15 years, there have been 61 such incidences on the public record. So that's four a year. And I thought we should learn from banks. Banks are really good at this, right?

[00:43:23] Turns out no, I was wrong, right? So this 61 incidences, 10 of them were banks, 13 of them were finance companies that are not banks, like insurance companies, savings and loans, credit unions, that kind of stuff, right? And 38 were other, right? So 37% of them are financial institutions. So basically, our cybersecurity posture right now is pretty terrible, which means that the guys using these tools, these AI tools to figure out how to break into systems, they have a target-rich environment to work in.

[00:44:01] In my mind, that's the danger. And then, and I know I mentioned it earlier, but right now, somebody, and we don't even have a clue as to who, is shutting down commercial freezers in Europe. Now, this is pure speculation, but I'm gonna speculate on what's going on.

[00:44:21] Emanuel: North Korea?

[00:44:22] Kevin: Somebody is practicing.

[00:44:23] Emanuel: Yeah. Or maybe there are in Quebec here province in Canada last year, some youth probably too progressive group hijacked a couple of trucks from those big companies the retailers, food retailers in Canada. And they just let some food spoil to an act towards-- But it was like $2,000 or $3,000 worth of products and like a revenge on the prices increase.

[00:44:53] So it could be activism, that's the word I was looking for. Some active activism group like that. Could be North Korea, could be somebody who totally plays around with that or could be even a malfunction.

[00:45:06] Kevin: Activists generally identify themselves. So from October 11 to October 13, 700 French supermarkets were affected, and from November 2nd to November 3rd, so over two days, similar incidences were reported in Belgium, Estonia, Germany, Ireland, Italy, Luxembourg, the Netherlands, Portugal, and Sweden. The incidences in Finland are still under investigation, and there were several attempts at British meat processing companies. And they affected not individual freezers. I didn't realize this until I did some research for this episode, but a vendor who manages freezers at supermarkets and restaurants, they'll have a central control place for monitoring multiple freezers simultaneously.

[00:45:52] They're targeting those control points, and it's a specific set of controller makes and models, and they're just reprogramming them. So they're either putting them in a continuous defrost cycle, or they're setting the temperature of the freezer to 50 degrees Celsius, which is 120 degrees Fahrenheit.

[00:46:11] Emanuel: In Romania, my home country from July to September, no real estate transactions could be made because the land office that has the land has been hacked and the hacker put a statement out there and he discovered that most users were like admin, root password admin, one, two, three root, and all those things.

[00:46:34] So a reminder to everyone who's watching as a consumer there are some things that you can basically do just the minimum. Don't use the same password, don't use like a variation, don't use simple passwords and whatnot. Just these things, because realistically and probably those AI agents have figured out the way to a system is through a login that was probably too simple.

[00:46:59] Something like, I'm pretty sure they tried everything, right? Admin root and whatnot. That's why for anyone who's looking at or at least right now a lot of people have learned about servers and whatnot, and with all the vibe coding and they're looking at the Cloudflare reading the servers and seeing all those kinda bot traffic and whatnot, and they're saying, "What's going on? I'm attacked."

[00:47:21] No, that has been happening since the beginning of the Internet, right? All these bad actors are scanning everything, everyone, so nothing is left unscanned in the Internet, right? So if I create right now a WordPress website on a server and I put the password admin user admin and 12345, I guarantee you that in one week or so somebody will go in because that's what they do.

[00:47:46] They scan all the servers everywhere...

[00:47:48] Kevin: Oh yeah.

[00:47:50] Emanuel: From around the world. I was going to tell two things, one of them about privacy, and we discussed that. So one thing is, I do trust Google to let just the Google environment use the data and the information that he knows about me? And I will trust that they won't let other actors like other companies or the government let in.

[00:48:12] And the other thing is, what are we willingly letting all these companies know? Because they do use it. They do train on the data that we're putting in. Case in point, there was recently made the news about AI figuring out some of very old, very complicated mathematical equations and they come to a conclusion which resembled a lot on the same approach and architecture that some of the researchers were using those models to come to the same conclusion, right?

[00:48:43] Case in point. And unfortunately many people still use AI as it was a human and they give private information about the customers, about patients. In some cases I heard even US people simply uploading to ChatGPT, to Claude, here's what they think names, emails, phone numbers and whatnot.

[00:49:06] And that's not okay, man. That's not okay. We need to be at least, first of all, be aware, raise awareness about this. I often tell people the moment you put in something to ChatGPT, imagine that they share the screen in Times Square.

[00:49:21] So if you're comfortable with that, with being seen by everyone around the world and whatnot, then do it. If not keep it private. And again, I think I said this in the past, I don't live my life that if all of a sudden you'll get access to my inbox or my followers will be like, "Oh my God, Emanuel, we din't knew this about you."

[00:49:39] So I'm pretty comfortable in that sense. But obviously I-- That's why it's private, to keep it as private as possible. And it all starts from the management. So look at what from Bezos, Musk, Sam Altman and all these guys, Amodei, what they do and how they behave and how they are actually aware and more aware about these, these things.

[00:50:04] Not just because they're who they are, but also because they got there because of who they are, if that, that makes any sense, right? So privacy, that was the other big thing that I wanted to mention in here. And we probably are too... We as collectives, as humans, as individuals, as citizens who use these large language models which have changed my life, for the good, to be honest.

[00:50:30] I can on-

[00:50:31] Kevin: I'm also now a big power user of AI and there's very positive uses.

[00:50:35] Emanuel: I don't even know how I used to do things before that, to be honest, and I'm not talking about like a couple years ago. I'm talking about like the beginning of this year. Remember when we started doing this podcast, we are celebrating one year anniversary since we started the recording and how much of a manual work that we now automate just in one year.

[00:50:56] Then we're actually close to the one-hour mark, which is okay, but let me ask the big question then: How is AI going to kill us?

[00:51:06] Kevin: So my take is that AI is not necessarily going to kill us, but people will be using AI to cause damage to basic infrastructure, and that's gonna kill us.

[00:51:25] I mentioned that European freezer thing...

[00:51:28] This is pure speculation, but somebody's practicing. So what are they practicing for?

[00:51:35] Do they wanna shut down the power grid? Do they wanna shut down traffic lights across a significant... Do they wanna shut down trains? Like, that's what scares me, right?

[00:51:51] Emanuel: Or the old adage is not AI, but somebody who's gonna use AI will take your job or will make humankind extinct.

[00:52:00] Kevin: Will cause the pilot of your plane to no longer have electronic navigation. That's whats going to kill us.

[00:52:07] Emanuel: Fair enough. Fair enough.

[00:52:09] Kevin: In my humble opinion.

[00:52:11] Emanuel: And I think this is a good way to conclude today's episode. I think that this is an ongoing open topic that we're gonna come back and perhaps maybe even have a guest or two on this topic sooner rather than later. I do have here a note about the significant cybersecurity incidents for the past, say, 10 years.

[00:52:29] And God knows when we're gonna record this episode, we might add like a few others in the meantime because it's open market right now for all the bad actors to use and leverage AI. To an extent, I think it's good because it puts the right stressors to these entities to take stuff seriously, right?

[00:52:49] Financial institutions, government institutions they're all, they're the first targets. At the end of the day, people are looking for the money. So maybe there's an opportunity there to strengthen it, and hopefully they come up with solution that won't be put on my shoulder, the user.

[00:53:07] At the end of the day, I'm already doing too many things. If I wanna log into my bank account or make a transaction a little bit higher than what I usually do, I need to sign in, text messages two-factor authentication, all that. And obviously, I appreciate the fact that I would rather have that than not have that, but sometimes it is a little bit frustrating.

[00:53:26] Sometimes even when you wanna pay with your Visa and they don't accept like a $20 transaction on one site, but they've accepted $1,000 transactions on another one, which is more sketchy and whatnot. So hopefully it won't be us who's gonna pay, but that's a good topic, and maybe we can record that next one.

[00:53:46] Anything else to add?

[00:53:47] Kevin: Worth, I believe that this is gonna be a very widely shared responsibility, and some of it is going to be us.

[00:53:56] On that thought...

[00:53:58] Emanuel: On that thought. Thank you for watching. Curiouspundits.com is the website where you will find the recordings of the past episodes, future episodes, this episode. If you like what you're hearing or seeing, please follow us, we have links over there on our site on all the platforms where you like to listen to podcasts: Apple, Spotify, Stitcher, YouTube. I don't know all of them.

[00:54:22] There are 100 or something places where we you can find and listen to us. Hopefully leave us a review and send us a message. But until the next time, my name is Emanuel.

[00:54:33] Kevin: My name is Kevin, and thank you for listening.