Smashing Security podcast #463: This AI company leaked its own code. It’s also built something terrifying

I had my data stolen once, Graham, from a governmental organization I worked at.

And they were selling it online for the bitcoin equivalent of $50 Canadian. And that made me feel very humiliated.

Were you like, please, please sell it for more?

I know, I was like, aren’t we worth more than that?

463. This AI company leaked its own code. It’s also built something terrifying. With Graham Cluley and special guest Tanya Janca.

Hello, hello, and welcome to Smashing Security episode 463. My name’s Graham Cluley.

And I’m Tanya Janca.

Tanya Janca, first time on Smashing Security. Hello. How the flip are you?

I am wonderful, Graham. How are you?

I’m gorgeous. Now, you are dialing in today from the beautiful Canadia. Thank you very much for doing that. Now, you are a famous name, right?

You’re a pretty big deal in the world of cybersecurity. So if people haven’t heard of you, how can you describe what you do and what you’re all about?

So I am a software developer turned application security expert who really likes to write. And now has written a bunch of books and tons of blogs.

I really like to speak, so I speak at conferences, and right now I’m giving secure coding training to large organizations and then kind of just doing contracts here and there, helping people change their application security program so it’s more AI aware.

Okay, so you are going into organizations and you’re helping those developers code more securely, which is a pretty good idea, I think, because we don’t want software which is full of security holes like Swiss cheese.

Well, we have a lot of that right now all over the internet. Right now, that is a giant problem, and especially not on the internet, embedded devices.

You know, you go into an emergency room, a hospital, all of those places, the security is usually much worse than it is on the internet, and it’s not great on the internet.

Now, a little birdie tells me, Tanya, that you have recently set up a rival podcast to Smashing Security, and you are basically I’m thinking that you can come in here and tell everyone about your podcast.

Is that correct?

It’s 100% correct, Graham. Right, right. My completely different topic podcast is called DevSecStation, and it’s 5 to 10 minute mini lessons for software developers about security.

So, this month I’m covering the supply chain and how to secure the supply chain and how software developers they’re a target now.

Malicious actors are actually targeting the actual developer, the human, and they need to know.

That’s interesting actually, isn’t it? Because of course, it’s easy to imagine how hackers could target people who work in the finance department, for instance.

But if they’re targeting the developers themselves, the idea, I presume, is to try to implant code within the code which these developers are writing, because eventually it will roll out to many, many organizations and could cause absolute mayhem.

Absolutely. So, often, the past couple years, people will say, oh, there was a software supply chain breach.

But if we look at maybe half of those, it was actually the software developer that was compromised.

And then as a result, multiple parts of the supply chain was breached because they have superpowers, because they can control the CI, and they control their IDE, and they control the repo, and they can go to prod, and, and, and.

And so, you get the developer’s credentials and suddenly you have everything.

And then on top of that, what some of the malicious actors have been doing, Graham, is then they rob the developer as well.

So they go and they try to empty their crypto wallets because why don’t we just kick people while we’re down?

Developers are the kind of people who quite often would have crypto wallets, wouldn’t they?

And so they understand the technology and so they may have a few thousand dollars or perhaps more.

They’d be significantly more likely to have a crypto wallet than the average person.

And I’m also thinking that, I mean, my background is I used to be a developer many years ago, used to write antivirus software.

And I remember from way back then that the programmers are also the kind of people who would demand to have admin privileges on their computers because they feel they have godlike capabilities anyway.

And so they would be arguing with the IT team, well, I need all of these rights. And that could be a security threat in itself, couldn’t it?

Oh yeah, for sure, Graham. And I was a software developer longer than I’ve worked in security. I was that person for sure.

And on top of having admin rights and being the lord of their workstation, I think a lot of people, when we think of the CI/CD, we think of it as a thing that publishes code and we don’t think about how it’s a thing that talks to the outside, does downloads, tells us if everything’s okay or not, decides to log or not log certain security things.

And very few organizations are currently logging or alerting, for instance, if a new admin gets added or if a new workflow gets added.

I worked at a place, I was contracting there, and we’re playing around with their CI because I’m going to add some stuff and—

Time, time, time, Tanya. Look, you’ve been developing code more recently than me, and I recognize that there’s a lot of listeners who may not work in the programming world.

You’re giving me some acronyms here. No, no, no, it’s all right. But what is that? What is that that you are talking about?

So a CI/CD, continuous integration, continuous delivery pipeline. It’s a piece of software that the software developers will put their code into, and then it will run lots of tests.

It will go and get things off the internet for them.

It’ll add some updates, it can log things, it can send alerts, and then it will put a copy of whatever the thing is they’re bat you can use to hold your source code and take care of it and manage it and store it.

There’s this setting that you can do called .gitignore, and you list all of these files to say basically no matter what I say, don’t upload this.

Yes. Override my own stupidity. Yes.

Yeah, exactly. And I take advantage of that all the time. So there should be a default for every org and it should include those map files.

So that’s step one is that we want to have the ignore file things set up properly. And then we always know we’re not supposed to have debug mode in production, right?

So, we know that we should have on the build server these settings turned off.

And so basically this is like security misconfiguration happening twice, which is on the new OWASP Top 10 2025, as a top risk to web apps.

Basically, they didn’t configure the build server correctly and then they didn’t configure Git correctly. And then they don’t have a process or a checklist to check that.

So I would love to see those three things. I teach supply chain security.

I’m expanding and expanding that class all the time because there’s more and more that we’re doing wrong there.

And I feel like if organizations had a checklist and they had, you know, a hardening of these things that they’re using that are part of their supply chain, like we talked about earlier, if we properly hardened our build server.

So, the CI/CD and build server, those are usually synonymous. They’re usually the same thing.

Or you have a build server and then you have a pipeline and you connect the two, but usually, it’s all one big thing.

And so, if we were properly hardening that, if we’re checking it at least once a year, if we analyzed who, you know, there’s an alert. Oh my gosh, there’s a new administrator.

Who’s that? Why do we have a new administrator? We could do a lot better.

So, it is a human error, but the human error happened because we didn’t have processes to protect that human from making that error. And I don’t like to blame Alice or Bob.

I like to look at, no, but did we train Alice or Bob on this? Did we? Right? Did we have a safeguard to stop them from making this error? Did we have a policy?

Or do we just assume they knew? Because when we assume, we’re let down a lot.

So what we have here, Tanya, is an AI company which has leaked the source code of its AI coding assistant. Via a packaging mistake, which is kind of ironic.

I’m going to give you a little bit of silver lining on the cloud, right? Because this has all been a bit depressing.

Which is this. Maybe this will give you a little bit of comfort.

Do you get any comfort at all from the thought that the people building these tools are still fundamentally human and therefore fundamentally fallible?

Thank goodness it’s not the AI, right? It’s human error. Hey, yes, us humans, haven’t we done great? Because we’ve really cocked up on this occasion by leaking the source code.

I think we should feel good about that rather than it being an AI which screwed up, which surely is only a short way away.

But we don’t know that.

That’s true, actually. That is true.

Right? Have you heard this term dark factory?

Oh, hello.

So we don’t know if Anthropic is becoming a dark factory. So in manufacturing, it means we just have robots, so we don’t need lights.

But there’s software dark factories being built now where you don’t have a single software developer anymore, and literally every single part is only written by the AI.

And wouldn’t you think the AI company might be most likely to do something like that? I don’t know.

Well, thank you very much, Tanya. There I was trying to be optimistic and cheer everybody up, and you’ve just made it all doomy and gloomy again. Great. That’s great. Thank you.

This episode of Smashing Security is brought to you with support from CoreView.

Now, Joe, quick question.

If someone broke into your Microsoft 365 tenant right now and quietly disabled your conditional access policies, grabbed global admin rights, turned off Defender, would you even notice?

I’d like to say yes.

Well, that’s the spirit, Joe. Good job. But here’s the uncomfortable reality. 63% of Microsoft 365 tenants hand out admin rights not that they’re going out of fashion.

One compromised account and an attacker can quietly reshape your entire tenant.

No alerts, no noise, just someone systematically dismantling your defenses while you’re none the wiser.

So wait, restore from backup doesn’t fix that?

No, no, no. Backups protect your data. They don’t restore tenant-level configurations. There’s no native rollback for that.

You could be rebuilding your tenant settings from scratch for weeks.

And who’s doing that?

Exactly. Who wants to do that? Well, CoreView have written a white paper called Total Tenant Takeover: The Microsoft 365 Disaster No One’s Ready For.

It’s actually a really practical read.

It covers how these attacks unfold step by step, where your existing tools are leaving gaps, and what it actually takes to recover control once it’s been lost.

So less detect and panic, more here’s how to actually get your tenant back.

That’s it. Exactly. And you can download this paper for free right now.

You can learn more at smashingsecurity.com/coreview and maybe do it before someone else does something bad to your organization.

That’s smashingsecurity.com/coreview. And thanks to CoreView for supporting the show.

And welcome back.

And you join us for our favorite part of the show, the part of the show that we like to call pickpocketing. Pick of the Week.

Pick of the Week.

Pick of the Week is the part of the show where everyone chooses something they like.

Could be a funny story, a book that they’ve read, a TV show, a movie, a record, a podcast, a website, or an app, whatever they wish.

It doesn’t have to be security related necessarily. Well, my Pick of the Week this week is actually security related.

In fact, my Pick of the Week this week, and this is gonna get very, very meta, not in a Mark Zuckerberg kind of way, because my pick of the week this week is actually about the Smashing Security podcast, because I’ve been busy doing a bit of vibe coding.

I know, very dangerous. I’ve been exploring the world of podcast transcripts, ladies and gentlemen.

I think it must have been about 9 years ago when I first got an email from a listener saying, why don’t you have a transcript? I’d much rather read rather than listen to you.

And I said, well, you know, it’s very hard putting together a transcript. I’d be up all hours typing my nonsensical words into a word processor.

Or I’d get some computer system to try and transcribe me into written English. And, you know, the quality is going to be diabolical anyway.

After quite a lot of work involving largely pipe cleaners and pots of treacle, bicycle chains, I have got together a Heath Robinson-type solution which now has, I believe, acceptable transcripts for this show.

Now, my podcast host, does create automated transcripts.

So if you go into your favorite podcast app at the moment and look at transcripts, if it supports that, you will see a very, very bad transcript of the show.

My intention is to replace all of those. And if you go to my website or to the Smashing Security website right now, you will find a much better transcript.

And in fact, it will even display the words as they are being said. So you can read as you are listening I think it works reasonably well most of the time.

Sometimes it makes a mistake, for goodness’ sake. Yes, I know. Sometimes it will mix up my name with someone else’s or something will go wrong.

But most of the time, I think it’s pretty darn impressive. So my pick of the week, rather self-referentially, is the new transcripts on the Smashing Security podcast.

Go to smashingsecurity.com or go and check out my articles on Graham Cluley.com.

And you will be able to see the transcripts in all of their glory there and tell me that it doesn’t work.

And then I’ll have to try and work out what the code’s doing and try and fix it. Cool. That is my pick of the week.

I your pick of the week, Graham.

Thank you very much.

That was awesome. Well done.

Do you have a pick of the week, Tanya?

I do. So my pick of the week is a television show on Apple TV called Shrinking.

And it is about three psychologists that are friends that are all grieving because one of the psychologists, his wife died.

And it shows how he grieves, how his daughter grieves, how the two other psychologists grieve. And they teach all these different psychology lessons essentially in the show.

And last year I did a talk about the psychology of bad code and applying economic behavior types of concepts to our security programs.

And how if we do that, we can get better results. ‘Cause just yelling at software developers actually doesn’t improve code quality at all, as it turns out.

Just being mean to them doesn’t work. We’ve tried that for two decades. So, I was what if instead we did something different?

Have you tried the old cricket bat trick of taking a cricket bat and just bopping them on the back of the head? Does that help at all?

My old boss was have you tried violence, Tanya? And I was no, I haven’t. And he’s you’re not really trying to problem solve at all, are you?

Oh, so I’ve just realized why your show is called Shrinking because of—

It shrinks. Yeah.

I’m so stupid sometimes. It’s taken me this long to work it out. Okay.

No, but so I’m fascinated by the reason that people do things and why people react the way they do. I’ve always been really curious about things like that.

And so also so that I could get better results, right? If someone blows up at me, it’s like, why did they blow up at me? And often it’s not because of something I did.

It’s because they feel insecure or afraid or whatever.

And so in the show, they’re always explaining these different concepts and I keep seeing them pop up in my life, whether it be at work or personally.

And so most shows aren’t very educational, Graham. Most of them are kind of garbage.

Oh, really? I’d never noticed. I’ve just been watching Married at First Sight Australia. So I thought all of them were really high quality, personally.

But so this one teaches lots of psychology lessons and why people do the things they do, but in an entertaining way. So I don’t know, I like that.

I think if people are curious about, you know, why people do the things they do, they might like this.

And is this a drama or a documentary? What is it?

So it’s sort of a drama and it’s sort of a comedy. So I think they call them dramedies.

I think that’s what you call a one-humped camel, actually. So anyway, yes, carry on. So a dromedary, right?

Basically, there’s a bunch of parts that are sad, and then there’s a bunch of parts that are funny.

And so I think they call it a drama comedy, which they literally put on Apple TV, Dramedy.

Oh, I don’t know if I like that word. Yeah. I’m not so sure about that.

You’re like, no, I do not accept.

Anyway. Okay. So your pick of the week is the TV show Shrinking.

Well, that just about wraps up the show for this week. Thank you so much, Tanya, for joining us. I think you’ve been absolutely smashing.

I’m sure lots of our listeners would love to find out what you’re up to and follow you online or listen to your podcast, of course. What’s the best way to do that?

So they should go to shehackspurple.ca. And if you sign up for my newsletter, which is free, you’ll get invites to everywhere I speak. You’ll see all my new content every month.

You’ll get the episode of the podcast and you’ll get at least one meme. And memes are important, Graham.

Yes, that’s what we need more of, is more memes.

That and emojis and animated GIFs. And of course, Smashing Security is on social media as well.

You can find me, Graham Cluley, on LinkedIn, or you can follow Smashing Security on Reddit or Bluesky or Mastodon. And don’t forget to ensure you never miss another episode.

Follow Smashing Security in your favorite podcast app, such as Apple Podcasts, Spotify, and Pocket Casts for episode show notes, sponsorship info, guest lists, and the entire back catalog of 463 episodes, check out smashingsecurity.com.

Until next time, cheerio. Bye-bye.

You’ve been listening to Smashing Security with me, Graham Cluley, and I’m very grateful to Tanya for joining us this week and this episode’s sponsors, CoreView, Vanta, and Meta.

And of course, to all of our fabulous supporters via Patreon. This week, we’re pulling out of the hat Watson Burney.

Sounds like a 19th century detective who probably solves crimes exclusively by monocle. Example name.

Now, I strongly suspect he’s a Patreon onboarding form that’s gained sentience and signed up.

Kenneth Ingham, Dan H, just the letter H because apparently everything after H is classified. Yuri Taraday, who has tremendous energy. We’re very glad that he’s on our side.

Ragnar Carlsen, of course, always arriving by longship. We’re not going to argue with them. J, just the letter J, not to be confused with Matt H or John W or Dan H.

This is just the letter J on its own, unadorned, magnificent. Ted Wilkinson sounds like a cricketer.

Govinda Charya, Travis West, who sounds like he should be presenting a true crime podcast of his own. Thank you all so much. You are wonderful, every single one of you.

And those are just a few of the folks who are supporting us via Smashing Security Plus, which means that they get episodes ad-free earlier than the general public and can be pulled out at random to have their names mocked at the end of the show.

If you would like that, all you got to do is join us at Smashing Security Plus. Head over to smashingsecurity.com/plus for all of the details and you too can become a patron.

But there are also ways you can support the show which don’t involve spending a penny.

You can like, subscribe, leave a 5-star review wherever you listen and tell your friends about the show. Spread the word. Every little bit helps.

And it really does make all the effort worthwhile. Well, thank you so much for listening. Well done for lasting this long into the show. Not everyone manages this. There’s too much.

You deserve a little badge or a pat on the back. But until next time, take it easy. Take care. Stay secure, my friends. Toodaloo and bye bye.