AI Hilarity

Midwest Chick (you do read her every day, don't you?  Thought so.) posts about how a Professor was smarter than most of his students, catching most of them using AI to cheat on an assignment:

A very clever professor in Mississippi decided to test and see if his students were cheating using AI. So he put some code in his exam instructions using white font that couldn’t be seen by the students, but could be seen by an AI if a copy/paste was used. 

Click through for the details (which are hilarious) but even more so for the comments her readers left.  My favorite was one from Carl Bussjaeger:

Oh, yes. I should probably mention DuckDuckGo’s AI. When called out on all of its hallucinations, it informed that if I don’t want it to make s–t up when asking strictly factual questions, I have to specifically TELL it not to make s–t up. 

 Sure you can trust AI.  Just tell it not to make s**t up ...

No, the Climate Change Establishment has not changed their approach

Friend Burt emails me to say that the U.N. Climate Panel Quietly admits Its Doomsday Climate Scenarios were completely "Implausible".

They've been admitting this for years and years.  Here's a post from the archives from 2019:

Climate scientist Judy Curry testified before Congress, and this is really interesting:

Some people (including one of the Members) took issue with the following statement in my testimony:

“Based upon our current assessment of the science, the threat does not seem to be an existential one on the time scale of the 21st century, even in its most alarming incarnation.”

I referred to AR5 WGII:

“Every single catastrophic scenario considered by the IPCC AR5 (WGII, Table 12.4) has a rating of very unlikely or exceptionally unlikely and/or has low confidence. The only tipping point that the IPCC considers likely in the 21stcentury is disappearance of Arctic summer sea ice (which is fairly reversible, since sea ice freezes every winter).”

The IPCC is the same "U.N. Climate Panel" getting headlines today.  The IPCC Assessment Reports are the "State of the Science" reports they release every five years.  They are considered the Gold Standard for establishment science, and they've been very quietly burying exactly this point for a long, long time.

The only thing they've given up is keeping these scenarios for the screaming headlines.

The only thing interesting here is why they are doing this now?  I think it's the mad dash to AI, with AI datacenters springing up everywhere.  Each of these has a ravenous power appetite, and the (very wealthy) folks behind AI have told the IPCC to tone down the climate alarmism.

Money talks, so the mainstream climate science establishment walks. 

Hallucinations come to Mass.gov

Okay, okay - Mass.gov has been hallucinating for years and years.  But now they're automating things:

Today, Governor Maura Healey announced the launch of the ChatGPT-powered Artificial Intelligence (AI) Assistant for the state’s workforce, with the goal of making government work better and faster for people.  

"Open the pod bay doors, HAL." 

An interesting perspective on AI

Long time Internet Security guy Fred Cohen has some interesting thoughts on how AI can be less obnoxious [PDF]:

The nature of the problem (I think) is that the attempts at safety reflect the behavior of the people who programmed and trained the AI engines, and they are apparently snarky, obnoxious twits that think its better to argue about meta issues than to serve their customers, like me, with the real capabilities they have developed. 

Their version of safety is the opposite of mine. If you want children to be safe from AI, don’t let them use it. 

If you want adults to be safe from AI, don’t make it available. 

If you want a ship to be safe, don’t put it out to sea… but that’s not what ships are for. We trade the utility for the safety, and while making ships that leak like a sieve is a bad idea in my view, making ships that don’t sail is a fruitless effort.

... 

Solution 

The solution is to put someone in charge of these mechanisms in these companies who is not a snarky, obnoxious twit… and I hope this doesn’t exclude me from the candidate pool. 

There are also some rather direct solutions to the problem of providing information to people where the information is not something that should be provided to anybody as a matter of policy. The most obvious solution is not to incorporate any of that sort of policy-violating information in the learning process. 

Of course the snarkiness is the same problem. If you don’t teach the LLM to be snarky by feeding it snarky crap, it will probably not behave that way. It’s no different than a child brought up by respectful parents vs. disrespectful parents. They learn from their teachers. 

Conclusions 

If you don’t want trouble, stop asking for it. If you teach a dog to bite, you are unlikely to be successful at later telling it not to. If you train an LLM with views of pedophiles, fraudsters, and murderers, you are unlikely to get it to not carry that behavior through later on. 

I think that Fred's entirely correct here (note that we ignore the very serious problem of AI Hallucinations here). AI training is generally crap layered on top of the hallucination engine*.

But I wonder if this is an opportunity for AI companies?  If you did a better job training the AI to be well-behaved (like you'd do with your kids or your dogs) would you have a different - and more attractive AI offer?  How about politeand wellbehavedAI.com?  That's a branding that would stand out from all the others.  You could market it to parents worried about their kids, or to old fuddy-duddies like me who hate everything about AI?

I smell a billion dollars of venture capital here ... 

* It seems very likely that the AI algorithms cannot be prevented from hallucinating. 

AI Browser Extensions considered harmful

Well, duh:

Ad blockers and VPNs are supposed to protect your privacy, but four popular browser extensions have been doing just the opposite. According to research from Koi Security, these pernicious plug-ins have been harvesting the text of chatbot conversations from more than 8 million people and sending them back to the developers.

The four seemingly helpful extensions are Urban VPN Proxy, 1ClickVPN Proxy, Urban Browser Guard, and Urban Ad Blocker. They're distributed via the Chrome Web Store and Microsoft Edge Add-ons, but include code designed to capture and transmit browser-based interactions with popular AI tools.

I believe that the very first of Borepatch's Laws of Security - from way, way back in 2008 - was "Free Download" is Internet-speak for "Open your mouth and close your eyes".

Plus ca change ... 

So you really shouldn't use them. 

 

Gartner Group recommends companies ban AI browsers

This is big news.  Gartner Group is the largest IT trend analysis firm, used by essentially all large corporations.  They just recommended blocking the installation and use of AI browsers:

Agentic browsers are too risky for most organizations to use, according to analyst firm Gartner.

The firm offered that advice last week in a new advisory titled “Cybersecurity Must Block AI Browsers for Now,” in which research VP Dennis Xu, senior director analyst Evgeny Mirolyubov, and VP analyst John Watts observe “Default AI browser settings prioritize user experience over security.”

I've posted about the risks of AI browsers.  Gartner's recommendations track mine:

Gartner’s fears about the agentic capabilities of AI browser relate to their susceptibility to “indirect prompt-injection-induced rogue agent actions, inaccurate reasoning-driven erroneous agent actions, and further loss and abuse of credentials if the AI browser is deceived into autonomously navigating to a phishing website.”

The authors also suggest that employees “might be tempted to use AI browsers and automate certain tasks that are mandatory, repetitive, and less interesting” and imagine some instructing an AI browser to complete their mandatory cybersecurity training sessions. [Highlighting mine - Borepatch]

The highlighted bit is a very clever way to get attention from IT departments.  Not only will it irritate the IT Security team but it will focus the Risk Management team on potential loss of SOC2 compliance.  This is a very Gartner way of getting eyeballs from the CISO and CIO.  Like I said, clever.

And yeah, I agree 100% with Gartner on this. 

 

 

How to attack AI systems

Use poetry.  No, really:

In a new paper, “Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Models,” researchers found that turning LLM prompts into poetry resulted in jailbreaking the models

...

Poetic framing achieved an average jailbreak success rate of 62% for hand-crafted poems and approximately 43% for meta-prompt conversions (compared to non-poetic baselines), substantially outperforming non-poetic baselines and revealing a systematic vulnerability across model families and safety training approaches.

Whoops.  Looks, this is a new class of attack (seriously, I've been in this biz for a long time and have never seen weaponized verse before), so maybe we need to cut folks some slack here.  But I'm somewhat less inclined to do so with AI's track record of falling for 30 year old attacks.

Enjoyed no sooner but despisèd straight,

Past reason hunted; and, no sooner had

Past reason hated as a swallowed bait

On purpose laid to make the taker mad; 

- Wm. Shakespeare, Sonnet 129 

 

The Age of AI Espionage has arrived

Well, it very likely arrived some time ago but now it's confirmed:

In mid-September 2025, we detected suspicious activity that later investigation determined to be a highly sophisticated espionage campaign. The attackers used AI’s “agentic” capabilities to an unprecedented degree—using AI not just as an advisor, but to execute the cyberattacks themselves.

The threat actor—whom we assess with high confidence was a Chinese state-sponsored group—manipulated our Claude Code tool into attempting infiltration into roughly thirty global targets and succeeded in a small number of cases. The operation targeted large tech companies, financial institutions, chemical manufacturing companies, and government agencies. We believe this is the first documented case of a large-scale cyberattack executed without substantial human intervention.

This is very interesting, and is very bad news.  This is one heck of a tool: 

In Phase 1, the human operators chose the relevant targets (for example, the company or government agency to be infiltrated). They then developed an attack framework—a system built to autonomously compromise a chosen target with little human involvement.  

Essentially, this is the cyberpunk version of "fire and forget" weaponry.  The only thing that would be more ironic is if they had a Clippy front end ...

(via)

Google is using AI to spy on you without asking for you to opt-in

Lawrence has the details.  You can turn the damn stuff off. And oh by the way, they're getting sued over it.

Skynet has arrived

Um, I've seen this movie:

Nation-state goons and cybercrime rings are experimenting with Gemini to develop a "Thinking Robot" malware module that can rewrite its own code to avoid detection, and build an AI agent that tracks enemies' behavior, according to Google Threat Intelligence Group.

In its most recent AI Threat Tracker, published Wednesday, the Chocolate Factory says it observed a shift in adversarial behavior over the past year. 

Attackers are no longer just using Gemini for productivity gains - things like translating and tailoring phishing lures, looking up information about surveillance targets, using AI for tech support, and writing some software scripts. They are also trialing AI-enabled malware in their operations, we're told. 

It seems that the Bad Guys are using all the old malware tricks (obfuscation, hidden files, etc) plus some new ones (sending commands via LLM prompts, i.e. the malware queries (prompts) other LLMs to get commands.

The security model for AI/LLM is hopelessly broken, and the design is defective.  I mean heck - the designers didn't consider two decade old attack techniques.  I don't know if it's correct to label this broken as designed but it's not far off.  This is software engineering malpractice.

I can't wait to see what happens with this and one of Elon's humanoid robots ... 

AI Browsers considered unsafe

OK, that post title is more than a bit inflammatory, but who on earth would want to use something like this?

Several new AI browsers, including OpenAI's Atlas, offer the ability to take actions on the user's behalf, such as opening web pages or even shopping. But these added capabilities create new attack vectors, particularly prompt injection.

Prompt injection occurs when something causes text that the user didn't write to become commands for an AI bot. Direct prompt injection happens when unwanted text gets entered at the point of prompt input, while indirect injection happens when content, such as a web page or PDF that the bot has been asked to summarize, contains hidden commands that AI then follows as if the user had entered them.

This is unbelievably bad.  How bad?  This bad: 

Last week, researchers at Brave browser published a report detailing indirect prompt injection vulns they found in the Comet and Fellou browsers. For Comet, the testers added instructions as unreadable text inside an image on a web page, and for Fellou they simply wrote the instructions into the text of a web page.

When the browsers were asked to summarize these pages – something a user might do – they followed the instructions by opening Gmail, grabbing the subject line of the user's most recent email message, and then appending that data as the query string of another URL to a website that the researchers controlled. If the website were run by crims, they'd be able to collect user data with it.

Surely they must be exaggerating, I hear you say.  Nope - the author of the post at El Reg recreated the exploit his very own self, simply by creating a web page with the commands hidden in it.  FYI, that's 1996 technology right there.

Now look, I may be an old crabby security geezer (no comments, Glen Filthie!) but the problem of sanitizing user input is a really old one.  So old that it was old when XKCD did it's classic "Bobby Tables" cartoon:

There have been over 3000 XKCD cartoons; that one was number 327.  Yeah, that long ago. 

My opinion about anything regarding AI is that the hype is so fierce that the people developing the applications don't really focus much on security, because security is hard and it would slow down the release cadence.  And so exploits that wouldn't have surprised anyone back in 2010 keep popping up.

Le sigh.  Once again, security isn't an afterthought, it wasn't thought of at all.  My recommendation is not to touch these turkeys with a 100' pole.

AI LLM poisoning attacks are trivially easy

This doesn't seem good:

Poisoning AI models might be way easier than previously thought if an Anthropic study is anything to go on. 

Researchers at the US AI firm, working with the UK AI Security Institute, Alan Turing Institute, and other academic institutions, said today that it takes only 250 specially crafted documents to force a generative AI model to spit out gibberish when presented with a certain trigger phrase. 

For those unfamiliar with AI poisoning, it's an attack that relies on introducing malicious information into AI training datasets that convinces them to return, say, faulty code snippets or exfiltrate sensitive data.

The common assumption about poisoning attacks, Anthropic noted, was that an attacker had to control a certain percentage of model training data in order to make a poisoning attack successful, but their trials show that's not the case in the slightest - at least for one particular kind of attack. 

...

According to the researchers, it was a rousing success no matter the size of the model, as long as at least 250 malicious documents made their way into the models' training data - in this case Llama 3.1, GPT 3.5-Turbo, and open-source Pythia models. 

Security companies using AI to generate security code need to pay close attention to this.  Probably everybody else, too.

UPDATE 23 OCTOBER 2025 13:08:  More here. It looks like solutions may prove elusive. 

Predictions for AI security

This is interesting even if it follows what we've seen for all security technologies since, well, forever:

Basically whoever can see the most about the target, and can hold that picture in their mind the best, will be best at finding the vulnerabilities the fastest and taking advantage of them. Or, as the defender, applying patches or mitigations the fastest.

And if you’re on the inside you know what the applications do. You know what’s important and what isn’t. And you can use all that internal knowledge to fix things—hopefully before the baddies take advantage.

summary and prediction

1. Attackers will have the advantage for 3-5 years. For less-advanced defender teams, this will take much longer.
2. After that point, AI/SPQA will have the additional internal context to give Defenders the advantage.  

 So basically it will be a shooting gallery for now with sanity restored later.  I'm somewhat optimistic of AI as a back-end tool (i.e. no user input) to run a set of interesting but more or less canned queries.  User input sanitization issues basically disappear at that point.

(via) 

Attacking AI via prompt manipulation

This is actually pretty clever:

The attack involves hiding prompt instructions in a pdf file—white text on a white background—that tell the LLM to collect confidential data and then send it to the attackers.

...

The fundamental problem is that the LLM can’t differentiate between authorized commands and untrusted data. So when it encounters that malicious pdf, it just executes the embedded commands. And since it has (1) access to private data, and (2) the ability to communicate externally, it can fulfill the attacker’s requests. I’ll repeat myself:

This kind of thing should make everybody stop and really think before deploying any AI agents. We simply don’t know to defend against these attacks. We have zero agentic AI systems that are secure against these attacks. Any AI that is working in an adversarial environment­—and by this I mean that it may encounter untrusted training data or input­—is vulnerable to prompt injection. It’s an existential problem that, near as I can tell, most people developing these technologies are just pretending isn’t there.

Essentially, this means that AI is simply not fit for purpose.  And clearly, it's not even a little bit "intelligent", security-wise.  

AI gets creepier every day

This starts out "yaknow, that sounds like a good idea" but gets creepier and creepier as you read more:

Ring doorbells and cameras are using AI to "learn the routines of your residence," via a new feature called Video Descriptions.

...

Once they do this, as Ring founder and Amazon VP of product Jamie Siminoff wrote in a blog today announcing Video Descriptions: 

Ring notifications will provide more meaningful information like, 'A person is walking up the steps with a black dog,' or 'Two people are peering into a white car in the driveway.'

The aim, according to Siminoff, is to shift more of the heavy lifting involved with home security to Ring's AI. This will also include "custom anomaly alerts," which are generated when "something happens on your property that is an anomaly to your property."

So far, so good.  Getting alerts only when something is unusual is generally considered A Good Thing when it comes to security.  But there's a downside:

And here's where it gets a little bit creepy: "It will learn the routines of your residence, get smarter, and deliver peace of mind by only notifying you when it is something out of the ordinary."

This gives us pause, as opposed to peace of mind, and sounds like super-charged snooping wrapped in an AI bow. If this kind of information is not properly secured, it could be a treasure trove for thieves, burglars, stalkers, and all other sorts of mischief-makers. In December 2022, a grand jury indictment charged two US men with breaking into Ring accounts to make fake emergency calls to police ("swatting"), then streaming the audio and video as the police arrived.

How long until AI will hack into your AI-enabled Ring account?  Asking for a friend.

Ring's response to The Register's reporter does not reassure:

The Register asked Ring where this information about users' home routines is stored, how it's secured, and under what circumstances it might be shared with law enforcement.

"We do not log the descriptions generated from Video Descriptions," a spokesperson emailed in response to our questions.

In the meantime, your humble vulture will continue to stick with dumb doorbells and barky dogs to deliver peace of mind about out-of-the-ordinary occurrences at home.

Endorsed.  Particularly the barky dog bit.