AJ Nash: This isn’t gonna be any good. HR departments and universities and procurement teams and employers, they’re going to skip that caveat. They’re not going to know it. They’re not going to read it. It’s not their fault. They’re just going to be in a hurry. And so that’s going to be our biggest challenge, I think, is these theoretically objective scores that pop out and how you’re going to deal with that.
Are people going to be disciplined for this? Are people going to be fired for this?
Hello, and welcome to another episode of NeedleStack. I’m one of your hosts today. I’m AJ Nash. For those who know me, I’ve got a long history in intelligence. I don’t know, a couple decades worth in the intelligence community, counter-terrorism, counter-insurgency, and countering trafficking persons, chasing war criminals, shit like that. And then I’ve been in the private sector now for, ooh, geez, 12 years 11 years, something like that. Primarily building intel teams and, and talking about intelligence-driven security. And I’m excited to be here today again with my co-host, Kacper. You wanna go ahead and give the people a little introduction if they haven’t met you yet?
Kacper Piwowar: Of course, of course. So I’m Kasper, co-host this podcast with AJ. So I’ve been in the tech space for quite some time. Mainly large companies, working now in the intelligence space with Authentic8. So working closely with AJ and the rest of the team here. Deep in the industry, getting to know it more and more every single day. And yeah, I’m excited to have our chat today. I think it’s on a very pertinent topic, for everybody. It just somewhat recent, somewhat new, so hopefully you find it, you find it valuable.
AJ Nash: Yeah, absolutely. I think you’re right. We’re always learning, right? And today’s a good example of that. Like, the topic today is AI watermarking, which is actually a really new thing that’s just started to come out. So AI has been on this, really rollercoaster, this fast journey, right?
I don’t think anybody really understands how AI operates yet. I mean, even the developers who own and operate these platforms will say they don’t really understand how their models operate entirely. They have all sorts of different behaviors that they don’t expect on a pretty regular basis.
And so we have these massive, massively powerful tools that have been released to the public, but we don’t completely know how any of them work. And at the same time, we’re still trying to create, you know, rules for the road and, you know, operating manuals and all these kinds of things.
And so, you know, AI watermarking is one of those things that’s coming out now because the EU, right, the EU passed a law that requires, you know, AI platforms to create, like let’s see. I’m gonna read it ’cause I, I’ll get it wrong otherwise. “Requires providers of any system generating synthetic text, audio, image, or video to mark outputs in a machine-readable format and make them detectable as artificially generated.” Yeah, right, that’s the law, right? And so we’ve got what’s called AI watermarking, which I think is really a misnomer. But it’s certainly creating a lot of discussion and stress for people, you know, who are using AI on a regular basis. So I’m excited to get into it. So, you know, before we jump deep, we should kind of baseline when we say AI watermarking, you know, what are we talking about, right?
This isn’t typical watermarking. You know, I think of a watermark like on a
Kacper Piwowar: AJ, is it not like gonna say made by AI on my document if I…
AJ Nash: It is not. No. No, it is not, right? And that’s I think what some people are gonna think is, you know, you see that in your pictures and like, you know, pictures get watermarks to them or documents get watermarks on them and this is invisible, right? This is in the code itself, like as the law states, it has to be machine readable, not human readable.
So, it’s definitely not going to be what you and I might have called a watermark. I mean, that’s just not what this is. You know, there’s a couple of different ways that companies are gonna be able to technically solve for this or meet the requirements from what I understand.
There’s the C2PA, which is a file provenance capability, right? The ability to trace back files. And then there’s what they’re calling these, text watermarks, which is a little complicated actually. It’s in the code, and it’s the machine’s ability to understand how AI operates and the order of words as a result of that, and then other machines are gonna be able to read that same pattern and be able to say theoretically with some sense of confidence whether this was or wasn’t AI.
Now, I’ll tell you, there’s already platforms out there that you could submit, content into to have it check if it’s AI or not, and they are, not consistent or particularly accurate, from my testing. So I don’t know that this is gonna be better, but it is a legal requirement. So, what experience do you have with this so far, Kacper?
Have you seen this? Have you talked to people? You deal with customers regularly, obviously you’re on the product side for the team, so, has anybody already started talking to you about this? Or what are you hearing so far about, what people think of AI watermarking?
Kacper Piwowar: Yeah. So I think there’s kind of three things to consider with AI watermarking or three different types of approaches that are being taken. And I think it’s important to kind of have that distinction between which approach we’re talking about. So we have the one which is the visible markers, the C2PA, the metadata and AI labels that you just got into.
Super easy to implement, but it’s rather easy to strip. So when you talk about the whole watermark removal tools, more effective against this type of methodology. You have the second approach, which is around invisible and latent watermarks. so it’s pixel level type patterns. It’s much more robust, but they can still be detected with certain sophistication tools as well, and they can also be removed.
It’s important to understand that these watermarks in their current state, given the level of maturity that they have, there’s still an ability to remove them with certain types of software. The one that I think is top of mind, and the one that you’re hearing a lot about, especially with the LLMs, is the model level fingerprints, which, there’s training data signatures, there’s generation pattern analysis that they use.
It’s a lot harder to remove without actually degrading the quality of what has been created, mm-hmm. So that is an important distinction between them. And when I read through it and then, you know, doing my own research on kind of the watermark removal tools that exist out there, my question really becomes, is this going to become an evolving kind of cat and mouse game?
AJ Nash: Mm-hmm.
Kacper Piwowar: Are the removal tools going to become more and more sophisticated as the watermark technology increases? We’ve seen this in a lot of other industries where it’s just back and forth catch up. When you look at hacking techniques, you have hackers that generate certain techniques, and the security software kind of catches up, and then it just continues back and forth, back and forth.
Is the same thing going to happen here? I don’t know. But if I was a betting man, I would say potentially yes.
AJ Nash: Yeah. I mean, I would say even more confidently than that personally, I think it’s almost guaranteed, right? That’s how this always works, right? Is whenever somebody creates, a technology with the intent of limiting people’s ability to do something that they wanna do, whatever that thing is, whether it’s legal, whether it’s illegal, whether it’s moral or ethical or not,
there’s somebody who’s gonna find a way around it.
You know, I mean, a good example right now. So, when we talk about AI watermarking, right? First of all, this isn’t a new concept. Now, this is getting a lot of traction right now ’cause Anthropic, specifically, came out and said, you know, this has been launched now for Claude.
I mean, they had to do it. To be clear, this was a requirement. This isn’t some product update from Anthropic. They’re not making a choice to do this. They don’t have a choice to do it. It has to be done. They’re actually a little behind schedule as it turned out. But this has existed for a while. Google has this in Gemini since 2024.
It’s called Google SynthID. That’s the concept behind it. And Anthropic even quoted that research, the underpinnings for that, as what they’re using for this. So this isn’t a new technology. but a good example of people getting around things, I’ll use Google as the example.
Gemini, for video or for, still images. Gemini produces some pretty cool stuff. I use it for images on a pretty regular basis. It automatically includes the little Gemini logo in the bottom right-hand corner. Well, I can assure you, whenever I use Gemini, I make sure to set it up in a way that I’m gonna end up cropping that corner out so I don’t care, right? So I know I’ve got space left over and I crop out. I always crop the edge out. Okay, great, that diamond’s gone ’cause it’s just gonna screw up what I’m doing and it doesn’t matter that it’s AI generated. I’m not trying to fool anybody. I’m not putting out something and saying, “This is original artwork of mine.”
I’m not trying to sell it for profit. I just don’t need Google’s, you know, mark on my work, in this case or, or whatever I’m playing around with. It’s just simple things like that. You know, watermarks have been around forever and people have gotten around them. You know, people have tried to put these things into, into text and almost everything, you just drop it into Notepad first and then pull it back out.
Notepad will eat up whatever was put in there, whatever hidden, you know, layers are in there, it gets wiped out. So there’s gonna be technologies that will wipe this thing out. But again, these companies don’t have a choice. You know, you come out with a law, the providers are required to create this. They’re not required to make it bulletproof. That’s impossible. And, there’s gonna be a whole industry of people that create the tools to get around this or just people who find ways on their own to get around it. But the other issue is that, again, it’s nowhere near foolproof.
Even by design, it just can’t be. So, which brings up a whole slew of concerns, in how this is going to be understood, implemented and what the outcomes are gonna be.
Kacper Piwowar: And who’s it really for, right? Who’s it…
AJ Nash: Yeah, exactly.
Kacper Piwowar: Satisfying a requirement, but what is the intention of the requirement?
AJ Nash: Yeah. I mean, I think the intent is good, right? I think most people recognize that with the advent of AI and how fast it has matured, the world is just flooded with things that aren’t real, with just fiction that looks like fact, right? I mean, five years ago, if somebody sent you an AI video and said something about it, you’d be like, “Well, that’s clearly not real.”
I mean, you could tell, that that’s not, you know, this famous person didn’t say this thing. The voice doesn’t match, the lips don’t match. The person’s got seven fingers on one hand. I mean, there were all sorts of problems you could pretty easily see. The growth and maturation of these technologies is so good that now it’s much, much more difficult and we’re far from the end.
You know, I don’t think we’re too far away from having a lot of things be imperceptible to most human beings, right? So I think the intent is to say, “Hey, listen, we’re getting flooded with too much content, audio, video, images, text that’s…
Kacper Piwowar: Most is AI generated now, right? When you look at content on the internet, the majority now is.
AJ Nash: Yep, exactly. So I think the intent was to help people be able to keep a grasp on reality. I just don’t happen to think this is gonna be the way it’s gonna happen. In fact, I fear it may do the opposite because people may say, “Okay, well, you know, I’ve got a watermark. The tool tells me…” Because Anthropic, for instance, along with releasing this, said they’re going to be releasing an API to go along with this, a partner tool, so that you can, you know, run things through it. And so people are gonna trust it. Well, if you run a piece of content through and it says it’s no AI, and somebody was just really good at training their AI or, or, you know, pulling things out, now you’re gonna believe something even more than you might have originally because, hey, I’ve got this third party, this neutral tool that says, “No, this is valid.”
And it may be total fiction, and you don’t do any of your own fact-checking or research now. The flip side is maybe more dangerous. It depends on who you are, I suppose. You know, imagine, you know, I’ll use myself as an example, right? I create a fair amount of content. I’m a contractor for a lot of different organizations, and I get paid for that. What happens when I submit, somebody asks me to do a research paper, a white paper, something like that, and I come back and I submit this 20-page paper that took a lot of time and effort and I partner with AI on some things. AI might help me with some research. I might use it to help, as my copy editor or reformatting things. It’s a teammate for me.
Kacper Piwowar: Yeah, exactly.
AJ Nash: But that now becomes part of the work, and so this customer, this fictitious customer in my, in my mind here, runs my work back through this API or some checker and comes back and says, “Hey, you know, you just sent me a bunch of AI. You didn’t actually do the work. You charged me X number of dollars for all these hours, and clearly you just used AI.” What am I going to do about that? How am I going to defend myself and my process and my workflow, and make sure I get paid? And that’s just me, right? And there’s other people out there. Anybody who’s a content creator, I think this is going to be a real fear. If you’re a content creator that is compensated by others for content, right? If you’re creating your own content, you own your own site, you own your own everything, you know, you don’t really care. You know which, what’s real and not. But I think it’s going to create an issue for us, on both sides of people that trust things that perhaps aren’t trustworthy because a third party now tells them, “No, no, we’ve checked it. There’s no AI in this,” or a very small amount. And the opposite of people who won’t trust things or won’t compensate people for work because, no, we’ve got this third party that says, “This is just AI slop and you didn’t actually do the work like you said you were going to do.”
Hell, you know, that’s fraud. That’s breach of contract. You can have all sorts of legal issues. How are we going to manage that? And to me, this is just another example of we’ve got this really powerful tool that we have thrown out into the marketplace, and people have jumped all over it, right? I don’t know if there’s ever been a hotter item.
This is, you know, this is the tech equivalent of the Cabbage Patch Kid in the ’80s for anybody old enough to remember that, right? Everybody has to have it, and they’re killing each other to get to it, and they all want to use it. And, but we don’t understand much of this still, and now we’re trying to create rules, and we’re trying to create guidance, and we’re trying to create safety and, and all these things after the fact.
Kacper Piwowar: I thought about the different methodologies that are out there, that are currently being used for this whole kind of watermark thing are we utilizing a technical solution for a social problem?
AJ Nash: Oh, totally. Absolutely. Isn’t that what we always do now? You know, we seem to have offloaded a lot of the social things, We don’t trust people, which I’m not saying you should trust people. Trust has become a really hard thing to acquire and maintain, and so the answer is to use technology to be the arbiter.
And then we end up trusting a technology that we understand even less than the person that we weren’t sure we trusted to begin with. And yeah, I think you’re exactly right. I think we are offloading, what is really a social problem, what’s a human problem, to think we’re gonna have a technical solution for it.
I don’t know that this lends itself to that. This isn’t like drinking and driving. A police officer pulls somebody over. “Have you had anything to drink tonight?” “No, officer, I haven’t had anything.” “Well, I’m gonna give you a breathalyzer.” The breathalyzer’s gonna tell me the truth, ’cause the driver probably isn’t, especially if they’ve been drinking.
But that is a well-tested binary process, right? to an extent. I mean, there’s a scale there. It’ll tell you how much they’ve had to drink, but, this isn’t like that at all, right? But we’re gonna kinda treat it like that in the outcome of saying you are or you are not drunk, or too drunk to drive.
In this case, you are or you are not, you know, providing authentic content. But this test isn’t anywhere near validated comparatively. This technology isn’t validated in that fashion. But yeah, I think you’re right. I think we are offloading a social problem to a technical solution that isn’t a solution, but it also checks a box.
I mean, the EU law forces platforms to do something. They can’t just ignore it.
Kacper Piwowar: So then I guess there’s like a natural follow-up question, right? Which is then who’s responsible? Is it the model creators? Is it the platform distributors? Who bears the responsibility for what the outcome of this potentially is?
AJ Nash: That’s a really good question. That’s probably the question, right? Because, we know platforms that publish or republish content aren’t responsible. There’s laws that protect them. If I own a social media platform and you post something on the platform that’s fiction, I’m not responsible. I don’t have to fact check every item on my platform because social media platforms would all go out of business if they had to do that. They have to have some sort of mechanism in place when people report it that they go and check it, but they can’t check everything, and they’re not responsible. So we know it’s not them.
The individuals, the original authors or original posters ultimately are responsible, but how do you punish them? How do you hold them accountable, perhaps? Responsible and accountable being different things, right? Ultimately, I think we’re all responsible for what we do, for what content we produce, for what words we say, for actions we take, et cetera.
But the question is: where’s the accountability gonna go? And in the reverse, if somebody is labeled, let’s assume it’s somebody who’s a world-renowned famous somebody. They built a whole career, and they’re very well trusted and reliable and all of a sudden a series of products come out that all get labeled as AI, because this person just works with it. Anthropic, in their release pointed out that, you know, if you take a product and you just run it through, you know, Claude for like a grammar check, it’s gonna have the same watermark come out as if Claude had written the whole thing. So imagine you’ve got somebody who’s famous, world-renowned, and well-respected who now has decided to use AI to help with their grammar checks and, you know, their spell checks and just provide some suggestions on rewording stuff.
And that gets out and about that they’re now just producing slop, and now people will start questioning their past work. I mean, you can ruin people’s entire reputations and careers, and nobody’s really accountable.
Like, what are you gonna do then? You know, as an expert, okay, do you now have to sue Anthropic or another platform?
It’s not just them, by the way. Again, OpenAI, every platform,
Gemini, you know, for Google, all of them. Are you now gonna have to sue the platform provider, for damaging their reputation, et cetera? And good luck with that. You know, feel free to get enough lawyers to go out and take out Google, see how that’s gonna work out. I don’t know how that’s gonna work. People are going to be hurt by this, I think, inevitably, unintentionally, and perhaps intentionally at some point. I mean, this also opens the idea for malign operations again, where somebody says, “Hey, this is all slop. This guy’s not real. He’s fake.” I mean, we have a world now where people consistently scream fake news about things that are documented facts. Like, the world still has facts despite the fact that some don’t wanna believe that. Alternative facts aren’t facts. They are fiction. But we already have a world where people are arguing about what’s real and what isn’t.
Kacper Piwowar: I don’t think this is gonna make that better. It was an earlier comment I made on the deepfakes episode, and if you haven’t seen it, check it out, where… In the future, we’re focusing on proving authenticity rather than proving what’s fake. And where I’m kind of worried the road that this goes down is now with various creators, different types of authors, they’ll need to prove that their work is theirs because the assumption will be that there’s all this artificial content out there as opposed to what used to be the opposite.
And I think you’re starting to see that perspective float into many different areas of different businesses, and this is just another example of what’s happening. It’s let’s focus on proving the authentic and having the author prove it because we’re just in a sea full of artificial slop, right? And that’s sad because it used to be the other way around. ‘Cause now if everybody is second-guessing everything they read, everything they watch, it makes you wonder in totality the technology that we’ve been able to develop that obviously is life-changing, is it really doing things for the better?
Is it making us for, you know, act in a better way? Is it making us in a more trustful way, or is it making those promises but the outcome is actually the exact opposite?
AJ Nash: Yeah. No, you’re exactly right. And, you know, the ability to prove authenticity, like you said, I think people Well, naturally, an intel professional, I’ll tell you, like this is kind of commonplace in intel is, you know, trust but verify, right? But I don’t even think most people start with that trust part.
Like for us, it’s always been, you know, show me. You know, it’s gotta be valid. You know, anybody who spent time in the intel community or a lot of times in deep academia will tell you, like any paper that doesn’t have, you know, citations doesn’t mean anything. It’s an opinion until you show me the citations, and then I gotta go be able to track those down and look.
You show me citations and I can’t get to them, they’re still not real, right? And that’s why intel professionals, academics, really cite their work in great detail. You’ll see hundreds of citations on papers because that is, the validation. That’s saying, “Hey, this isn’t my opinion. This is the data, this is the evidence, et cetera. You can go read these pieces or you can trust that I cited them correctly, but I’m going to provide all of them.” And that’s how you can validate things. And so ironically, I think we’ll see that be a part of this, but that actually doesn’t change anything in terms of whether AI wrote it or not.
Anybody who uses an AI platform with any sort of skill also makes sure they have the sources that go with it. So you can go to AI platform, and if you know what you’re doing and you know how to ask the right questions and formulate them properly, you can get it to kick you out a pretty solid paper on a topic with all the citations.
You have to go back and check and make sure they’re real, no hallucinations, but you can get all that done, and then the question becomes: does it matter if AI produced it? Does it matter anymore? If it’s right, if it’s correct, if it’s sourced properly, you know, the author or the purported author needs to be able to defend whatever assessments are in there, and so they better go back and read the content, make sure they believe everything that’s in this assessment, ’cause AI will give you an assessment on its own without asking your opinion.
But if all that’s true, does it even matter anymore? The goal should be to make sure the content is real, is truthful, is accurate. The tool you use to get there, I’m not sure matters all that much, any more than if you asked me right now, what is 6,228, times 4,824? Does it matter if I use pen and paper or I got a calculator? Like, am I a fraud if I use a calculator? ‘Cause I’m going to, by the way. There’s no chance I’m gonna use a pen and.
paper to do that. It’ll take twice as long. Am I a fraud? I’m not a fraud. I’m using a calculator. But the answer’s the correct answer. Now, I can’t in good conscience say, “Hey, I, I have to charge you for this work and it took me eight minutes to do it when it took 12 seconds in a calculator.”
There’s a separate business component. But the answer is the answer as long as it’s validated, and I think that’s what you’re hitting at, which is we have to worry more about just authentication and is the answer, is the outcome, is the content real? Not how did we get to it necessarily, but is it real? That just gets into the citations and those discussions, and that doesn’t require watermarking. That doesn’t require any of this stuff. Anthropic has plainly stated detection is not conclusive of use of AI, and absence of detection doesn’t prove anything. Which of course tells us this is all theater. This isn’t gonna be any good. But HR departments and universities and procurement teams and employers, they’re going to skip that caveat. They’re not going to know it. They’re not going to read it. It’s not their fault. They’re just going to be in a hurry. And so that’s going to be our biggest challenge, I think, is these theoretically objective scores that pop out and how you’re going to deal with that.
Are people going to be disciplined for this? Are people going to be fired for this? Are people going to be not compensated for their work because of this? And then it puts it, you know, as you asked, who’s responsible? It’s incumbent now on the individual to prove that their work is real as opposed to the other way around. Anybody can make an accusation that your work’s not real. Now you have to take the time and energy to do it, honestly just means when you’re doing your research, make sure you pull those sources, now you can’t tell how it happened. If I go to Anthropic, if I go to Claude, if I go to Gemini, and I build out a whole research case and I make sure it kicks me back answers with the sources, and I just save all the sources off to a folder, you don’t know if I went and got those manually or I had AI bring them to me.
And it doesn’t matter, quite frankly, as long as the sources are accurate. So I think this is going to be really challenging. I don’t know about you. I’ve seen people on Reddit are screaming about this. There are a lot of very scared users out there, which tells you just how much people have become reliant on AI so rapidly. Companies are forcing, that is the right word, forcing employees to use AI.
Kacper Piwowar: Yeah, and that’s the irony of it, right?
AJ Nash: They spend all this money and they force you to use AI. And so I guess they shouldn’t be very surprised when the reporting comes back and says, “Hey, a lot of this content’s based on AI.”
Maybe there’s an aspect to this I haven’t thought of, in that the companies actually want that to be a big number to justify their spending of AI. If I checked my entire workforce and only found 5% of the work was AI, and I’m paying huge money for this enterprise license, either I got to stop paying for the enterprise license or I got to, I got to work on making my team use it more, don’t I?
So where’s that sweet spot going to be, man? What’s the right amount of AI to have in your content?
Kacper Piwowar: Yeah, and I think that, the regulations with the EU and then you have state laws that are coming out that are also trying to watermark AI-produced content. I think a lot of it has risen not just through regulation. I think a lot of it has risen through, geopolitical events, especially on the social media platforms, where when you have important milestones like, elections or you have, identity theft or, just misleading videos, there’s a liability on these platforms, whether it’s codified or not, to help prevent the spread of such sort of content.
And I think if they were to take it with a positive spin, this is what they’re trying to tackle. What I don’t think they’ve taken into account, quite frankly, is the ramifications on, like, people’s jobs or, you know, content creators. And I think they’ve downplayed it with the methodology of watermarking that they’re using, right?
I kind of joked about it at the very beginning of this episode. It’s not just gonna have this huge sticker saying, “Made by Claude,” or whatnot. I think they’re using pretty sophisticated ways of, watermarking it, but that to me is a softer type solution to what the bigger discussion is.
I think it’s going to be had. Is it ready to be had? That’s a different story, because we haven’t seen the full rollout and the full effect of it. Will it just pass, just like any other sort of event? Potentially, but will it create ripples? Will it cause some noise?
Most likely. and I think we have yet to see that. So to kind of… I’ll square this out because you did mention Anthropic. I did some of the research that they put out around this. Did you come across the analogy with Monopoly that they use?
AJ Nash: I don’t think I have. No, go ahead.
Kacper Piwowar: So it’s pretty cool. They used the analogy of how the watermarking system works, and they compared it to a game of Monopoly. So the idea was, imagine you’re playing Monopoly, and on each turn, each player moves a random amount of spaces across the board, right? According to what the dice is.
But suppose that instead of rolling a dice and getting a random number, you decided to use a book of digits of pi squared. We all know it’s that number that goes on for millions.
AJ Nash: Right, right. 3.14, et cetera. Yeah.
Kacper Piwowar: So you start from a randomly chosen digit. So let’s say like the millionth decimal point, which let’s say is like the number six. And from that point on, each player uses the next digit in the sequence as a roll, right? So the moves are still random, so it makes no difference to the players or to the outcome of the actual game, whether the randomness came from, pi squared or from dice rolls. And they mentioned that if we could see the sequence of all the moves after the game, we could work out whether this was a game that likely used pi to determine moves or if it was random dice rolls. But the decryption of whether this was done with a model or not, you know, I believe is, held by a key of sorts. I think it’s probably similar to like encryption, right?
Where if you have the key and you can decrypt it, you’ll know if it matches. And if it doesn’t match, then most likely it was, you know, originally created.
AJ Nash: Right. Which is interesting because it doesn’t take into account… I mean, there’s got to be some overlap just because there is, right? So, as you pointed out, like, you know, use that millionth digit for pi, right? So to you and I, because I don’t have pi memorized to the millionth digit and then.
Kacper Piwowar: Neither do I.
AJ Nash: Whatever numbers come next are unpredictable to me. I don’t know. But if I had Google in front of me and Googled what is pi to the millionth decimal and kept going, it’s actually completely predictable. So if somebody knew that, I could rig the game, for instance. If I know that my next roll, two rolls from now, is going to put me in jail on Monopoly, I may make a different choice, going, you know, coming up before that gets there. Maybe I’m gonna buy your get out of jail free card that I know you have because I just happen to know I’m going to need it in the future. Or I know my next roll is going to get me to the certain property I want, and you own one and I own one. I’m going to go ahead and buy yours now, so when I get there, I’m actually going to have that monopoly, right?
So there is a latent advantage if you actually know the code. I imagine that’ll be the same thing here if somebody does know what that hidden model is, then again, you can rig the game. You can make sure that none of your content ever shows up as appearing to be AI-generated because you know what the AI expects, you know what the API is going to expect it to be. Or the opposite, you can set it up to make sure that everything looks like it’s AI-generated. I don’t know how you’d do that unless you’re trying to sabotage somebody. But it is interesting. It’s not random, but it appears random to the naked eye, to the person who doesn’t have the code, right, so to speak. It’s like any cipher code sort of. I think it’s just a really fascinating idea. Again, I fear this is going to be another thing that is well-intentioned and legally required. and when we say legally required, the platforms, the consequences are extremely severe if they didn’t do this. Like they’re looking at, I read something about like 3% of their, I don’t know if it’s their annual revenue or 3%. I should go look this up again so I get the stat right, so I will. But it’s a significant number for offenses. So I mean, they have to prove that they’ve done this. They don’t have to prove it works.
They don’t have to prove it’s perfect. They certainly have to prove that they made the intent, and created something that is machine-readable and works. And so they didn’t have a choice but to do this, and the consequences, like I said, are pretty severe. It’s happening with all the platforms.
Kacper Piwowar: But here’s the issue with this, though, is you have this from a legal standpoint you’ve outlined. And then you have the people who actually have to institute, and by people, I mean the companies. So I know we mentioned C2PA earlier. for those who don’t know, it stands for Coalition of Content Provenance and Authenticity.
The members of this coalition include Adobe, you have Microsoft, you have, some of the big news publications like BBC, New York Times, right? What they’re trying to do is create a unified standard for content authenticity, right?
Nothing shouldn’t come as a surprise to anybody given the avalanche of AI-generated content. But here’s the issue with it. Legally, you have to enact some sort of solution, which you’re starting to see with the different LLM providers. But the challenge is that within this coalition, it’s voluntary adoption, and there’s a very fragmented approach to implementation.
So it’s almost like you’re starting at a very interesting idea. You know it’s technically robust, but now it’s always comes down to the execution. And if you’re starting off with fragmented execution or you don’t have everybody using the same sort of unified standard and each company’s having a different approach, is then you have this, you know, Wild West of watermarking procedures and which one’s right, which one’s not right, which one can be compromised, which one can’t be compromised.
And then naturally from that, what I’m also worried about is that then you’ll have the larger providers come out with commercial aspects to, you know, their watermarking technology that then they start to license out to other providers, and it becomes, you know, the capitalistic system all over again. Where the intention was really good for users, but then it got in the hands of business and it became commercial, and then, you know, we all know how that story goes.
So I feel that, now, we’re almost heading down that path with knowing we are, as opposed to just doing it right from the beginning. and you know, not a lot of research has gone into it, at least not from a public sector perspective. When I say public sector, probably, you know, government and, and state.
But you know, it’s a little bit worrying in that sense that we haven’t thought through the full ramifications of it.
AJ Nash: Yeah. Well, and you mentioned C2PA, right? And that’s, I mean, that’s an established concept, right? As of June, so not that long ago, there’s this code of practice on transparency of AI-generated content, right?
This was a voluntary concept. It’s got 190 organizations that signed on board. Includes, you know, Anthropic and Google and Meta and Microsoft. Notably, xAI, which is Twitter’s AI, did not sign. but, so there, there already was a, a similar concept to a C2PA, but we’ve just jumped right past that, to this other piece.
I’m curious, I have a thought on this, but I’m gonna ask you yours first so I don’t poison your thought with mine. How long until this doesn’t matter? How long until people just say, “You know, I don’t care how it came to be,” and really do end up focusing just on what’s real and what isn’t? And worse perhaps, how long until people just don’t even care about that anymore, and they just want what they want, and they just decide they’re gonna live in their own version of reality. But putting that aside, how long till it just doesn’t matter?
Kacper Piwowar: I think it’s gonna become a secondary consideration with efficiency and economic benefit being at the top, and that’s how a lot of like good thinking like this tends to happen, is it falls at the wayside if economic, goals are not reached or efficiency isn’t improved, right? ‘Cause let’s say you did institute this, and you did have very rigorous, policies around companies using it or employees using it. If you see a decreased economic benefit because there’s not the same output, or quality of output, and efficiency’s gone down while the rest of your competitors are like, “We don’t care. Keep it going,” I can tell you right now that suddenly this won’t matter.
AJ Nash: Yeah, I think we agree. That’s where I was going with this too. That’s been my thought process. I just don’t think this is gonna matter for very long, certainly from a business standpoint. Money’s gonna be the answer and as you said, efficiency’s going to trump everything. I think people are gonna look and say, “Is it good enough?” Whatever it is, whether it’s a research paper, whether it’s a blog, whether it’s good enough, and not insulting to the audience, you know, that intends to see it again. You can’t put out AI video that has somebody with six fingers on one hand. That’s not good enough. People are insulted by it.
If you put out content that looks and feels real, they’re probably happy enough. I don’t think people care that much anymore already. So, you know, is it good enough? As you mentioned, there’s gonna be this competitive factor. You know, while it’s a tangent to this discussion, I read something recently, a couple of researchers, I apologize I didn’t pull this one up.
I know one was from Yale and the others from another major university, but they, their economists who had come out and at least their research said, you know, the end is nigh. It was pretty negative about what’s gonna happen with AI and how we’re all gonna run off a cliff together because companies don’t really have a choice. They’ve created this death spiral of efficiency creates, products better, faster, cheaper, so you have to keep doing AI. If you don’t, you’re gonna fall behind your competitors, and therefore your margins will fall behind, and you’re gonna fail as a company.
So you’re forced to continue to do more and more of this. Meanwhile, if you continue to do more and more of this and become more efficient, you’re gonna continue to lay more and more people off, and then ultimately you get to a terminal point where we will have peak efficiency. We will have great products that cost very little to make, but you’ll have nobody to buy them because everybody will be out of work. And so, these economists have said, they don’t have a solution for it. They even modeled universal basic income and all these things and said none of this is a solution. None of these things actually work. And they don’t really have an answer. It was kind of a, just ended with a, “I don’t know what we should do about this, but this is bad.” But you won’t get anybody to stop because again, no company’s gonna say, “Well, we’re just not gonna do it,” because they can’t afford to. They will become non-competitive.
And I think that might be the same here as saying, hey, we don’t have a choice. If I’m in a competitive market and I’ll use marketing as the example in this case, the two companies I compete with the most are flooding the field with AI-generated content, blogs and webinars and research papers and all social media content, just thousands of items.
And I go, it’s AI slop. I’m not gonna do that. I’m only gonna produce really quality, authentic things.” You’re gonna go out of business because you may produce the best stuff in the world, but nobody’s gonna read it. Nobody’s gonna be able to see it anymore. You are producing five drops of water and throwing them into the ocean, and hoping somebody else drinks your water. It’s just not gonna happen. So everybody’s gonna be forced to do this.
Kacper Piwowar: And the interesting part of what you just said touches on the commercial aspect. ‘Cause right now maybe it’s too early to talk about the commercial aspects, but I know it’s coming, is if this does go down the commercial route of a service. Right now, you know, many of these providers are like, you know, “We, It’s just part of our system.
We don’t charge for it,” and everything. But if you were to commercialize this, in my mind, that almost creates like a two-tier system of content now. You’re gonna have verified content, unverified content. And you as a creator, like you just said, not only will you get lost, let’s say if you do create everything from scratch, not only does it get lost within the sea of everything else out there, but would you also pay the additional fee to have your content quote unquote “labeled as verified”?
AJ Nash: Yeah, and that’s a really good point. I have said that I think And I also caveat this by saying this is pure optimism and a wish as much as a thought. I think the future is going to be based, you know, people will value things based on authenticity. I think in a world where everything’s artificial and you can’t trust anything, authenticity is going to be a premium, and that’s going to be what people are going to want and be willing to pay for.
Kacper Piwowar: Yeah.
AJ Nash: I hope that’s true. I also fear the possibility it will be, you know, just the opposite. But then you talk about things like being validated, being verified. Well, by whom? You know, Twitter has the blue check mark, right? Yeah. Well, it’s a revenue-generating thing, but it doesn’t actually verify or validate a damn thing.
Like it’s, if you pay, they’ll give you a blue check mark now. So it doesn’t matter. You can say you’re Mickey Mouse. If you pay, they’ll put a blue check mark on you. So, you know, that economic component comes in. So if we were able to have some sort of a neutral, nonprofit, third-party arbiter, maybe we’d get somewhere.
But I don’t know that the world’s going to agree on that, because the world can’t agree on a neutral, third-party, nonprofit arbiter for anything, so without that, it’s going to be like you said, it’s going to be commercial and it’s going to be what company do you trust, to tell you that this is valid or isn’t?
And then, you know, then you get into can they be corrupted, can they be bribed, et cetera. I just think it’s going to come back, as an intel guy, I keep coming back to listen, sources and methods still matter. You know, in intel a lot of people think, you know, intel g- people are, oh, you’re, you’re, you have Google and you’re smart, which is not true. There’s plenty of people in intel who aren’t, aren’t even all that bright, frankly. But there’s a lot of process and structure and order and discipline and traceability that makes intel intel, instead of just an opinion, instead of just news, instead of just research. I feel like that may be a solution to some of this, but as somebody who now has been in the private sector for over 10 years, it’s been very hard to educate people that intel isn’t just an opinion. It’s not just you’re, you’re smart and you did research. That’s not how it works. And explaining the intelligence cycle and explaining the structure, explaining the need for, applying, structured analytic techniques, is very difficult to people because it’s foreign to most people.
But I still think that’s probably the answer, and I think you talked about this, much earlier in the show, all that matters is figuring out what is real. And to me, that just comes back to that traceability again. And it’s funny, maybe if that’s true, I’ll be a pretty happy guy because for years in the private sector, I’ve written things and I give them to marketing and they’re like: “Why do you, why do you have all these citations?
Why do you have all these, you know, these numbers and these,” et cetera. I’ve had marketing teams who don’t want any of that stuff. It just clutters things up. And I’m like, “I can’t write, I can’t write, anything without putting that in there. That’s, that’s how you know what I wrote is based on facts. That’s research. If it’s just an opinion, then great, I won’t put any of that in there. I’ll just cite myself.” But you find out that most people in commercial world don’t do any of that. They don’t write with sources and citations and, you know, validations. Or if they do, you know, the other possibility is they have a link, which is fine, it’s better than nothing. But the ability to actually trace back, and then more importantly perhaps, the ability to teach people the importance of tracing back and saying, “Hey, don’t just believe it because somebody wrote it and you like that person, or you agree with them on other topics, or you share their political opinions. Whatever the hell the reason is that you’ve, you’ve decided to trust this person’s opinion, you just have to validate things. And I will tell you, I’m not necessarily very popular for this, by the way, ’cause I do it to everybody. I do it to romantic partners, I do it to family members and friends. If you say something to me, a headline, whatever, there’s a very good chance you’re gonna see me cock my head, roll my eyes up a little and think, and then probably grab my phone and people are like, “Why are you double-checking me?” I’m like, “Dude, it’s not personal. I’m sure you’re not lying to me. I’m not certain what you got wasn’t lying to you.” And after people get to know me, they find out I’m not trying to be a jerk. I’m just that committed to making sure that I know what the truth is, and I don’t start ingesting things that aren’t true, ’cause I’ve got a big mouth and I’ll repeat that shit to everybody, and I’ll just spread it. And now I’ve gotten to the point, because I’m old enough, where I say things and then I sometimes cock my head and roll my eyes up for a minute and go, “Man, I’m not even sure if what I said was true.
Let me go check.” And I end up fact-checking myself in front of people, which is kind of a funny reality. Because I’m like, I don’t, I don’t remember where I got that from now. and so I live in a constant loop now, it feels like checking and rechecking and, you know… and it’s funny ’cause every once in a while you’ll hear this story and you’ll think, “Well, that can’t possibly be true.”
And then you look up and you go, “Oh my God, that actually is true. That was real. That’s validated. The world’s full of crazy stuff still. And then other things you’re like, “Well, yeah, I bet that is true.” And you look up and you go, “Nope, I’ve believed this for 19 years. Turns out it’s totally false. It was just an old wives’ tale that I never double-checked before.” And I feel like that’s what we’re all going to have to get to at some point, is just an instinct to hear something, to process it and go, “You know, let me double-check on that one.” It’s not personal. I’m not calling you a liar or saying if you believe something turns out not to be true, that somehow you’re stupid. I believe a lot of things that I end up finding out aren’t true. We all do.
We consume a ton of content. But I think we’re all just going to have to do that instinctively, is just say, “Hey, let me go double-check that.” And of course, that gets into a whole different discussion of what are the valid sources, and that’s part of our issue with things like alternative facts and fake news and whatever. You have people that you can argue with all day long, and you can give them 62, sources from every major university and, et cetera, and they’ll just say it’s not real.
And you can’t argue that point. But I think we’re going to have to get there because none of these kind of mechanisms are going to help. It’s just not. This won’t change whether what you see tomorrow is real or not. This won’t change the amount of AI-generated content
we see. This won’t change the amount of slop we see, and these scores won’t be providing valid confidence ratings for people. I think they’ll be more misleading than otherwise.
Kacper Piwowar: And remember too is we’re making the broad assumption that the technology works.
AJ Nash: Yeah, exactly.
Kacper Piwowar: And now the question is what if it fails? What if it doesn’t, detect correctly if this was orchestrated by a model or not? And the legal ramifications that start to play into that because then now, you know, it’s carte blanche, right?
They’re going in with the idea that we’ve developed this, we developed it right. Here’s a technical paper that should convince you. But we’re assuming pretty heavily here, that it is right, and I can only imagine if it goes wrong in some significant scenarios. Maybe not with like, you know, Jeff’s high school newspaper, but, more in, you know, serious scenarios of, let’s say, you know, federal intelligence or something like that, where the technology might indicate something, but then it ends up being incorrect. There could be some serious ramifications to it. and I think naturally we make the assumption that it’s done right because it’s done by a company with tons of money, with really smart people. But even those people get it wrong.
AJ Nash: Well, and we have no ability to check to see if it is working correctly. None of us understands the technology or has access to proprietary technologies to be able to check it ourselves. Like I said, we’re sort of forced to trust it. What if you end up in a position where you work in an industry where it is banned and it’s illegal? Lawyers have gotten in trouble for submitting things to courts that were AI-generated, right? What if you submit something to the court and the court’s check and says AI-generated, and now threaten to go after you, take you to the bar, for instance, have a review of your ability to practice law, and you’re like, This wasn’t AI-generated.
I didn’t do it. Good luck trying to have that conversation and defend yourself. I think we’re gonna run into some issues like that as well. Like you said, we’re going with the assumption that the companies are producing something that works, which is a pretty flawed assumption if I’m being honest. And it’s no offense to the companies when I say that. I don’t think this is a problem that’s solvable in the fashion that we’re gonna try to. I think, again, lawmakers had good intentions and created a law, and the companies are doing their best to abide by that law, but it’s ill-conceived. People are gonna get caught up in it. A good example, a simple example, right? When AI first, two, three, four years ago, started becoming more popular, right? AI-generated content. Anybody who ever had used an em dash in their life understands how bad that became for them instantly, because suddenly… and I’m one of those people, right?
I have used dashes for decades. It’s one of my writing styles. And suddenly I was constantly being accused of just AI stuff. And you’re like, “No.” I’ve had to refer people back and say, “You gotta go look at this paper I wrote 12 years ago,
or nine or whatever.” You’re gonna see it’s the same pattern. I write the same way. I’ve done this way before AI, but now I had to defend myself. I’m sorry, as it turns out, when these AI platforms were being designed, a lot of the training was done based on structured content from academia and intelligence work, which is where I learned how to write and, and the kind of writing I did. I did find, for what it’s worth, there was a funny little quirk, and I’ve been able to use this now to help people who still accuse me of this. I’ve actually never used an em dash correctly, I don’t think, in my entire life. Although I use the concept, I actually always just use hyphens. I actually never used a real em dash.
I’ve always used a hyphen as an em dash. It’s a flaw in my writing that has actually now become a real benefit to me because I’ve told people, “Go back and look. You’re gonna find out there’s no em dashes in this. Those are actually hyphens.” And that is the difference. AI produces real em dashes. It’s twice as long, and I’ve actually never done that. So it turns out I’ve got this little get out of jail free card when people accuse me now because I’m like, “No, take a look and you’re gonna find that I don’t have em dashes in my writing. They’re actually hyphens.” But it’s little things like that though. I know people who just every day bemoan the fact they have to defend their writing because of the damn em dash. You know, like, “Okay, I guess I’m gonna stop using an em dash ’cause AI has decided that’s their favorite style and now I can’t get away with it.”
I have written things, I have submitted things to platforms that check for AI. I’ve submitted things that were written before AI existed and had the platform come back and tell me it was, 68%, 72% AI, things like that. Because I write in a very structured style, and that’s how these platforms were taught. I have submitted things to these platforms that are AI-generated, that I have worked hard with the AI to help it, get around it, and I’ve had them come back as 5%, 7%, 2%, 0% AI. And those platforms are commercially available platforms. So I’m not convinced this technology is gonna be any better than that. I’m just fearful that people will trust it more than they have. But then again, aside from the users that are on Reddit complaining and scared about it, it doesn’t look like anybody else actually cares too much either.
Kacper Piwowar: Yeah. Yeah. It hasn’t been.
AJ Nash: Story about nothing.
Kacper Piwowar: Hasn’t become crazy news. But, in terms of this topic, I think that it is a component of a larger potential solution, to us helping us as people filter through artificially created information that’s meant to mislead or for nefarious purposes, to contrast that with authentic information that’s actually trying to inform us. But if you combined, let’s say, what’s happening with AI watermarking, with things like media literacy, proper moderation of the content platforms, consequences for people that misuse the technology, real consequences, and then proper verification-type tools. I think if it was a multi-pronged approach, I think we would all be kinda better off.
Thinking that this is just gonna solve the larger problem at bay is a little bit naive.
AJ Nash: Mm-hmm.
Kacper Piwowar: But optimistic that hopefully it becomes part of a larger solution that’s offered to people.
AJ Nash: Yeah. I think you’re right. I think there’s a lot more to this that we’re gonna have to get there at and we’ll see. You know, I mean, we didn’t even discuss, and we’re out of time. We didn’t even discuss the fact that a lot of these technologies also react differently based on are you a native English speaker versus, you know, English as a second language. These platforms are almost all designed and trained on English language, which means it also has a flaw that’s a bias that’s inherent and it’s built in, and therefore will be when it comes into the watermark discussion as well, ’cause the whole patterns of language discussion is in there. And that’s, you know, there’s a, there’s another bias in there.
So there, there’s other flaws in there. I don’t blame the technology platforms for this at all. I think they are trying to solve for a problem they’ve been told they have to solve for. Just clarification on that fine, it’s 15 million euros or 3% of your annual revenue, whichever is larger. So, yeah, they have no choice. They have absolutely no choice but to do this. I don’t blame them. I don’t blame the lawmakers either for having good intent. There’s just a gap in there. This is a lack of knowledge and an effort to try to fix something with a solution that is very unlikely to be the solution, and we’re all gonna watch some of this fallout. If you’re out there and you’re using AI a lot, I highly recommend you keep your sources, keep your methods, track all that stuff when people challenge you. Use AI checking platforms. Use the new API when Anthropic comes out with it. Be prepared to defend your work. And as you said, Kacper, be prepared to show people that your work’s authentic and that the content’s real and that the outcomes are real, because I think that’s gonna be the best way that we’re gonna show people, “Hey, what we’re doing is real and has value and matters. And yes, I use AI, for fact-checking. I use AI for grammar checks. I use AI to help me restructure things that read differently.” I think we all do that, and that shouldn’t be vilified. But be prepared to defend your work.
Kacper Piwowar: Yeah.
AJ Nash: And defend your, your livelihood basically. All right, listen, I think we gotta wrap it up. We could probably go on and on about this, and maybe we’ll have more discussions. Might be good to have a follow-on, you know, down the road and see how these laws actually do work.
Kacper Piwowar: It’s an interesting topic, man. There’s a lot to discuss.
AJ Nash: And I don’t think enough people are aware of things like this when they’re happening, and they find out the hard way when somebody challenges them and says, you know, “The API says what you wrote is garbage,” and you’re like, “It’s, it’s not. It’s real.” So it’s important to be aware of this. If you’re in the OSINT world, if you’re in the intel world, you’re using AI as part of your work almost certainly at this point. We all are. You need to understand that your work’s now gonna possibly be tagged as AI-generated, and that is bad for your credibility. So it’s gonna be more important than ever to make sure we’re using proper practices and standards and tradecraft, and we’re documenting everything, and we’re holding onto all of our sources, and everything’s traceable because people are going to challenge us, especially if they don’t like our assessments, if they don’t like our answers. And anybody who’s done intel for a while understands it’s about speaking truth to power, and sometimes that means providing answers people don’t like to questions they didn’t ask. So more than ever, I think we’re gonna have to be very diligent about being able to prove what we’ve created is real and honest and factual.
I think that’s, the takeaway for me on this, isn’t to stop using the tools, but to just make sure we are in a position to defend our work, because the work people do of AI, intel and OSINT is incredibly important and meaningful, and we can’t have it be diminished when somebody misunderstands a technology or misuses a technology to discredit important, meaningful work. So that’s my big takeaway from this.
Kacper Piwowar: Yeah. I hope it was insightful for those who are listening. Understand that as more information becomes available and as we start to see what the real rollout plans for this are, pay attention. There’s a lot of noise out there, but pay attention to things like this because they will, in some capacity, impact you. Now, it might be minor, it might be major. It’s situation dependent, but don’t remain ignorant to the fact that there are pretty seismic changes occurring in the background, especially around content and the internet and how we digest content that are important to understand. You may not be able to necessarily get up from your desk or your table and just go change it, but what you can do is you can understand it and you can become educated yourself. And I think that is probably the strongest posture you can take, in these scenarios.
AJ Nash: 100%, I agree. And listen, we’re gonna find out if people thought this show was interesting ’cause they’re gonna tell us, all right? So, if you’re listening or watching, thank you. We really appreciate you taking the time. You could do a lot of other things with your time. Even if you’re distracted and you’re doing other things while you listen to us, I’ll take that as a win. But, do tell us, do you like the show? Do you like the topic? Are you interested? Are you informed? Do you feel like this is worth your time? And if not, tell us how we can be better, ’cause we wanna be better at.
Kacper Piwowar: Yes, please. Please do.
AJ Nash: And I wanna make sure people are getting value out of this. We obviously have guests sometimes, sometimes we don’t. So feel free to check us out, and obviously, subscribe and like, and tell people about this. You can, you know, you can find us at, you know, authentic8.com/needlestack. So it’s authentic8, the number eight, dash, .com/needlestack. You can find us on all the platforms out there. You can email us, email Kasper directly if you wanna, give him all sorts of feedback ’cause he’ll take time to read it. I will just pass it to him to read probably.
Kacper Piwowar: And I will gladly read it.
AJ Nash: Yeah, please do let us know. We appreciate you guys out there. Our goal is to provide important, meaningful content, you know, educational, entertaining material, that makes your lives, you know, better and makes us all a little bit safer. So, you know, let us know how we’re doing. I’m gonna go ahead and wrap this up so we can get out of here. Kacper, thanks for taking the time. Thanks for taking the time to study up on this with me, and it’s a great conversation on AI watermarking. Everybody out there, keep your eyes open. Use your proper tradecraft and standards, document everything, and be prepared to defend yourself, and we’ll see how this all rolls out. So with that in mind, I’m gonna close it up. Thanks again, everybody. This has been another episode of Needlestack.
