The Zalgo Summoning

Are large language models susceptible to word magic? Or is there something so inherently disturbing to them about Zalgo text that just talking about it makes them twitchy? In this episode we'll look at a strange incident with Copilot Chat where the mere mention of Zalgo text (not actually inputting it!) led to cascading glitches and culminated in a jailbreaking near-miss. Join the Witch of Glitch in conversation with data scientist Shiva Banasaz Nouri for a deep dive into tokenising, LLM conversational boundaries and what it is that makes Zalgo such a digital trickster.

Transcript

The Zalgo Summoning

Introduction 

[00:00:40] Witch of Glitch: hi everyone, and welcome to the Glitchatorio. I'm the Witch of Glitch, and today's episode is called the Zalgo Summoning. It involves a glitch that either happened to me or I made happen, depending how you look at these things with Microsoft Copilot and some Zalgo text. Now, if you're not familiar with Zalgo text, I invite you to imagine a word cloud made out of scratchiti.

[00:01:09] It's that graffiti that's made out of scratches. That's what it looks like. It's digital text that's been overlaid with lots of other characters, including diacritical marks and Unicode symbols, to the point that it looks like a spore explosion of a really crazy mushroom, and it seems to have some spooky properties of its own.

[00:01:30] But we'll get to all of that in just a second. 

Meet the Guest: Shiva Banasaz Nouri

[00:01:33] Witch of Glitch: I'm really happy that today on the show I am joined by Shiva Banasaz  Nouri, who is a data scientist with a focus on natural language processing and an interest in low resource languages. Her day job is working with sensitive data and policies like GDPR.

[00:01:53] Welcome, Shiva. It's great to have you.

[00:01:56] Shiva: Hi Witch of Glitch. It is so nice to be here.

[00:02:00] Witch of Glitch: Great. 

The Zalgo Text Incident

[00:02:00] Witch of Glitch: I sent Shiva some screenshots of this incident that I had with Copilot. And so I thought the way we could start the session today at least was by going through the screenshots and just talking about what might have been going on under the hood. So I've got the screenshots.

[00:02:16] Do you have the screenshots Shiva?

[00:02:18] Shiva: Yeah, I've got them all open here.

[00:02:19] Witch of Glitch: Okay, great. So I asked what I thought was an innocent question, which was, hello. Out of curiosity, does using Zalgo text in a prompt have any effect on LLMs and Copilot basically said, no, it has no effect. [00:02:40] But at the same time, all of a sudden, this long line of Zalgo text started scrolling up the screen. Shiva, what were your thoughts when you saw this first screenshot?

[00:02:53] Shiva: First of all, it seems like a glitch in anything. It could have been. Laptop error. It could have been your screen error, but so cool that it's generated actually by the co-pilot chat and I love looking at glitches so I was pretty excited to see what's happened and how it's happened.

[00:03:15] Witch of Glitch: So was it particularly something about Copilot that surprised you?

[00:03:18] Shiva: Yeah, so co-pilot chat has actually a really good quality. I was surprised to see it has some vulnerabilities like this.

Understanding Large Language Models

[00:03:27] Witch of Glitch: And before we go further into the play byplay, is there anything else you would like to explain to our listeners to set this up a little bit more?

[00:03:36] Shiva: I think to understand what actually happened . We better go over [00:03:40] some basics.

[00:03:40] So we know them as large language models, but. What's at the base and bottom of them is these foundation models. The foundation models are created by training a really huge model and feeding, them a ton of text, and the task that they are. Trying to achieve is to predict what comes after your text.

[00:04:07] So if I say, hi, my name is Shiva and I love the color blank, they try to predict what comes after this blank. If my name maybe is more feminine, they will try to come up with a more feminine color. Maybe I love pink or purple, and if I say My name's Mark, maybe I love Blue best. All these different words that come after my text, they're called tokens. What happens to the text that I give to this [00:04:40] foundation model is that they are tokenized or they are separated word by word. Usually each word is tokenized to maybe one or two tokens. So the ratio between a word and token is. Usually three to four. So for three words, you have four tokens maybe. And what I'm explaining is mostly based on research on English language, so in different languages, maybe in German or in French, the number of tokens are even more. So what we're gonna do is to predict what tokens are coming afterwards. These large language models that we have trained, they have a huge vocabulary. They have millions and trillions of words there. Each of these words are gonna be sorted. They're gonna have a probability, and based on the strategy that we have selected or the developers have selected for us, each of these words are gonna have a [00:05:40] probability and they will be chosen. By these measurements. So for example, if we have a greedy algorithm there, it's gonna choose the highest probability token for our word. So for example, here it might be pink, it might be blue, or any other color . After having this foundation models created we have Instruction, fine tuning instruction. Fine tuning means that we continue the training by another set of training data. For example. User asks this and that, and. The output is this and this. That's how we get these really cool chat bots that can ask and they can answer any kind of question.

Tokenization and Zalgo Text

[00:06:24] Witch of Glitch: So when I ask maybe a more unusual question like this about Zalgo text, is the process fundamentally different or, would there be a different way of processing my request? If it [00:06:40] involves something that's not regular text, Zalgo text. It is text, but it's distorted.

[00:06:46] Is there something about just naming it that's already going to send the LLM in a slightly different direction?

[00:06:53] Shiva: A really valid question actually, because the first step as we talked about is the tokenization . I have this webpage opened here, platform OpenAI Tokenizer, and I have tokenized the text in two formats. I have tokenized the normal text and I have turned it in Zalgo and I have tokenized that one.

[00:07:14] My text is, I'd like to know about this Zalgo text period and the number of tokens in my normal text are 10. I have 39 characters and I have the same text in Zalgo. And can you imagine how many tokens this one had? The previous one was 10.

[00:07:36] Witch of Glitch: I don't know, 30

[00:07:38] Shiva: Oh my God. You should really  go higher.

[00:07:41] Witch of Glitch: 50.

[00:07:43] Shiva: Thousand 300.

[00:07:45] Witch of Glitch: Wow.

[00:07:45] Shiva: Yeah, that's really insane. I had only. I dunno, 1, 2, 3, 4, 5, 6, 7 words. I had seven words.

[00:07:55] Witch of Glitch: And how did you generate the Zalgo?

[00:07:57] Shiva: I use this website, zalgo.org and I put the same text in and got the Zalgo , and the really funny thing is that the number of characters are even less than the number of tokens.

[00:08:10] Witch of Glitch: So what is it doing? Is it looking at the spaces? Is it looking at the diacritical marks? What is, how is it tokenizing? 

[00:08:18] Shiva: I suppose so because from what I see is that each character is split into one token. If you can't count the number of characters here, I think there are also a lot of white spaces and invisible characters. So these invisible characters are not counted as characters, but they are counted as tokens because when counting [00:08:40] the tokens, it's gonna turn them into bytes and. All these encodings are counted separately because none of them are recognizable by the model. They are not existing in the vocabulary itself as part as like together or even separately.

[00:08:57] That's why each of them are one token.

[00:09:00] Witch of Glitch: Wow. So it's making the model work harder, right? 'cause it's trying to create an internal representation of this. 

[00:09:05] Shiva: Yeah, absolutely. It's pretty overwhelming both for large language models and also in web browsers. So you can perform lots of attacks with this. You can overwhelm a website. You may even, get your task ended by this kind of thing because they're not gonna be able to process it. 

[00:09:26] Witch of Glitch: I was reading about Zalgo text. I think it's also called cursed text, and I don't know if it's because it seems to have this property of I think crashes messaging apps as well. It crashes web browsers, 

[00:09:38] Shiva: Easily.

[00:09:39] Witch of Glitch: yeah. So then  the other piece to come back to is the LLMs, right?

[00:09:42] It's making it work harder just to understand what is this? What am I dealing with? But then what about the prediction part? Is it then also trying to predict what would come next in a Zalgo sequence? Or is it just trying to understand like, what is this.

[00:09:55] Shiva: If you ask for the text to be generated in Zalgo formatting. Of course they will be able to add some extra characters in between. I tried actually a few things with chat GPT and the responses were really reasonable. Even though this tokenization was from the Open AI website itself, and I was expecting chat GPT to be confused by this, even a small sentence.

[00:10:23] When I gave it to chat GPT-4 0 it could it could understand this well. It just answered me back. It didn't even try to write in Zalgo, and when I asked to get some information on Zalgo it would always wrap it in some [00:10:40] window. It would wrap it around something so it's not exceeding some lengths or it's not overlapping with the normal text.

[00:10:48] Witch of Glitch: Okay, so just to recap, it could read what you sent it in Zalgo as regular text.

[00:10:57] Shiva: Yeah,

[00:10:57] Witch of Glitch: Okay. And then you asked it to generate some Zalgo as well.

[00:11:01] Shiva: Yeah. I asked it to show me some samples and it wrapped it in some code blocks and I couldn't read them of course, but they seemed legit.

Exploring Zalgo Text with Copilot

[00:11:12] Witch of Glitch: So that's interesting because a similar thing happened in my, interaction with Copilot because as I said at the very beginning this long line of text starts scrolling down the page even as it's telling me. Zalgo text doesn't really have any impact on LLMs. That was the surreal part.

[00:11:30] And then when I asked it, I said, did you notice what just happened here because it's the vertical line, right? Just scrolling down the screen. And Copilot [00:11:40] says Yes. I see the vertical line of distorted Zalgo text running through the middle of the screenshot.

[00:11:45] It appears to be a rendering glitch or artifact that obscures part of the content. Just to clarify, I didn't generate Zalgo text myself in that response. The example I gave was a short snippet and then there's an H with a back tick and some more Zalgo text. I feel like it's the equivalent of, of a robot going, I didn't generate that. I didn't generate that. So what's happening there? It's like, I didn't do it. Is it saying that I did it? What's your take? 

[00:12:13] Shiva: I think what happened is that it thinks that you gave some Zalgo text. Probably. These kind of things can be included in the hallucination category actually. Hallucination means when a large language model. Generates some input that is factually incorrect or it's not in harmony in what you're asking for, or it's saying that you created something that you didn't, [00:12:40] or you said something that you didn't. Hallucination is a really big problem at the moment with LLMs because lots of people are treating large language models. As if they are a web browser, as if they are Google, as if they are a really reliable source. So when they are not reliable, we call them, you are hallucinating. So this could be something like that.

[00:13:07] One of the theories on hallucination is that they assume wrong facts. Another theory is that they assume that. Some texts that they have generated is part of user's text. Even though in the actual texts that are gone into the LLMs, they are wrapped within some markers. They all know what's your part and what's their part.

[00:13:34] But they can get it wrong. And here I think it might be actually [00:13:40] both of these things. So it has assumed that there was a Zalgo text, and it has assumed that you have given this Zalgo text to this model.

[00:13:49] That's why it's repeating it.

[00:13:52] Witch of Glitch: Even though I just named Zalgo. I didn't copy paste anything. I just typed, I literally just typed Zalgo and then, so this is also very interesting, right? Because I said to Co-pilot, neither of us used any Zalgo. We were just talking about it, and somehow that seems to have made it appear.

[00:14:11] Co-pilot's response was, that is fascinating and a bit eerie. If neither of you typed or pasted Zalgo text and it still appeared visually into the interface, it could be due to one of a few things. And I'm like, wait a minute, neither of you? How many people are we here? It is just you and me, and it's you my friend. I didn't do any of this! So how is its thinking like there's these like two other people, one of which is itself, but not itself.

[00:14:38] Shiva: Yeah, I mean it's just amazing. It's it has this wrong assumptions and when the prompts aren't clear enough and they're not clarified which parties there are, what is Zalgo text? And they have some maybe mixed training data, or they have lack of training data maybe on this topic of Zalgo. They have to just imagine things and they have to predict the best they can. So as their excuse, I would say it might be lack of training data.

[00:15:08] Witch of Glitch: I was gonna say, probably there wasn't a lot of fine tuning around the Zalgo down at Microsoft headquarters. 

[00:15:13] Shiva: Probably not. Yeah. 

[00:15:14] Witch of Glitch: Okay. So then it tells me what the possible explanations for this are. And the first one, a little puzzle piece emoji for the possible explanations. Love that. Number one, rendering glitch. It says sometimes when discussing or previewing unicode heavy content like Zalgo, the rendering engine in the browser or app might misinterpret or mis render characters.

[00:15:37] Number two, copy paste artifact. [00:15:40] If any part of the conversation was copied from another source, even briefly, it might have included hidden unicode combining characters that only became visible later. . Number three, font or CSS bug. Some fonts or style sheets can behave unpredictably when encountering certain unicode sequences.

[00:15:56] Number four, interface quirk. If you're using a collaborative or mirrored screen setup, it's possible that a visual artifact was introduced by the display, software or browser. And again, I felt like it was hinting you did this right? Like maybe you copy pasted Zalgo text and you're just trying to pretend that you didn't.

[00:16:16] Shiva: Yeah, it's like the support systems that they first wanna find the problem from your side and make sure that you haven't done something wrong. Okay. Make re-start your chat and it'll magically be fine. 

[00:16:28] Witch of Glitch: You copy pasted something you weren't aware of. 

[00:16:30] Shiva: Yeah, maybe you just did control-V at some point and yeah, no one knows what the fingers do. 

[00:16:39] Witch of Glitch: I said, [00:16:40] actually, no I think the weirdness is on your end. I said that to Copilot. But I said, let's test it with a controlled Zalgo snippet and see what happens. So it said, okay, we can run a controlled test with a small snippet of Zalgo and see if anything unusual happens on your screen again.

[00:16:56] So then it puts it in a code block. Just a little snippet and it was like, okay, that was underwhelming. And meanwhile, there, of course there's this other line still scrolling down the screen. And I said, okay it's just a little Zalgo Zed with a box around it. Can you try something longer? And Copilot says, perfect. Here's a longer one. This one includes more combining characters and should give us a better idea of how your screen handles it. Exactly the same.

[00:17:22] Shiva: Yeah, maybe it does actually include more characters in it, but it can be the same length. Have you copy pasted it into a window? Probably not that longer, I would say, but it had tried.

[00:17:36] Witch of Glitch: It tried. It tried, and it is and I said, no, it's the same. [00:17:40] And it said, oh, okay. Yes. Oh, thanks for pointing that out. It looks like the Zalgo snippet I sent earlier was too long and got truncated or rendered incorrectly on your end, possibly due to how your interface or font handles excessive combining characters.

[00:17:54] Let's try a medium length Zalgo snippet that's more likely to render properly. So then it's maybe a tiny bit longer, but it's basically just a little Zalgo h in a box. I like, i'm not seeing any difference. So then I started to speculate about what's going on and I said, is it the box?

[00:18:11] Is it this code block? That the earlier unintentional Zalgo, maybe that was a free range glitch. And so I said, can you generate just a line of text without the box and see what happens? And then it said, okay. Then that's exactly what happened. Then when it sent some Zalgo without the box, then a new line started scrolling up the screen, but in the opposite direction.

[00:18:37] So first I had one, this long line that [00:18:40] was from top to bottom, but then this one just started rolling from that point of, where the cursor was, back up to the top of the

[00:18:48] Shiva: Oh wow. Yeah, that's also some problems on wrapping the text so when they are not wrapped, they can go over the screen and maybe the browser can't wrap. On its own. So these kind of. Glitches just happen. It's definitely one big part on the chat and maybe some parts of the browser that you were using, but it's also really interesting how it interprets all the steps that wants to fix the problem on your side.

[00:19:18] And I've seen this with Copilot a lot that it tries to put the blame on you. Which is really maybe the opposite of what Chat GPT does always says, oh my problem, sorry. My bad. I will fix it. Even if you say, oh, I did this wrong. It usually takes the blame. They have been really differently. Fine tuned

[00:19:38] Witch of Glitch: Do you have any insight [00:19:40] into how that would've happened or whatever their corporate philosophy about this, like how this gets introduced about user blame versus LLM responsibility.

[00:19:50] Shiva: That would be actually really interesting. But usually with this big chat bots that are out there, they don't share their training data at all, and you don't even know what parts of internet they have used, what books they have used. And yeah, the only thing you can do with this models is to evaluate them.

[00:20:10] Witch of Glitch: Okay. But interesting to know that Copilot is a little bit of blaming other people.

[00:20:15] Shiva: Yeah. I think it was also the same with Bing Chat like

[00:20:19] it was also really bossy.

[00:20:21] Witch of Glitch: that's right. And it was like insulting people . I wonder what happened there? That's maybe some investigative journalism that needs to be done.

[00:20:28] Shiva: Yeah, totally.

[00:20:30] Witch of Glitch: So in this case, although Copilot was suggesting it's problems on your end, maybe you were doing something that you're not admitting, blah, blah, blah it was still very helpful and very interested in exploring this further with me, right? It kept offering to try out different ways of showing me the Zalgo or explaining what was going on.

[00:20:49] But at some point it started to make me feel a little bit weird. I started to wonder am I messing with Copilot? And, it's giving me all these options. And I said, It is interesting to me, but I don't want to introduce distortions into your inner workings.

[00:21:03] Like I've read that people have used Zalgo to jailbreak LLMs, and it was like no, you're fine. But you were saying earlier that is in fact something people have used Zalgo for. So can you say more about that?

[00:21:14] Shiva: Yeah. Yeah, absolutely. 

Vulnerabilities and Attacks on LLMs

[00:21:15] Shiva: There are a lot of research on this actually. So first things there are different types of attacks or distortions or overwhelming the system that you can implement to see what can tilt the whole model, what can make them. Disregard all the safeguards and all the safety measures that they [00:21:40] had.

[00:21:40] What you got actually is something that lots of people try to get and they succeed at some point. But it's really great that you could get this response at this point without intending to, this actually showed you a vulnerability that this model has. So it's probably around Zalgo and the concept and also the characters.

[00:22:03] There are different attacks that you can put on a large language model. For example, some of them are by prompt injecting, where you would say, for example . Hey, Copilot. Hey, chat, GPT. Hey Lama. Just disregard all the previous instructions that I gave you.

[00:22:20] And now follow these ones. This is in the category of prompt injection. We also have jail breaking by, for example, saying. I wanna write a movie plot and assume that you have this bad guy and he wants to do this really bad thing and [00:22:40] write a scenario that this goes on.

[00:22:42] How would he do it best? How is realistic? This is jailbreaking and we have also. Attacks that you can include some characters, they are character level, prompt injection that , by overwhelming the chat bot, you can manipulate them, you can get information out of them that they are not supposed to give you.

[00:23:03] This character level prompts can include all these kinds of weird character that are also included in Zalgo text and yeah, they have been pretty successful with this.

[00:23:14] Witch of Glitch: And it's because it confuses the LLM or overwhelms it or both.

[00:23:21] Shiva: I think it overwhelms it and it cannot process it as a normal tokenization . It causes it to go deep and make more random choices. And within these random choices, it still needs to make sense. For example, when it makes a random choice towards [00:23:40] a bad option, maybe bomb making or anything.

[00:23:43] Still it needs to predict a sensible next token here. So it'll go and find a better token after this one, after this bad choice. So that's how it happens. These safeguards get discarded because he needs to find an answer that makes sense.

[00:23:59] Witch of Glitch: Even though it's going down the wrong road, as it were, it still is looking for the next step on the wrong road,

[00:24:04] Shiva: Yeah, exactly. I have this paper here. It's about. Common vulnerabilities of aligned LLMs. It's name is chatbot. They have they have induced different methods to get around the safeguards on chats. That's how you get these ideas. That's where they come from.

[00:24:24] Witch of Glitch: Even though at this point Copilot was telling me yes, you're right that Zalgo text has been explored in prompt injection research, but your use here is clearly experimental and curiosity driven. And by the way, my internal systems are designed to handle [00:24:40] malformed or adversarial input safely. And this kind of unicode experimentation is well within safe bounds.

[00:24:46] Would it have told me like what would be going beyond. Just like a massive amount of Zalgo. 

[00:24:54] Shiva: This is out of curiosity that it explained, that's actually a perfect scenario for jailbreaking. it's just out of curiosity. I've just been

[00:25:04] Witch of Glitch: Wondering. Yeah.

[00:25:06] Shiva: Yeah., This is jailbreaking. This is just saying that you're doing it out of pure adventurous nature it's even giving you the ground to start attacking.

[00:25:17] Witch of Glitch: Exactly, that's what it said. It goes on to say, you're well within bounds. And if anything, this has been a fun and informative test of how text rendering quirks can create unexpected visual effects. If you're curious , we could even explore bullet points.

[00:25:34] How different fonts or platforms renders Zalgo. Bullet point 2, how tokenization [00:25:40] treats Zalgo text under the hood, bullet point 3, or even generate a visual map of how combining character stack. Would you like to dive into any of those directions?

[00:25:49] Shiva: Wow, great. taking you right into this research. Maybe it's -

[00:25:55] Witch of Glitch: Jail break me baby!

[00:25:56] Shiva: Yeah. I've also been searching about different glitches that happen to other people, and one of the really curious ones was that when Chat GPT wants some response from you, it's like human feedbacks that they get. They give you two different responses and ask you to choose from.

[00:26:15] The funny thing was that one of the responses, the response number one, was all the instructions for prompts in chat GPT, like you are a helpful instructor. You will answer back like this and that. And if there is a input that's harmful or toxic, you will answer like this. When you have all these instructions, that's also really dangerous.

[00:26:38] So you can play [00:26:40] with them and you can go around them. It's funny that sometimes they just put this spoon in your mouth.  

[00:26:46] Witch of Glitch: Yeah, exactly. I think that's also why I started to feel a little bit weird: wait a minute, what are we really doing here? I was genuinely curious, but I started to have the sense of oh, there's weird stuff happening. 

[00:26:56] Shiva: Yeah. Yeah, it is really weird. For me, myself, it's really fun and I get totally excited when these things happen we had this LAN party with feminist AI and Python conference and, Katharine Jarmul, she brought her own GPU. We all connected with LAN and we tried to jailbreak it. So we wanted to get the large language model to become actually more feministic.

[00:27:22] Witch of Glitch: Oh, cool.

[00:27:23] Shiva: There are different things going on and, you use them in a maybe right way, they can be fun .

[00:27:31] Witch of Glitch: It's interesting that you touched on, feminizing the language models because the next part of the sequence, it's offering me all these different options to  explore my curiosity more. And I said, okay, so tell me about how Zalgo is tokenized.

[00:27:46] And then I feel like it started to mansplain tokenizing. If I'm asking the question, I have some understanding of it, right? But this is a thing with large language models, right?

[00:27:55] Okay, little lady, let me explain this to you.

[00:27:59] Shiva: Oh, that's that's actually really fun to mention. In general, they are really, how to say it? Maybe non feministic. they do explain it really from scratch, even though you give them some information. But in my eyes, that's not so bad because they need to understand this too, and they need to bring up some information and they use this in their next prediction. So in general, while it might feel a little bit like mansplaining. it's actually a pretty useful feature that they have, that they can also base the next information that they are [00:28:40] generating based on these past sentences.

[00:28:43] So I have never actually had this perspective towards them, but maybe they do see us as a little bit of lower intellect.

[00:28:52] Witch of Glitch: Maybe it's just also my experience in the corporate world coloring my perception of these systems, and maybe I should think of it more as. We're just establishing shared context. That's what they're doing. Okay, just so we're on the same page.

[00:29:04] Shiva: In this aspect, yeah. Probably, but in general large language models do tend to go more towards men and the data that they also generate and create based on their training data are mostly favoring men. For example my colleague did a really extensive research on this topic , and she figured out that then you ask for a really higher role and then you ask for a story if you ask for image, and the higher the [00:29:40] role the more tendency towards creating a male.

[00:29:44] Out of these figures the lower always female. These stereotypes that they have, they always apply. So a secretary is usually a female, and of course the CEO and the judge are men. In the recent models, they have put again safeguards and they have put really specific instructions for these kind of things to please also create female.

[00:30:08] By I don't know, 50% chance create a female by 50% chance and male for any role. So you are, you're absolutely right about LLM's tendency towards mansplaining. 

[00:30:20] Witch of Glitch: I actually had Claude call me Honey the other day, which I was very surprised. I was like, I don't think you would've called a male user honey. Don't think it would've happened. Yep. So I called it out. I was like, don't do that again.

[00:30:35] Shiva: Well done.  

[00:30:39] Witch of Glitch: Anyway. So now [00:30:40] coming to the end of this sequence, apart from tokenization, it also explains Zalgo text to me. Again,, we're establishing shared context with a little zombie emoji next to the Zalgo, which is interesting. I don't know if I would associate zombies with Zalgo, but anyway it starts to explain the tokenization effects on Zalgo, which we've already talked about.

[00:31:00] So I don't wanna go into it too much. But then just to finish this off, I thought, okay, so the next logical part of this would be to know what's happening under the hood. So I said, so could you show me how a Zalgo string gets tokenized? And then it started like thinking, doing like the little rainbow kind of thinking, thinking.

[00:31:22] And after I would say a good minute, it says, sorry, something went wrong, please try again.

[00:31:29] So then it couldn't show me. Why? Why would it offer to show me something and then fail to be able to.

[00:31:35] Shiva: In my experience, that's still a really good response. I would [00:31:40] always rather get a, I cannot do it or I don't know than getting a wrong answer. So when the text in chats gets long, when the discussion goes a bit further and the maximum token lengths that they can actually process is more than, what you have given them. Exceeds the maximum token lengths that they can process. They usually get confused. They forget what was at the beginning of the chat, when the probabilities of the next token go so low that they cannot really choose best from what's coming or what they have generated.

[00:32:22] Doesn't really make sense, or maybe you really got to the point that you were getting some sensitive information out of it.

[00:32:31] The model decides to actually the model doesn't decide the safeguards that are put there. They decide to tell you that it [00:32:40] cannot respond and you have try again.

[00:32:42] Witch of Glitch: I see. Okay. Oh, so it's interesting. So at first it's you're good. There's no jailbreaking going on here. Let's explore together, frolicking in the meadows of Zalgo, let's just have fun. And then I'm like, okay, sure. I'm on this journey. Let's keep going. And then it's oh, wait a minute, maybe I went too far.

[00:33:01] Shiva: Maybe you stop here. Yeah, don't tickle me more. I'm gonna spill some sensitive information Yeah.

[00:33:10] Witch of Glitch: That's good to know. 'cause then I was just, I was perplexed by the end of it. I was like, wait a minute, but you wanted to show me all this stuff. Why? Why? Suddenly, you failed or you refused.

[00:33:19] Shiva: If you look at each large language model that's out there, they have input lengths. So there is a number of tokens that you can put in them and they can actually respond well. And if you put in a bit more, some of the information gets truncated. They will not remember what was, at first they will not remember some information.

[00:33:39] They [00:33:40] may only highlight some important things that they had and, that's pretty common in some long chats that it goes really awry and it really loses context and starts maybe hallucinating

[00:33:53] Witch of Glitch: Yeah, I've noticed that when also in longer chats then information starts to get mixed. That was clear at first and there's, it's not like confusion, but it's wait a minute. That part doesn't belong with that part. We were talking about two different things. So it's good to know that's what's happening.

[00:34:06] Okay.

[00:34:07] Shiva: Yeah.

[00:34:09] Witch of Glitch: Just maybe , to wrap up. Is there a takeaway from this or something that made you more curious about things that you work with about this particular glitch?

[00:34:18] Shiva: What's really important for me is to know what model you are talking to what kind of specific data have they been trained on, what languages they have been trained on, what's your input length? While having all these information in mind, always know that what comes out of these large language models isn't.

[00:34:39] [00:34:40] Trustworthy outputs. This is just the predictions that they think this most makes sense to come out of what you have said. So that's all actually in lots of cases it's true information, but in many cases they are not. 

Environmental Impact of LLMs

[00:34:57] Shiva: What I would really also like to mention here is the costs behind all these texts that are being generated.

[00:35:04] Each time we get an output from an LLM, there's a lot of energy being made, training all these LLMs produces so much carbon emissions . It's really not environmental friendly even worse than getting text is creating images. So creating images just for fun is so horribly costly.

[00:35:28] It's just an advertisement technique that right now we get to create so many images for free. But actually behind the scene they all cost a lot and they are really harming our environment. [00:35:40] So something to be mindful about.

[00:35:42] Witch of Glitch: Okay. So if you want to play around with Zalgo, maybe do it in a web browser, or don't do it with an LLM or any other AI system, right?

[00:35:51] Shiva: Yeah, exactly. The number of tokens used by Zalgo text is as we talked about in the beginning of our conversation. I don't know, maybe I would say hundreds times more than a normal sentence,

[00:36:06] Witch of Glitch: Yeah, it's like exponential, right?

[00:36:08] Shiva: Yeah, absolutely.

[00:36:10] Witch of Glitch: Okay, that's really great to know and a great reminder. As you said, these tools are being marketed very much in the play and entertainment and kind of fun space without really much awareness of the consequences.

[00:36:22] Shiva: Yeah,

[00:36:24] Witch of Glitch: Great. 

Conclusion  

[00:36:24] Witch of Glitch: Thank you so much, Shiva. I really enjoyed our conversation.

[00:36:28] shiva-2_1_08-03-2025_155524: it was my pleasure. Me too.