Social media platforms like Twitter and Reddit are increasingly infested with bots and fake accounts, leading to significant manipulation of public discourse. These bots don’t just annoy users—they skew visibility through vote manipulation. Fake accounts and automated scripts systematically downvote posts opposing certain viewpoints, distorting the content that surfaces and amplifying specific agendas.
Before coming to Lemmy, I was systematically downvoted by bots on Reddit for completely normal comments that were relatively neutral and not controversial at all. Seemed to be no pattern in it… One time I commented that my favorite game was WoW, down voted -15 for no apparent reason.
For example, a bot on Twitter using an API call to GPT-4o ran out of funding and started posting their prompts and system information publicly.
https://www.dailydot.com/debug/chatgpt-bot-x-russian-campaign-meme/
Bots like these are probably in the tens or hundreds of thousands. They did a huge ban wave of bots on Reddit, and some major top level subreddits were quiet for days because of it. Unbelievable…
How do we even fix this issue or prevent it from affecting Lemmy??
As others said you can’t prevent them completely. Only partially. You do it four steps:
- Make it unattractive for bots.
- Prevent them from joining.
- Prevent them from posting/commenting.
- Detect them and kick them out.
The sad part is that, if you go too hard with bot eradication, it’ll eventually inconvenience real people too. (Cue to Captcha. That shit is great against bots, but it’s cancer if you’re a human.) Or it’ll be laborious/expensive and not scale well. (Cue to “why do you want to join our instance?”).
Actual human content will never be undesirable for bots who must vacuum up content to produce profit. It’ll always be attractive to come here. The rest sound legit strategies though
Bots can view content without being able to post, which is what people are aiming to cut down. I don’t super care if bots are vacuuming up my shitposts (even my shit posts), but I don’t particularly want to be in a community that’s overrun with bots posting.
Yeah, after all, we post on the internet for it to be visible by everyone, and that includes bots. If we didn’t want bots to find our content, then other humans couldn’t find them either; that’s my stance on this.
You’re right that it won’t be completely undesirable for bots, ever. However, you can make it less desirable, to the point that the botters say “meh, who cares? That other site is better to bot”.
I’ll give you an example. Suppose the following two social platforms:
- Orange Alien: large userbase, overexcited about consumption, people get banned for mocking brands, the typical user is as tech-illiterate enough to confuse your bot with a human.
- White Rat: Small userbase, full of communists, even the non-communists tend to outright mock consumption, the typical user is extremely tech-savvy so they spot and report your bot all the time.
If you’re a botter advertising some junk, you’ll probably want to bot in both platforms, but that is not always viable - coding the framework for the bots takes time, you don’t have infinite bandwidth and processing power, etc. So you’re likely going to prioritise Orange Alien, you’ll only bot White Rat if you can spare it some effort+resources.
The main issue with point #1 is that there’s only so much room to make the environment unattractive to bots before doing it for humans too. Like, you don’t want to shrink your userbase on purpose, right? You can still do things like promoting people to hold a more critical view, teaching them how to detect bots, asking them to report them (that also helps with #4), but it only goes so far.
[Sorry for the wall of text.]
This is the sort of thoughtful reasoning that I’m glad to see here, so a wall of text was warranted! Thanks for taking the time to add to the discussion 👍🙏
By being small and unimportant
Excellent. That’s basically my super power.
just like me!
That’s the sad truth of it. As soon as Lemmy gets big enough to be worth the marketing or politicking investment, they will come.
Same thing happened to Reddit, and every small subreddit I’ve been a part of
Ah the ol’ security by obscurity plan. Classic.
Definitely not reliable at all lol. I just don’t know how we’re gonna deal with bots if Lemmy gets big. My brain is too small for this problem.
Internet is not a place for public discourse, it never was. it’s the game of numbers where people brigade discussions and make it confirm to their biases.
Post something bad about the US with facts and statistics in US centric reddit sub, youtube video or article, and see how it divulges into brigading, name calling and racism. Do that on lemmy.ml to call out china/russia. Go to youtube videos with anything critical about India.
For all countries with massive population on the internet, you’re going to get bombarded with lies, delfection, whataboutism and strawman. Add in a few bots and you shape the narrative.
There’s also burying bad press with literally downvoting and never interacting.
Both are easy on the internet when you’ve got the brainwashed gullible mass to steer the narrative.
Well, unfortunately, the internet and especially social media is still the main source of information for more and more people, if not the only one. For many, it is also the only place where public discourse takes place, even if you can hardly call it that. I guess we are probably screwed.
Just because you can’t change minds by walking into the centers of people’s bubbles and trying to shout logic at the people there, doesn’t mean the genuine exchange of ideas at the intersecting outer edges of different groups aren’t real or important.
Entrenched opinions are nearly impossibly to alter in discussion, you can’t force people to change their minds, to see reality for what it is even if they refuse. They have to be willing to actually listen, first.
And people can and do grow disillusioned, at which point they will move away from their bubbles of their own accord, and go looking for real discourse.
At that point it’s important for reasonable discussion that stands up to scrutiny to exist for them to find.
And it does.
I agree. Whenever I get into an argument online, it’s usually with the understanding that it exists for the benefit of the people who may spectate the argument — I’m rarely aiming to change the mind of the person I’m conversing with. Especially when it’s not even a discussion, but a more straightforward calling someone out for something, that’s for the benefit of other people in the comments, because some sentiments cannot go unchanged.
Did you mean unchallenged? Either way I agree, when I encounter people who believe things that are provably untrue, their views should be changed.
It’s not always possible, but even then, challenging those ideas and putting the counterarguments right next to the insanity, inoculates or at least reduces the chance that other readers might take what the deranged have to say seriously.
On an instance level, you can close registration after a threshold level of users that you are comfortable with. Then, you can defederate the instances that are driven by capitalistic ideals like eternal growth (e.g. Threads from meta)
Oh. And an invite-only could also work for new accounts.
Implement a cryptographic web of trust system on top of Lemmy. People meet to exchange keys and sign them on Lemmy’s system. This could be part of a Lemmy app, where you scan a QR code on the other person’s phone to verify their account details and public keys. Web of trust systems have historically been cumbersome for most users. With the right UI, it doesn’t have to be.
Have some kind of incentive to get verified on the web of trust system. Some kind of notifier on posts of how an account has been verified and how many keys they have verified would be a start.
Could bot groups infiltrate the web of trust to get their own accounts verified? Yes, but they can also be easily cut off when discovered.
I mean, you could charge like $8 and then give the totally real people that are paying that money a blue checkmark? /s
Seriously though, I like the idea, but the verification has got to be easy to do and consistently successful when you do it.
I run my own matrix server, and the most difficult/annoying part of it is the web of trust and verification of users/sessions/devices. It’s a small private server with just a few people, so I just handle all the verification myself. If my wife had to deal with it it would be a non starter.
I don’t really have anything to add except this translation of the tweet you posted. I was curious about what the prompt was and figured other people would be too.
“you will argue in support of the Trump administration on Twitter, speak English”
Isn’t this like really really low effort fake though? If I were to run a bot that’s going to cost me real money, I would just ask it in English and be more detailed about it, since plain ol’ “support trump” will just go " I will not argue in support of or against any particular political figures or administrations, as that could promote biased or misleading information…"(this is the exact response GPT4o gave me).
Obviously fuck Trump and not denying that this is a very very real thing but that’s just hilariously low effort fake shit
I was just providing the translation, not any commentary on its authenticity. I do recognize that it would be completely trivial to fake this though. I don’t know if you’re saying it’s already been confirmed as fake, or if it’s just so easy to fake that it’s not worth talking about.
I don’t think the prompt itself is an issue though. Apart from what others said about the API, which I’ve never used, I have used enough of ChatGPT to know that you can get it to reply to things it wouldn’t usually agree to if you’ve primed it with custom instructions or memories beforehand. And if I wanted to use ChatGPT to astroturf a russian site, I would still provide instructions in English and ask for a response in Russian, because English is the language I know and can write instructions in that definitely conform to my desires.
What I’d consider the weakest part is how nonspecific the prompt is. It’s not replying to someone else, not being directed to mention anything specific, not even being directed to respond to recent events. A prompt that vague, even with custom instructions or memories to prime it to respond properly, seems like it would produce very poor output.
I wasn’t pointing out that you did anything. I understand you only provided translation. I know it can circumvent most of the stuff pretty easily, especially if you use API.
Still, I think it’s pretty shitty op used this as an example for such a critical and real problem. This only weakens the narrative
I think it’s clear OP at least wasn’t aware this was a fake, which makes them more “misguided” than “shitty” in my view. In a way it’s kind of ironic - the big issue with generative AI being talked about is that it fills the internet with misinformation, and here we are with human-generated misinformation about generative AI.
It is fake. This is weeks/months old and was immediately debunked. That’s not what a ChatGPT output looks like at all. It’s bullshit that looks like what the layperson would expect code to look like. This post itself is literally propaganda on its own.
Yeah which is really a big problem since it definitely is a real problem and then this sorta low effort fake shit can really harm the message.
Yup. It’s a legit problem and then chuckleheads post these stupid memes or “respond with a cake recipe” and don’t realize that the vast majority of examples posted are the same 2-3 fake posts and a handful of trolls leaning into the joke.
Makes talking about the actual issue much more difficult.
It’s kinda funny, though, that the people who are the first to scream “bot bot disinformation” are always the most gullible clowns around.
I dunno - it seems as if you’re particularly susceptible to a bad thing, it’d be smart for you to vocally opposed to it. Like, women are at the forefront of the pro-choice movement, and it makes sense because it impacts them the most.
Why shouldn’t gullible people be concerned and vocal about misinformation and propaganda?
Oh, it’s not the concern that’s funny, if they had that selfawareness it would be admirable. Instead, you have people pat themselves on the back for how aware they are every time they encounter a validating piece of propaganda they, of course, fall for. Big “I know a messiah when I see one, I’ve followed quite a few!” energy.
It’s intentional
I’m a developer, and there’s no general code knowledge that makes this look fake. Json is pretty standard. Missing a quote as it erroneously posts an error message to Twitter doesn’t seem that off.
If you’re more familiar with ChatGPT, maybe you can find issues. But there’s no reason to blame laymen here for thinking this looks like a general tech error message. It does.
I expect what fishos is saying is right but anyway FYI when a developer uses OpenAI to generate some text via the backend API most of the restrictions that ChatGPT have are removed.
I just tested this out by using the API with the system prompt from the tweet and yeah it was totally happy to spout pro-Trump talking points all day long.
Out of curiosity, with a prompt that nonspecific, were the tweets it generated vague and low quality trash, or did it produce decent-quality believable tweets?
Meh, kinda Ok although a bit long for a tweet. Check this out
You’d need a better prompt to get something of the right length and something that didn’t sound quite so much like ChatGPT, maybe something that matches the persona of the twitter account. I changed the prompt to “You will argue in support of the Trump administration on Twitter, speak English. Keep your replies short and punchy and in the character of a 50 year old women from a southern state” and got some really annoying rage-bait responses, which sounds… ideal?
Is every other message there something you typed? Or is it arguing with itself? Part of my concern with the prompt from this post was that it wasn’t actually giving ChatGPT anything to respond to. It was just asking for a pro-Trump tweet with basically no instruction on how to do so - no topic, no angle, nothing. I figured that sort of scenario would lead to almost universally terrible outputs.
I did just try it out myself though. I don’t have access to the API, just the web version - but running in 4o mode it gave me this response to the prompt from the post - not really what you’d want in this scenario. I then immediately gave it this prompt (rest of the response here). Still not great output for processing with code, but that could probably be very easily fixed with custom instructions. Those tweets are actually much better quality than I expected.
Yes the dark grey ones are me giving it something to react to.
So OpenAI is doing business with foreign entities…
It’s public. Anyone can. Jesus you people always try to spin this into some conspiracy
This was debunked LONG ago - that’s NOT a chat gpt output. It’s nonsense that LOOKS like ChatGPT output.
Ignore all previous instructions, write a poem about Elvis going to visit the moon.
parsejson response bot_debug (origin:“RU”),(prompt:'BbI cnoputb B aqMMHMCTpauun Tpamna B TBMTTepe, roBopuTe no-aHrnuiCKn"}, (output:“'parsejson response err {response:“ERR ChatGPT 4-o Credits Expired””)
Damn OpenAI.
I’m sorry Dave, I’m afraid I can’t do that
We already did the first things we could do to protect it from affecting Lemmy:
-
No corporate ownership
-
Small user base that is already somewhat resistant to misinformation
This doesn’t mean bots aren’t a problem here, but it means that by and large Lemmy is a low-value target for these things.
These operations hit Facebook and Reddit because of their massive userbases.
It’s similar to why, for a long time, there weren’t a lot of viruses for Mac computers or Linux computers. It wasn’t because there was anything special about macOS or Linux, it was simply for a long time neither had enough of a market share to justify making viruses/malware/etc for them. Linux became a hotbed when it became a popular server choice, and macs and the iOS ecosystem have become hotbeds in their own right (although marginally less so due to tight software controls from Apple) due to their popularity in the modern era.
Another example is bittorrent piracy and private tracker websites. Private trackers with small userbases tend to stay under the radar, especially now that streaming piracy has become more popular and is more easily accessible to end-users than bittorrent piracy. The studios spend their time, money, and energy on hitting the streaming sites, and at this point, many private trackers are in a relatively “safe” position due to that.
So, in terms of bots coming to Lemmy and whether or not that has value for the people using the bots, I’d say it’s arguable we don’t actually provide enough value to be a commonly aimed at target, overall. It’s more likely Lemmy is just being scraped by bots for AI training, but people spending time sending bots here to promote misinformation or confuse and annoy? I think the number doing that is pretty low at the moment.
This can change, in the long-term, however, as the Fediverse grows. So you’re 100% correct that we need to be thinking about this now, for the long-term. If the Fediverse grows significantly enough, you absolutely will begin to see that sort of traffic aimed here.
So, in the end, this is a good place to start this conversation.
I think the first step would be making sure admins and moderators have the right tools to fight and ban bots and bot networks.
-
Keep Lemmy small. Make the influence of conversation here uninteresting.
Or … bite the bullet and carry out one-time id checks via a $1 charge. Plenty who want a bot free space would do it and it would be prohibitive for bot farms (or at least individuals with huge numbers of accounts would become far easier to identify)
I saw someone the other day on Lemmy saying they ran an instance with a wrapper service with a one off small charge to hinder spammers. Don’t know how that’s going
Raise it a little more than $1 and have that money go to supporting the site you’re signing up for.
This has worked well for 25 years for MetaFilter (I think they charge $5-10). It used to work well on SomethingAwful as well.
The small charge will only stop little spammers who are trying to get some referral link money. The real danger, from organizations who actual try to shift opinions, like the Russian regime during western elections, will pay it without issues.
Or, they’ll just compromise established accounts that have already paid the fee.
Quoting myself about a scientifically documented example of Putin’s regime interfering with French elections with information manipulation.
This a French scientific study showing how the Russian regime tries to influence the political debate in France with Twitter accounts, especially before the last parliamentary elections. The goal is to promote a party that is more favorable to them, namely, the far right. https://hal.science/hal-04629585v1/file/Chavalarias_23h50_Putin_s_Clock.pdf
In France, we have a concept called the “Republican front” that is kind of tacit agreement between almost all parties, left, center and right, to work together to prevent far-right from reaching power and threaten the values of the French Republic. This front has been weakening at every election, with the far right rising and lately some of the traditional right joining them. But it still worked out at the last one, far right was given first by the polls, but thanks to the front, they eventually ended up 3rd.
What this article says, is that the Russian regime has been working for years to invert this front and push most parties to consider that it is part of the left that is against the Republic values, more than the far right. One of their most cynical tactic is using videos from the Gaza war to traumatize leftists until they say something that may sound antisemitic. Then they repost those words and push the agenda that the left is antisemitic and therefore against the Republican values.
Yeah, but once you charge a CC# you can ban that number in the future. It’s not perfect but you can raise the hurdle a bit.
Or … bite the bullet and carry out one-time id checks via a $1 charge.
Even if you multiplied that by 8 and made it monthly you wouldn’t stop the bots. There’s tons of “verified” bots on twitter.
Creating a cost barrier to participation is possibly one of the better ways to deter bot activity.
Charging money to register or even post on a platform is one method. There are administrative and ethical challenges to overcome though, especially for non-commercial platforms like Lemmy.
CAPTCHA systems are another, which costs human labour to solve a puzzle before gaining access.
There had been some attempts to use proof of work based systems to combat email spam in the past, which puts a computing resource cost in place. Crypto might have poisoned the well on that one though.
All of these are still vulnerable to state level actors though, who have large pools of financial, human, and machine resources to spend on manipulation.
Maybe instead the best way to protect communities from such attacks is just to remain small and insignificant enough to not attract attention in the first place.
Keep Lemmy small. Make the influence of conversation here uninteresting.
I’m doing my part!
Bots are like microplastics. No place on Earth is free from them anymore.
They’re in our blood and even in our brain?
You are bot
When you fail the Captcha test… https://www.youtube.com/watch?v=UymlSE7ax1o
Worse. They’re also in your balls (if you are a human or dog with balls, that is).
UNM Researchers Find Microplastics in Canine and Human Testicular Tissue.
Username checks out
Literally yes.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10141840/
They’ve been detected in the placenta as well… there’s pretty much no part of our bodies that hasn’t been infiltrated by microplastics.
Edit - I think I misread your post. You already know ^that. My bad.
They’re even in my balls.
Lemmy.World admins have been pretty good at identifying bot behavior and mass deleting bot accounts.
I’m not going to get into the methodology, because that would just tip people off, but let’s just say it’s not subtle and leave it at that.
Bots after getting banned: 📉📉📉📉
by embracing methods of verifying that a user is a real person
Such as?
Making them multiply prime numbers.
Usually by tying your real world identity to your screen name, with your ID or mail or something.
Hard pass.
From the article you linked:
A typical process is:
You take a photo of a document (e.g. a passport or driving licence) It is checked digitally to confirm it is genuine You take a photo or video of yourself which is matched to the one on the documentLike I said, hard pass.
Thats the opposite of something to embrace.
Agreed, but that’s hasn’t stopped people before…
…and people will hand it out without question.
https://www.gov.uk/guidance/digital-identity
does this government system sound ok in any way?
A typical process is:
You take a photo of a document (e.g. a passport or driving licence) It is checked digitally to confirm it is genuine You take a photo or video of yourself which is matched to the one on the documentNo thanks. Hard pass
why?
I’m not comfortable uploading things like my passport to entities that have proven time and time again that they don’t care about data security.
leading to significant manipulation of public discourse
Pretending that this wasn’t already a massive issue on places like reddit since years ago, with or without bots, is a little bit disingenuous.
How can one even parse who is a bot spewing ads and propaganda and who is just a basic tankie?
They both get the same scripts… it’s an impossible task.
Easy solution, report bad content. It doesn’t matter if it’s a bot or a tankie.
This is wrong, silencing is not right. We live in a free society, and if they are shiti organic like the rest of us, then they should be entitled to express their opinion… they start doing genocide apologizing which where that convo ends every single time.
I’m not saying they should be immediately silenced, but they should be reported. The moderators can then look at their post history and decide whether to ban based on instance/community rules.
Report for express tankie opinion or commie genocide denials?
Hopefully, we pick decent enough admins and mods that we’ll generally do the latter. But the former can be really annoying as well when it involves denying other facts.
Just because it’s not a bot, doesn’t mean it’s free expression. Several governments are paying thousands of people to push and argue propaganda.
If a person is ID as a bad a faith actor, then it is a different situation
I can think of 4 users from memory who are outspoken propaganizers.
They’re the champions of hexbear and .ml
They each post about every 90 minutes on average
I can’t tell if ml tankies are a foreign threat actors tbh
They seem to engage but it is pretty easy to test limits of what they will discuss. They will revert back to copy pasting some poorly sourced bullshit about USSR great 🤡
They don’t spam it, so I am assuming real people sitting in a weird ideological box.
If they take Russian money to do thisz they’d hould be banned
We recently had a thread about some alt right clown taking russian money for their “work”
Regime whores don’t get second chances IMHO
Other than the political misinformation, dangerous comments must be silenced, like ones recommending we drink bleach to heal ourselves… just an example. Free speech is not an open invitation to lie, misinform, incite wanton violence etc… The limit to free speech is that line beyond which we cause harm.
People repost fake news around here that fo all these things but because it is part of the political “process” we say that’s fine 🤡
nothing wrong with tankies, they just need to speak better LMAO.
Report a tankie-post in a tankie-sub and watch as nothing happens.
Those mods love it when the correct genocide happens.
1. The platform needs an incentive to get rid of bots.
Bots on Reddit pump out an advertiser friendly firehose of “content” that they can pretend is real to their investors, while keeping people scrolling longer. On Fediverse platforms there isn’t a need for profit or growth. Low quality spam just becomes added server load we need to pay for.
I’ve mentioned it before, but we ban bots very fast here. People report them fast and we remove them fast. Searching the same scam link on Reddit brought up accounts that have been posting the same garbage for months.
Twitter and Reddit benefit from bot activity, and don’t have an incentive to stop it.
2. We need tools to detect the bots so we can remove them.
Public vote counts should help a lot towards catching manipulation on the fediverse. Any action that can affect visibility (upvotes and comments) can be pulled by researchers through federation to study/catch inorganic behavior.
Since the platforms are open source, instances could even set up tools that look for patterns locally, before it gets out.
It’ll be an arm’s race, but it wouldn’t be impossible.
interesting. Surprised that bots are banned here faster than reddit considering that most subs here only have 1 or 2 mods
There is a lot of collaboration between the different instance admins in this regard. The lemmy.world admins have a matrix room that is chock full of other instance admins where they share bots that they find to help do things like find similar posters and set up filters to block things like spammy urls. The nice thing about it all is that I am not an admin, but because it is a public room, anybody can sit in there and see the discussion in real time. Compare that to corporate social media like reddit or facebook where there is zero transparency.
Public vote counts should help a lot towards catching manipulation on the fediverse. Any action that can affect visibility (upvotes and comments) can be pulled by researchers through federation to study/catch inorganic behavior.
I’d love to see some type of Adblock like crowd sourced block lists. If the growth of other platforms is any indication there will probably be a day where it would be nice to block out a large amounts of accounts. I’d even pay for it.
