Given a sufficient amount of text, the method is said to be 99.9 percent effective.
If that’s really the case, they should release some benchmarks. I am skeptical. Promising the world is a key component of their “business model”.
I don’t think these grifters know what a benchmark is.
I think given enough output I could probably detect it that accurately as well. ChatGPT has a particular voice and the longer it goes, the more that voice comes out.
What is a sufficient amount? Most comments are short af.
“A 99.9% accurate ChatGPT AI text detector? At this time of year! At this time of day! In this part of the country! Localized entirely within your company?!?”
“Yes”
"May I see it?“
“No”
The detection rate is worthless, an algorithm that says anything is Chatgpt would have a detection rate of 100%. What would be more interesting than that is the false positive rate but they never talk about that.
The detector provides an assessment of how likely it is that all or part of the document was written by ChatGPT. Given a sufficient amount of text, the method is said to be 99.9 percent effective.
That means given 100 pieces of text and asked if they are made by ChatGPT or not, it gets maybe one of them wrong. Allegedly, that is, and with the caveat of “sufficient amount of text”, whatever that means.
It’s actually 1 in 1000, 99.0% would be 1/100.
A false positive is when it incorrectly determines that a human written text is written by AI. While a detection rate of 99.9% sounds impressive, it’s not very reliable if it comes with a false positive rate of 20%.
I know what a false positive is, and it’s not a thing when talking about effectiveness, they claim it gets it right 99.9% of the time.
Right, I see what you mean now. I misread your comment as explaining something that was already clear.
it’s only 99.9% accurate because they haven’t released it. As soon as they do, it will quickly fall to 50% as usual. Because this type of thing is exactly what’s needed to develop tech to defeat itself.
What?
Once you have an AI detector, you can use it’s results to train your AI to pass the detector.
If they aren’t willing to release it, then the situation is no different from them not having one at all. All these claims openai makes about having whatever system but hiding it, is just tobtry and increase hype to grab more investor money.
Total coincidence that this “news” appears about a day after several articles saying the AI bubble is starting to burst.
It is nut. Who is paying for all these articles and why are they hell bent on convincing everyone that AI is to the left like immigrants are to Republicans
Lots of money in the AI hype game, as tech stocks are massively inflated from just this year alone.
Language models in the end, are just statistics. And to make statistics more accurate, you need more data. Exponentially more data. At the same time, the marginal utility of precision decays exponentially. Exponentially increasing marginal costs are met with exponentially decaying marginal utility.
Why does everything have to be about the USA these days? I’m tired of this joke of a wannabe democracy. Don’t want to hear it. Nobody cares. Just stop and leave it to yourself.
I call bullshit.
If you believe this, I have a bridge in Brooklyn to sell you
A routine that just returns “yes” will also detect all AI. It would just have an abnormally high false positive rate.
My model has 100% recall and 50% precision, not bad eh?
But - that model would not have 99.9% accuracy.
Agreed. Personally I think this whole thing is bs.
Ofc they just look in their database if this is something it has ever said and to who
ALL conversations are logged and can be used however they want.
I’m almost certain this “detector” is a simple lookup in their database.
Probably because it doesn’t work. It’s not difficult for Open AI to see if any given conversation is one of their conversations. If I were them I would hash the results of each conversation and then store that hash in a database for quick searching.
That’s useless for actual AI detection
She goes to another school
(for intelligent ificial art)The detector is most likely a machine learning algorithm. That said, releasing that would allow for adversarial training. (An LLM that would not be detected). Therefore they can only offer maybe an api to use it but can not give unlimited access to the model.
This is the reason. Releasing it would invalidate it.
If u release an api for it u can still use that to make training data to beat it.
That’s what the Chinese tried with chatgpt. Didn’t go well.
Huh? Use chatgpt to generate training data to train another ai? Thats pretry common actually I believe even mistral does that hence why u need somthing like dolphin to remove the alignment by openai.
They’re keeping everything anyway, so what’s preventing them from doing a DB look up to see if it (given a large enough passage of text) exist in their output history?
I believe the actual detector is similar. They know what sentences are likely generated by chatgpt, since that’s literally in their model. They probably also have to some degree reverse engineered typical output from competing models.
Doubt
There is no way it’s that accurate, which is why they don’t want to release it.
