gedaliyah@lemmy.world to Technology@lemmy.worldEnglish · 2 months agoReddit blocking all major search engines, except Googlereadwrite.comexternal-linkmessage-square50fedilinkarrow-up118arrow-down10
arrow-up118arrow-down1external-linkReddit blocking all major search engines, except Googlereadwrite.comgedaliyah@lemmy.world to Technology@lemmy.worldEnglish · 2 months agomessage-square50fedilink
minus-squareleopold@lemmy.kde.sociallinkfedilinkEnglisharrow-up1·2 months agothis is just going to cause indexers to ignore robots.txt
minus-squaregedaliyah@lemmy.worldOPlinkfedilinkEnglisharrow-up1·2 months ago“We always obey the robots.txt” A bunch of corporations that have no accountability and plenty of incentive to just ignore it and have all been caught training AI on off-limits data.
minus-squareKairos@lemmy.todaylinkfedilinkEnglisharrow-up0arrow-down1·2 months agoThey’re likely blocking user agents too, which I think also doesn’t have legal enforcement (as in DuckDuckGo can just use “Google” unless they said otherwise.
this is just going to cause indexers to ignore robots.txt
“We always obey the robots.txt”
They’re likely blocking user agents too, which I think also doesn’t have legal enforcement (as in DuckDuckGo can just use “Google” unless they said otherwise.