Il file robots può essere usato anche per bloccare gli user agent indesiderati, come ad esempio tutti gli spider dei motori di ricerca provenienti da altre nazioni indesiderate, questa tecnica può essere applicata se volete escludere per esempio tutti gli spider provenienti dal Giappone, dalla Korea o spider Cinesi e Russi, così da evitare bot spam e attacchi malware.
Grazie alla compilazione del file Robots.txt puoi decidere di bloccare gli user agent che vuoi escludere, esempio:
# MOTORI DI RICERCA GIAPPONESI
User-agent: moget
User-agent: ichiro
Disallow: /# MOTORI DI RICERCA KOREANI
User-agent: NaverBot
User-agent: Yeti
Disallow: /# MOTORI DI RICERCA CINESI
User-agent: Baiduspider
User-agent: Baiduspider-video
User-agent: Baiduspider-image
User-agent: sogou spider
User-agent: YoudaoBot
Disallow: /# MOTORI DI RICERCA RUSSI
User-agent: Yandex
Disallow: /
User agent lista completa
Questa Lista contiene una buona parte di user agent esistenti
User-agent: Googlebot-Image Disallow:#Disable User Agent
User-agent: A1 Sitemap Generator
Disallow: /User-agent: ABACHOBot
Disallow: /User-agent: ABCdatos BotLink
Disallow: /User-agent: Aboundexbot
Disallow: /User-agent: AboutUsBot
Disallow: /User-agent: Accoona-AI-Agent
Disallow: /User-agent: AddSugarSpiderBot
Disallow: /User-agent: adidxbot
Disallow: /User-agent: Ahoy! The Homepage Finder
Disallow: /User-agent: AITCSRobot/1.1
Disallow: /User-agent: AMZNKAssocBot
Disallow: /User-agent: Arachnophilia
Disallow: /User-agent: ArchitextSpider
Disallow: /User-agent: ASpider/0.09
Disallow: /User-agent: AURESYS/1.0
Disallow: /User-agent: AhrefsBot
Disallow: /User-agent: Alexibot
Disallow: /User-agent: Aqua_Products
Disallow: /User-agent: asterias
Disallow: /User-agent: b2w/0.1
Disallow: /User-agent: BackRub/.
Disallow: /User-agent: BackDoorBot/1.0
Disallow: /User-agent: BaiduSpider
Disallow: /User-agent: Baiduspider-image
Disallow: /User-agent: Baiduspider-news
Disallow: /User-agent: Baiduspider-video
Disallow: /User-agent: BecomeBot
Disallow: /User-agent: BeslistBot
Disallow: /User-agent: Big Brother
Disallow: /User-agent: Bizbot003
Disallow: /User-agent: Black Hole
Disallow: /User-agent: BlackWidow
Disallow: /User-agent: BLEXBot
Disallow: /User-agent: BlowFish
Disallow: /User-agent: BlowFish/1.0
Disallow: /User-agent: Bookmark search tool
Disallow: /User-agent: BotALot
Disallow: /User-agent: BotRightHere
Disallow: /User-agent: BuiltBotTough
Disallow: /User-agent: Bullseye/1.0
Disallow: /User-agent: BunnySlippers
Disallow: /User-agent: CACTVS Chemistry Spider
Disallow: /User-agent: CatchBot
Disallow: /User-agent: ccbot
Disallow: /User-agent: Cegbfeieh
Disallow: /User-agent: Checkbot/x.xx LWP/5.x
Disallow: /User-agent: CheeseBot
Disallow: /User-agent: CherryPicker
Disallow: /User-agent: CherryPickerElite/1.0
Disallow: /User-agent: CherryPickerSE/1.0
Disallow: /User-agent: coccoc
Disallow: /User-agent: ChinaClaw
Disallow: /User-agent: combine/0.0
Disallow: /User-agent: conceptbot/0.3
Disallow: /User-agent: Copernic
Disallow: /User-agent: CopyRightCheck
Disallow: /User-agent: cosmos
Disallow: /User-agent: Crescent
Disallow: /User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /User-agent: Custo
Disallow: /User-agent: CyberPatrol SiteCat Webbot
Disallow: /User-agent: CyberSpyder/2.1
Disallow: /User-agent: Daumoa
Disallow: /User-agent: Deweb/1.01
Disallow: /User-agent: DISCo
Disallow: /User-agent: DISCo Pump 3.0
Disallow: /User-agent: DISCo Pump 3.2
Disallow: /User-agent: DISCoFinder
Disallow: /User-agent: DittoSpyder
Disallow: /User-agent: dotbot
Disallow:/User-agent: Download Demon
Disallow: /User-agent: Download Demon/3.2.0.8
Disallow: /User-agent: Download Demon/3.5.0.11
Disallow: /User-agent: dumbot
Disallow: /User-agent: eCatch
Disallow: /User-agent: eCatch/3.0
Disallow: /User-agent: EirGrabber
Disallow: /User-agent: EmailCollector
Disallow: /User-agent: EmailSiphon
Disallow: /User-agent: EmailWolf
Disallow: /User-agent: EnigmaBot
Disallow: /User-agent: envolk
Disallow: /User-agent: EroCrawler
Disallow: /User-agent: es
Disallow: /User-agent: ExaBot
Disallow: /User-agent: explorersearch
Disallow: /User-agent: Express WebPictures
Disallow: /User-agent: ExtractorPro
Disallow: /User-agent: EyeNetIE
Disallow: /User-agent: FairAd Client
Disallow: /User-agent: FDSE robot
Disallow: /User-agent: FelixIDE/1.0
Disallow: /User-agent: fido/0.9 Harvest/1.4.pl2
Disallow: /User-agent: Fish-Search-Robot
Disallow: /User-agent: Flaming AttackBot
Disallow: /User-agent: FlashGet
Disallow: /User-agent: FlashGet WebWasher 3.2
Disallow: /User-agent: Freecrawl
Disallow: /User-agent: FrontPage
Disallow: /User-agent: Foobot
Disallow: /User-agent: Gaisbot
Disallow: /User-agent: Genieo
Disallow: /User-agent: GetRight/4.2
Disallow: /User-agent: GetWeb!
Disallow: /User-agent: Gigabot
Disallow: /User-agent: Girafabot
Disallow: /User-agent: Go!Zilla
Disallow: /User-agent: Golem/1.1
Disallow: /User-agent: GrabNet
Disallow: /User-agent: Grafula
Disallow: /User-agent: grapeshot
Disallow: /User-agent: Gromit/1.0
Disallow: /User-agent: gsa-crawler
Disallow: /User-agent: grub
Disallow: /User-agent: grub-client
Disallow: /User-agent: Harvest/1.5
Disallow: /User-agent: HappyFunBot
Disallow: /User-agent: Hatena Antenna
Disallow: /User-agent: Hazel’s Ferret Web hopper
Disallow: /User-agent: hloader
Disallow: /User-agent: HMView
Disallow: /User-agent: httplib
Disallow: /User-agent: HTTrack
Disallow: /User-agent: Huaweisymantecspider
Disallow: /User-agent: humanlinks
Disallow: /User-agent: ia_archiver
Disallow: /User-agent: Image Stripper
Disallow: /User-agent: Image Sucker
Disallow: /User-agent: inagist.com url crawler
Disallow: /User-agent: IncyWincy/1.0b1
Disallow: /User-agent: Indy Library
Disallow: /User-agent: InfoNaviRobot
Disallow: /User-agent: Iron33/1.0.2
Disallow: /User-agent: Informant
Disallow: /User-agent: INGRID/0.1
Disallow: /User-agent: InterGET
Disallow: /User-agent: Internet Ninja 6.0
Disallow: /User-agent: IRLbot
Disallow: /User-agent: IsraeliSearch/1.0
Disallow: /User-agent: ITI Spider
Disallow: /User-agent: JennyBot
Disallow: /User-agent: JetCar
Disallow: /User-agent: JOC Web Spider
Disallow: /User-agent: JubiiRobot
Disallow: /User-agent: jumpstation
Disallow: /User-agent: Katipo/1.0
Disallow: /User-agent: Keyword Density/0.9
Disallow: /User-agent: Kenjin Spider
Disallow: /User-agent: LabelGrab/1.1
Disallow: /User-agent: larbin
Disallow: /User-agent: LeechFTP
Disallow: /User-agent: LexiBot
Disallow: /User-agent: libWeb/clsHTTP
Disallow: /User-agent: Linguee Bot
Disallow: /User-agent: linkdexbot/2.1
Disallow: /User-agent: LinkedInBot
Disallow: /User-agent: LinkextractorPro
Disallow: /User-agent: linklooker
Disallow: /User-agent: LinkScan/8.1a Unix
Disallow: /User-agent: LinkWalker
Disallow: /User-agent: LNSpiderguy
Disallow: /User-agent: looksmart
Disallow: /User-agent: lwp-trivial/1.34
Disallow: /User-agent: lwp-trivial
Disallow: /User-agent: magpie-crawler
Disallow: /User-agent: Mass Downloader/2.2
Disallow: /User-agent: Mata Hari
Disallow: /User-agent: MediaFox/x.y
Disallow: /User-agent: MerzScope
Disallow: /User-agent: METAGOPHER
Disallow: /User-agent: MIDown tool
Disallow: /User-agent: MIIxpc
Disallow: /User-agent: MIIxpc/4.2
Disallow: /User-agent: MJ12bot
Disallow: /User-agent: Microsoft URL Control
Disallow: /User-agent: Microsoft URL Control – 5.01.4511
Disallow: /User-agent: Microsoft URL Control – 6.00.8169
Disallow: /User-agent: Mnogosearch
Disallow: /User-agent: Mister PiX
Disallow: /User-agent: moget
Disallow: /User-agent: moget/2.1
Disallow: /User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow: /User-agent: MOMspider/1.00 libwww-perl/0.40
Disallow: /User-agent: Motor/0.2
Disallow: /User-agent: MSIECrawler
Disallow: /User-agent: MSR-ISRCCrawler
Disallow: /User-agent: MSRBot
Disallow: /User-agent: NetMechanic
Disallow: /User-agent: naver
Disallow: /User-agent: NaverBot
Disallow: /User-agent: Navroad
Disallow: /User-agent: NearSite
Disallow: /User-agent: NeoScioCrawler
Disallow: /User-agent: Net Vampire/3.0
Disallow: /User-agent: NetAnts
Disallow: /User-agent: NetCarta CyberPilot Pro
Disallow: /User-agent: NetScoop/1.0 libwww/5.0a
Disallow: /User-agent: NetSpider
Disallow: /User-agent: NetZIP
Disallow: /User-agent: NHSEWalker/3.0
Disallow: /User-agent: NICErsPRO
Disallow: /User-agent: Nomad-V2.x
Disallow: /User-agent: NorthStar
Disallow: /User-agent: NPbot
Disallow: /User-agent: Nutch
Disallow: /User-agent: oBot
Disallow: /User-agent: Occam/1.0
Disallow: /User-agent: Octopus
Disallow: /User-agent: Offline Explorer
Disallow: /User-Agent: OmniExplorer_Bot
Disallow: /User-agent: OGspider
Disallow: /User-agent: Open Text Site Crawler V1.0
Disallow: /User-agent: Openbot
Disallow: /User-agent: Openfind
Disallow: /User-agent: Openfind data gathere
Disallow: /User-agent: Oracle Ultra Search
Disallow: /User-agent: PageGrabber
Disallow: /User-agent: PagePeeker
Disallow: /User-agent: Papa Foto
Disallow: /User-agent: pavuk
Disallow: /User-agent: pcBrowser
Disallow: /User-agent: PerMan
Disallow: /User-agent: PGP-KA/1.2
Disallow: /User-agent: ProPowerBot/2.14
Disallow: /User-agent: ProWebWalker
Disallow: /User-agent: psbot
Disallow: /User-agent: Python-urllib
Disallow: /User-agent: QueryN Metasearch
Disallow: /User-agent: R6_CommentReader
Disallow: /User-agent: R6_FeedFetcher
Disallow: /User-agent: Radiation Retriever 1.1
Disallow: /User-agent: RealDownload/4.0.0.42
Disallow: /User-agent: ReGet
Disallow: /User-agent: RepoMonkey
Disallow: /User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /User-agent: Resume Robot
Disallow: /User-agent: RMA
Disallow: /User-agent: Roverbot
Disallow: /User-agent: SafetyNet Robot 0.1
Disallow: /User-agent: ScoutJet
Disallow: /User-agent: searchpreview
Disallow: /User-agent: SurveyBot
Disallow: /User-agent: SiteSnagger
Disallow: /User-agent: Senrigan/xxxxxx
Disallow: /User-agent: SeznamBot
Disallow: /User-agent: SiteSnagger
Disallow: /User-agent: SlySearch
Disallow: /User-agent: SmartDownload
Disallow: /User-agent: Snooper/b97_01
Disallow: /User-agent: Solbot/1.0 LWP/5.07
Disallow: /User-agent: sootle
Disallow: /User-agent: Sosospider
Disallow: /User-agent: Sogou web spider
Disallow: /User-agent: SpankBot
Disallow: /User-agent: spanner
Disallow: /User-agent: Spanner/1.0 (Linux 2.0.27 i586)
Disallow: /User-agent: Speedy Spider
Disallow: /User-agent: spyder3.microsys.com
Disallow: /User-agent: SuperBot
Disallow: /User-agent: SuperHTTP/1.0
Disallow: /User-agent: Surfbot
Disallow: /User-agent: suzuran
Disallow: /User-agent: Szukacz/1.4
Disallow: /User-agent: tAkeOut
Disallow: /User-agent: Teleport
Disallow: /User-agent: TeleportPro
Disallow: /User-agent: Telesoft
Disallow: /User-agent: The Intraformant
Disallow: /User-agent: TheNomad
Disallow: /User-agent: TightTwatBot
Disallow: /User-agent: Titan
Disallow: /User-agent: toCrawl/UrlDispatcher
Disallow: /User-agent: True_Robot
Disallow: /User-agent: True_Robot/1.0
Disallow: /User-agent: turingos
Disallow: /User-agent: TurnitinBot
Disallow: /User-agent: Twiceler
Disallow: /User-agent: UCSD-Crawler
Disallow: /User-agent: UnisterBot
Disallow: /User-agent: UnwindFetchor/1.0
Disallow: /User-agent: URL Control
Disallow: /User-agent: URL_Spider_Pro
Disallow: /User-agent: urlck/1.2.3
Disallow: /User-agent: URLy Warning
Disallow: /User-agent: Valkyrie/1.0 libwww-perl/0.40
Disallow: /User-agent: vBSEO
Disallow: /User-agent: VCI
Disallow: /User-agent: VCI WebViewer VCI WebViewer Win32
Disallow: /User-agent: VoidEYE
Disallow: /User-agent: VoilaBot
Disallow: /User-agent: Web Image Collector
Disallow: /User-agent: Web Sucker
Disallow: /User-agent: WebAuto
Disallow: /User-agent: WebBandit
Disallow: /User-agent: WebBandit/3.50
Disallow: /User-agent: WebCopier
Disallow: /User-agent: WebEnhancer
Disallow: /User-agent: WebFerret
Disallow: /User-agent: WebFetch
Disallow: /User-agent: WebLeacher
Disallow: /User-agent: WebReaper
Disallow: /User-agent: WebmasterWorld Extractor
Disallow: /User-agent: WebmasterWorldForumBot
Disallow: /User-agent: Website Quester
Disallow: /User-agent: WebSauger
Disallow: /User-agent: Website Quester
Disallow: /User-agent: Webster Pro
Disallow: /User-agent: WebStripper
Disallow: /User-agent: WebVac
Disallow: /User-agent: webwalk
Disallow: /User-agent: WebWatch
Disallow: /User-agent: WebWhacker
Disallow: /User-agent: WebZip
Disallow: /User-agent: WebZip/4.0
Disallow: /User-agent: Wget/1.6
Disallow: /User-agent: Wget/1.5.3
Disallow: /User-agent: Wget
Disallow: /User-agent: WWW-Collector-E
Disallow: /User-agent: WhoWhere Robot
Disallow: /User-agent: Widow
Disallow: /User-agent: Xaldon WebSpider
Disallow: /User-agent: Xenu’s
Disallow: /User-agent: Xenu’s Link Sleuth 1.1c
Disallow: /User-agent: Yandex
Disallow: /User-agent: YandexBot
Disallow: /User-agent: Zeus
Disallow: /User-agent: Zeus Link Scout
Disallow: /User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /User-agent: ZyBorg
Disallow: /User-agent: 008
Disallow: /User-agent: 200PleaseBot
Disallow: /User-agent: 360Spider
Disallow: /User-agent: 4seohuntBot
Disallow: /
