User-agent: msnbot Crawl-delay: 100 Disallow: / User-agent: psbot Crawl-delay: 100 Disallow: / User-agent: msnbot/1.0 Crawl-delay: 100 Disallow: / User-agent: msnbot* Crawl-delay: 100 Disallow: / User-agent: msnbot-media/1.0 Crawl-delay: 100 Disallow: / User-agent: Googlebot-Image Disallow: / User-agent: Googlebot/2.1 Disallow: / User-agent: Moreoverbot/5.00 Disallow: / User-agent: VoilaBot BETA 1.2 Disallow: / # # Added from http://gallery.menalto.com/node/8329#comment-29818 # User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: ArribaPacketRat Disallow: / User-agent: autoemailspider Disallow: / User-agent: baiduspider Disallow: / User-agent: Bilbo Disallow: / User-agent: DigExt Disallow: / User-agent: dloader(NaverRobot)/1.0 Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DTS Agent Disallow: / User-agent: DotBot* Disallow: / User-agent: *Feedfetcher-Google* Disallow: / User-agent: Feedfetcher-Google Disallow: / User-agent: fast Disallow: / User-agent: Getleft 1.1b2 Disallow: / User-agent: girafa Disallow: / User-agent: Gigabot/1.0 Disallow: / User-agent: googlebot-Image Disallow: / User-agent: grub-client Disallow: / User-agent: HTMLAB Disallow: / User-agent: httrack Disallow: / User-agent: ia_archiver Disallow: / User-agent: ImageVampire Disallow: / User-agent: k2spider Disallow: / User-Agent: Krugle Disallow: / User-agent: mail sweeper Disallow: / User-agent: Mozilla/5.0* Disallow: / User-agent: MSIECrawler Disallow: / User-agent: NetCaptor Disallow: / User-agent: NewsGatorOnline* Disallow: / User-agent: NITLE Blog Spider/0.01 Disallow: / User-agent: NPBot Disallow: / User-agent: nutch Disallow: / User-agent: oBot Disallow: / User-agent: Offline Explorer Disallow: / User-agent: PSBot Disallow: / User-agent: QuepasaCreep v0.9.13 Disallow: / User-agent: Scooter/3.3 Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: Slurp/2.0 Disallow: / User-agent: Speedy Spider* Disallow: / User-agent: SpiderKU/0.9 Disallow: / User-agent: Steeler Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: szukacz Disallow: / User-agent: Szukacz/1.5 Disallow: / User-agent: TurnitinBot Disallow: / User-agent: Twiceler* Disallow: / User-agent: Vagabondo/2.1 Disallow: / User-agent: vischeck_spiderBot/0.1libwww-perl/5.48 Disallow: / User-agent: vscooter Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier Disallow: / User-agent: webcrawler Disallow: / User-agent: Web Downloader/4.9 Disallow: / User-agent: Web Downloader/5.8 Disallow: / User-agent: WebGather 3.0 Disallow: / User-agent: WebStripper/2.56 Disallow: / User-agent: WebZIP/3.65 Disallow: / User-agent: WebZIP Disallow: / User-agent: Wget Disallow: / User-agent: Zao Disallow: / User-agent: Zeus 2.6 Disallow: / User-agent: * Disallow: /cgi-bin/