Advertisement:

Author Topic: Question about Crawlers & Bots  (Read 256 times)

nightfly13

  • Jr. Member
  • **
  • Posts: 96
Question about Crawlers & Bots
« on: September 08, 2018, 01:55:34 pm »
Dear community…

I have a question regarding bots and crawlers, ….

i have about 2000 listings and 500 users in my site, until now i have about 100 listings visualizations by day, at 3 days ago this number icrease allot and i start waving arround 800 / 900 listings visualizatons by day…. So my questions is, can the  bots and crawlers made a kind of a click on the listings wile they crawl my site????

Best Regards

davidyan

  • Newbie
  • *
  • Posts: 16
Re: Question about Crawlers & Bots
« Reply #1 on: January 12, 2019, 02:47:50 pm »
Hi,
If you wish to get rid of BOTS you can use this .htacess TRICK:
Quote
RewriteBase /
# IF THE UA STARTS WITH THESE
RewriteCond %{HTTP_USER_AGENT} ^(aesop_com_spiderman|alexibot|backweb|bandit|batchftp|bigfoot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(cosmos|crescent|curl|custo|da|diibot|disco|dittospyder|dragonfly) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(drip|easydl|ebingbong|ecatch|eirgrabber|emailcollector|emailsiphon) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(emailwolf|erocrawler|exabot|eyenetie|filehound|flashget|flunky) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(frontpage|getright|getweb|go.?zilla|go-ahead-got-it|gotit|grabnet) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(grafula|harvest|hloader|hmview|httplib|httrack|humanlinks|ilsebot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(infonavirobot|infotekies|intelliseek|interget|iria|jennybot|jetcar) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(joc|justview|jyxobot|kenjin|keyword|larbin|leechftp|lexibot|lftp|libweb) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(likse|linkscan|linkwalker|lnspiderguy|lwp|magnet|mag-net|markwatch) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(mata.?hari|memo|microsoft.?url|midown.?tool|miixpc|mirror|missigua) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(mister.?pix|moget|mozilla.?newt|nameprotect|navroad|backdoorbot|nearsite) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(net.?vampire|netants|netcraft|netmechanic|netspider|nextgensearchbot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(attach|nicerspro|nimblecrawler|npbot|octopus|offline.?explorer) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(offline.?navigator|openfind|outfoxbot|pagegrabber|papa|pavuk) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(pcbrowser|php.?version.?tracker|pockey|propowerbot|prowebwalker) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(psbot|pump|queryn|recorder|realdownload|reaper|reget|true_robot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(repomonkey|rma|internetseer|sitesnagger|siphon|slysearch|smartdownload) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(snake|snapbot|snoopy|sogou|spacebison|spankbot|spanner|sqworm|superbot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(superhttp|surfbot|asterias|suzuran|szukacz|takeout|teleport) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(telesoft|the.?intraformant|thenomad|tighttwatbot|titan|urldispatcher) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(turingos|turnitinbot|urly.?warning|vacuum|vci|voideye|whacker) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(libwww-perl|widow|wisenutbot|wwwoffle|xaldon|xenu|zeus|zyborg|anonymouse) [NC,OR]

then you can use this trick that I use personnally:
Quote
<IfModule mod_headers.c>
  Header set X-XSS-Protection "1; mode=block"
</IfModule>
<IfModule mod_headers.c>
  Header always append X-Frame-Options SAMEORIGIN
</IfModule>
<IfModule mod_autoindex.c>   
    Options -Indexes
</IfModule>
<IfModule mod_headers.c>
    Header unset X-Powered-By
</IfModule>
<IfModule mod_headers.c>
Header always set X-Content-Type-Options "nosniff"
</IfModule>

<IfModule mod_headers.c>
Header always set X-XSS-Protection "1; mode=block"
</IfModule>

<IfModule mod_headers.c>
Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains"
</IfModule>

Have a nice time
Yan

nightfly13

  • Jr. Member
  • **
  • Posts: 96
Re: Question about Crawlers & Bots
« Reply #2 on: January 16, 2019, 09:54:34 am »
Thanks allot for your help.. i will try..

Best regards
Nightfly