Understanding Bots

Not at all. Having the bots does not stop visitors in any way, shape nor form.

For those people still not sure what a bot is, Romeo
China
Guangzhou
and several others not mentioned here are search bots. We NEED them

Why do we need bots…. and why should you allow them some access to your site

A search engine is a site that allows people to put in a search term and find a site for what they are looking for.

A bot, or web crawler, is the little programme (think of a little robot made up of 0s and 1s) who runs around around the web with a thousand mates, checking out all the sites by following links. One link leads to another 5 links and those 5 links lead to another 10 links etc etc. The bots all scurry around and report back to the big programme, the search engine, what they find on the pages the links lead to.

The search engine waits, all knowing and, when someone then comes along and asks for FISH, the search engine smiles to itself and, patting a bot on the head that brought back the FISH descriptions, tells the person all the sites that are talking about fish.

That’s a bot.

MOST of our buyers come here because of the Google bot

Abbie has a great post about them...I will try and find the link...but in general...they are a good thing! Google, bing, yahoo etc all use them to catalogue the internet...if the big search engines don't know you exist..they can't find you. And the bots do the leg work for the search engines. They don't keep others away, they give you a better chance of being found.


I'll try to find the link! Hope this helps.

Matt

And I don't need to find it...because she already did!

oh i read that as - the bots were finally understanding... an i'm thinking, oh how nice, it's about time... but about what....

the bots are the worker ants collecting food for the colony.


---Mike Savad

LOL Matt

I got it! Thanks to all! I understand perfectly...

Just curious as to why one of my images (and only one) is repeatedly being visited by the Guanzhou bot. Over 90% of the 103 visits are from this bot, the rest are from elsewhere in China (e.g Fort Lauderdale, china).

It may be a buyer bookmarking it. We have buyers from there also

Erfurt - Germany -seems to be the new bot.

Well... somebody needs to give an explanation that all of us can understand! Thanks Mike!!!

M

Here you go Suzanne,

An Internet bot, also known as web robot, WWW robot or simply bot, is a software application that runs automated tasks over the Internet. Typically, bots perform tasks that are both simple and structurally repetitive, at a much higher rate than would be possible for a human alone. The largest use of bots is in web spidering, in which an automated script fetches, analyses and files information from web servers at many times the speed of a human. Each server can have a file called robots.txt, containing rules for the spidering of that server that the bot is supposed to obey or be removed.

In addition to their uses outlined above, bots may also be implemented where a response speed faster than that of humans is required (e.g., gaming bots and auction-site robots) or less commonly in situations where the emulation of human activity is required, for example chat bots.

Bots are also being used as organization and content access applications for media delivery. Webot.com is one recent example of utilizing bots to deliver personal media across the web from multiple sources. In this case the bots track content updates on host computers and deliver live streaming access to a browser based logged in user.

I gave a good example surely, in my first post in this thread?

I need to brush up my writing skills LOL

Thanks Conor

Thanks for the info, All!

So . . . I won't worry about people using my photos to create prints. But, I'm now DEPRESSED that probably the only views to my work are by these BOTS, and not real people! Talk about feeling dejected! LOL.

Suzanne McKay

Interesting thread. Can someone explain why a bot would be viewing just one of my photos? It's from Guangzhou, and it views the image several times a day. Thanks!

Harold we actually have buyers from there also. (s)he may have added it as a homepage on their browser

I have a couple of photos where no bots have visited. Strange... I don't know if it has something to do with the tags I'm using or not using.

Guangzhou sis till visiting only one of my images - it's up to near 600 visitors over 500 are from Guangzhou. It tends to visit this image multiple times a day and usually, but not always, in groups of 2. There are also a couple of pairs of visits from Xiamen.

Soooo, the visitors from Beijing, China who look at at a few of my photos and one of them, in particular, are bots (short for robots, I assume)Éand that is a good thing???

For those people still not sure what a bot is, Romeo
China
Guangzhou
and several others not mentioned here are search bots. We NEED them

Why do we need bots…. and why should you allow them some access to your site

A search engine is a site that allows people to put in a search term and find a site for what they are looking for.

A bot, or web crawler, is the little programme (think of a little robot made up of 0s and 1s) who runs around around the web with a thousand mates, checking out all the sites by following links. One link leads to another 5 links and those 5 links lead to another 10 links etc etc. The bots all scurry around and report back to the big programme, the search engine, what they find on the pages the links lead to.

The search engine waits, all knowing and, when someone then comes along and asks for FISH, the search engine smiles to itself and, patting a bot on the head that brought back the FISH descriptions, tells the person all the sites that are talking about fish.

That’s a bot.

MOST of our buyers come here because of the Google bot

Bots are the minions of the search engine gods. Without these god there shall be no buyers. Make sacrifices to the gods by giving their bots keywords and descriptions to catalog!

All hail the search engine gods. May we optimise to appease them.

There are good bots, and bad bots. These are good bots. Bad bots are the ones that infect your machine and make it a zombie for malicious purposes.

I 'clicked' Abbie's " click me" at another thread, and am so incensed, that I repeat my post lost to the rightly closed now thread, to say....

Joshua, I personally abhor your language!


http://fineartamerica.com/showmessages.php?messageid=1569968-GregJackson

I suddenly have become a favorite of one in Mountain View, California.

yes billy

your work has been here long enough for the title to show up in the search. adding keywords won't change anything more in the google search, because your not indexed every second of the day. you won't show up at all in the FAA search, where many people do go to look for art. and showing up in google is kind of pointless considering that:

a. its not for sale.
b. its almost summer.


---Mike Savad

how do you know if its a bot that viewed your page or a human?

You never really can tell if it is a bot or a person if all you have is the location. Plus a lot of what people call bots are actually proxy servers, so you don't know where he viewer is actually located.

when they buy something. otherwise they are all bots.


---Mike Savad

wow..then i have a lot of bot lurkers. maybe i should feed them lol

You can get a fairly good idea with a few things.

If you look at your image, and then click the 'view recent' link, it will expand all the recent visitors (if you are logged in as your FAA user).

If you mouse over each 'location' you will see the IP pop up in a tool tip.

Take that IP, and put it into these places, and it can give you some idea of what they or it are.

The location isnt always right, as it can be your ISP or something along your route, rather than the end IP.

For example, one of the common ones on FAA (and everywhere to be honest) is the 'New York' one. This is actually a company called ahrefs.com (Although lots of other IPs are also labelled New York), and they are actually based in Singapore. But, they also seem to have servers in the Netherlands. Not sure where FAA get New York from, but there are lots of databases out there, some better than others.

So, one of the many IPs for ahrefs is 37.58.100.142 (Taken form a visit to one of my images)

1) http://who.is This will give you info about the IP's ISP and who has it registered, in this example the CEO of hrefs.com in Singapore.
2) http://www.whereisip.net/index.php This will give you location data of the actual IP, in this case Netherlands, probably one of their offices / data centers.
3) http://www.projecthoneypot.org/search_ip.php This will tell you if its 'bad' etc and some other info if its a bot and the type etc.
4) http://software77.net/geo-ip/ Another service giving you the country of an IP.

There are many of these sites about. Just Google 'ip location'

One of the bits that help quite a bit to find bots is the 'User Agent' field that is sent on each page request, but we dont get to see that here on FAA, only the admins can see that. (You can find it for your own web site that you have control of though, through your logs) So its harder to find what they are. Project honeypot gives you some of that info, and that can tell you what Bot etc it is.

Most traffic/bots arnt that helpful. Its people grabbing data for their own business use, hacking/exploits or just data collection. Google & Bing are the most important ones to get visits from.

Ahrefs is one of the biggest bots, and all they do is analyze backlinks and then try to sell that info to you, so visits from them just arnt going to help you sell your art ;)

HTHs.

--------------------------------------------------------------------------------
Art Web Site | Facebook | Flickr | Pinterest | Google+

When checking visitors on my personal site, I just google the IP address. Often I find its a bot that is a spammer, hacker, content scraper, etc from the Google search results. From my personal perspective, I have no use or need of bots from Ukraine, Russia, China, Nigeria,Latvia, Estonia, etc. Low likelihood of sales, high likelihood of image theft from any of those locations. I have blocked many bots from those locations trying to login/hack into my site. So I just block them. I wish I had that level of control on my FAA and AW site portal. Probably the only useful bots (for me personally) are google, yahoo, twitter and bing.

@Susan - feed them keywords, descriptions, titles - they love text.

lol edward...i do try. some of this info is scary though.

@FireFlux -
"You can see the last time Google indexed a page by clicking on the little arrow at the end of your url in the search results (just before the 'Fine Art America' text) and then select 'Cached'
That will show you what the page looks like currently in Google's index, and at the top of the page it should show you the date it was last indexed. "

Thank you! This was very informative! Using the exact title, I tried this for the last image I posted on FAA. It showed up in 9th position in Google search and I found that it indexed it last on May 24th. I don't quite know yet how to use this information to my benefit, but it does give me a good feeling that Google DOES know I exist! I was somewhat skeptical until now.

Also, thank you for the great detailed explanation about visits from 'New York' and how to possibly identify a bot.

So much to learn - but the process itself is fun!

Lol. Not at all. Don't worry bots won't hinder you're talents!!!! :)

xoxo

Http://fineartamerica.com/profiles/alejandra-flores.html

Milford, Ohio.......bot, or not? Anyone else visited?

content scrapers like the one from kiez germany is different. we should have the right to modify our page to block them.

do you really know that kiez is a scraper? as far as i'm concerned it's a google drone, maybe the main one for all i know. and if you remove them, you might find yourself without any hits from any search.


---Mike Savad

yes. i found their site. they have a code you can put in your site if you don't want to be scraped by them. but since we have no control here, i can't do it. so far all i do here is waste my time putting images up here so that they can get stolen by a scraper. i am effectively buried unless i go out and do mass marketing to drive people to this site where they will get directed to someone else. so don't really equate a scraper who is a thief to an actual buyer.

um.....Milford, Ohio ? Anyone else?



Maybe you need your own site, to control,print,fulfil,send,return,share...etc,complain....?

its not a google drone, not google at all.

@ Vivian, Its not the description we need but the IP address (if you mouse over the location description it will show you that).

I'm guessing its from your image Precipice 3?

If it is, that IP to me doesnt look like its a bot, but of course its hard to tell 100%.

@Mike, Kiez is 'normally' bots for gathering up data and selling it back to users, quite a few I looked at link back to this company: http://www.ubermetrics-technologies.com/en/product/ which has a large IP block allocated to them.

Nearly all the kiez IPs link back to a company http://www.hetzner.de/ which rents out servers.

But, I would need a specific IP address to try and tell better.

I've actually not seen an IP for Google (or Bing to be honest), so I guess FAA do do some kind of filtering and remove those ones.

Rob.

I don't think there's anyway FAA could allow us to choose which bots to block on an individual basis. We're all on the same domain so I think its probably an all or nothing thing. Bots do not exist to help artists sell their artwork although a very few are useful to artists such as the search engine bots like google, yahoo and bing which help customers to find our work. FAA seems to allow all bot traffic. I think lots of bot traffic gives the illusion of many visitors and helps with site rankings which may be one reason why.

The other problem is that its difficult to stop bots from visiting. I have a small personal site with low traffic volume and its still a challenge to keep the bots out that I don't want. I've tried changing the robots.txt file but most of the time the bots don't bother checking that file or ignore it if they do. Because I'm on a WordPress powered site, I get lots of hackers from Russia, Ukraine, India and China trying to login to my site - possibly to control my site and use it for spamming purposes or redirects to other sites.

google IP's often start with 66.249.

Yeah, I dont think they should be blocked.

I just think that the visit counter should be adjusted to not include them.

Blocking them could cause problems, since its not a 100% rule, and you could block one that was actually an index bot, rather than spam or hacking one...

I dont normally block any unless they hit me hard (100/1000s of requests), as that slows down my server and effects the rest of the site, happens once every couple of weeks on my server. I've recently written a script that blocks bots automatically for 24hrs, that try to brute-forece the WP login.

Of course, just giving us Google Analytics on our FAA account would solve most needs, at least for me ;)

GA on the FAA site would be a huge bonus for sure.

Rob, thank you for checking the IP to help identify the visitor from MilfordOH, who visited 15 of my works, which is what made me question whether it's a bot or not......the IP for all those visits is the same from the physical address...............174 103 158 158

Maybe I'll get 'lucky" , huh? Hope so. Anybody else getting that IP/bot or visitor.?

I wonder what the origin of the Palo Alto, CA bot is?

Facebook.

This is a good discussion about Bots. I have learned a lot. I took the painstaking time to Google+ all 722 of my images. I am now seeing a lot more variety in locations for views… At last count I am up to 32,832 views. That is good…well sort of. I've only sold one image and two sets of greeting cards. I'm also experimenting with Google adwords but I think I'm going to stop that as it is not generating many clicks, no sales and it is costing me money. Off to look for other ideas. TTYAL

RJ