Sometimes you don’t want some parts of your website to be indexed by search engine bots. Perhaps some directories are for private use only or perhaps you are working on a test site. If the page is not linked from anywhere, it’s supposed to be safe from spiders because no one knows the exact url, in theory, at least. But you want to make sure that the private part of your site remains private.

There are 3 common ways to prevent search engines from indexing some directories or files.

1. Using robots.txt
   When a search enginge crawlers visits your site, it looks for a file named robots.txt for instructions for your site. In order to prevent any well-behaved crawlers from indexing a directory, you will put the following in robots.txt.
   
User-agent: *
Disallow: /directory_name/

   If you have 2 directories, MyData and TestData, that you want to keep private, robots.txt will look like:
   
User-agent: *
Disallow: /MyData/
Disallow: /TestData/

   The asterik * indicates that this instruction is for all spiders. If you only want Google bots from indexing, then you will use:
   
User-agent: googlebot
Disallow: /MyData/
Disallow: /TestData/

2. Using meta tags in each file
   You can put the following meta tag between the < head > and < / head > tags in any file you do not want search engine spiders to index. (Remove the space before and after < and > .)
   
< meta name="robots" content="noindex" / >

3. Using a password
   You can password protect files and directories either using a web server control panel or using applications that have a password option.

I found out today that a great Photoshop site with lots of useful tutorials and tips along with tons of beautiful images vanished because of a website server crash. The site is Photoshoptalent.com. I used to visit their site almost daily to learn more about Photoshop. Then about a month ago, I noticed that the site was down for a while.

Server Crash
Now that my day job’s workload is getting normal, I tried to go to the site to learn some more tips. To my surprise, photoshoptalent.com resolved to pxleyes.com. The home page looks similar but the tutorials I was looking for were not there. I looked around more and found the webmaster’s message titled “What Happened to PST?” The webmaster said,

To my deepest regret I have to inform you that the PST server had a major hardware failure and everything on the hard drive was wiped. The last few days we were trying to recover as much data as possible, but until now it is unsuccessful… every file is corrupted or broken. This is also the reason why the site was down for an entire week. The only backup there was, was one from several months ago, which was in the same server and disappeared with all the rest.

I was appalled. It was a great site with tons of good contents. All of them are lost…. I am sure the webmaster and his/her community will rebuild the site again. But it won’t be the same. Well, the new site may be better!

Importance of Backups
I do backup my PC/laptops data almost religiously for various reasons as you can see in this post. But I haven’t backed up my website for a while. As are most WordPress-powered sites, I need to back up databases in addition to regular files. Learning about what happened to Photoshoptalent.com was a good reminder for me. I’ll do a full backup tonight.

From time to time, I see web sites that do not have directory listing disabled. This happens on an Apache server that is not configured properly. If the webmaster create a directory and does not place an index.html (or index.php) file, the contents of the directory are there for everyone to see.

Not allowing the visitors see the directory listing does not make a website secure per se. But visitors can go up and down the directory trees if the listing is not disabled. In order to disable a directory listing of a particular directory and all of its subdirectories, you can just place a file called .htaccess with the following line on it.

Options -Indexes

As long as a subdirectory does not have its own .htaccess file that allow directory listing, it works.

Today, I will talk about how to put an icon to url. The tiny image is typically called a “favicon” and it’s in an ico format, which means a typical file name is favicon.ico. Its size is usually 16×16 but I usually make a 48×48 gif file in Photoshop. To convert it to an ico format, you can download a free Photoshop plugin from telegraphics.com.au and use Photoshop. But an easier way is to create it online from dynamicdrive.com.

So here’s the step to put the tiny image next to your website’s url.

1. Create a 16×16 or 48×48 image file and save it as a gif file.
2. Go to dynamicdrive.com and click “Create Icon.”
3. Download the created file (favicon.ico) to your PC.
4. Upload the icon to the root directory of your website.
5. Add the following line to your header… in between < head > and < / head >
   
< link rel="shortcut icon" href="favicon.ico" >< /link >

Note: You need to remove spaces from the example html tags shown above.

Another note: favicon may not show up immediately. It takes some time. If you change your favicon image, clear the browser chache and refresh the page and wait.