Dynamic robots.txt with Zend Framework

Here’s a quick little tutorial to show you how you can create a dynamic robots.txt file with the Zend Framework.

My robot files aren’t usually complex since I’m really careful about not putting anything that could be indexed in the web root, but there are some cases where you may want to have more control over the robots file at the application level. In the example below, we’ll be serving up different robots content depending on if the file is accessed from HTTPS or not.

You may also want to put your robot settings in your application ini file or control what gets indexed at the Controller level where the other business for your application logic lives.

Step 1 – Create a custom route in your bootstrap

This route will catch requests from robots.txt and route them to the robots action of the RobotsController.

$router->addRoute('robots',
  new Zend_Controller_Router_Route('/robots.txt', array('controller' => 'Robots', 'action' => 'robots'))
);

Step 2 – Create the robots.txt Controller

I opted to not use the default indexAction for my robots.txt action. This would to try and avoid any chance of Google indexing additional versions of the robots controller/action urls. In the controller below, /robots redirects to /robots.txt. It still leaves /robots/robots open, but that’s less likely to get hit that just /robots.

Notice that I’ve also set the content-type to text/plain which is what it would have been had this been an actual text file. I’m not sure it makes any difference to Google, but this will make the page resemble as close as possible an actual robots.txt file.

class RobotsController {
  public function indexAction() {
    $this->_redirect("/robots.txt", array('code' =>; 301));
  }

  public function robotsAction() {
    $this->_helper->getHelper('layout')->disableLayout();
    $this->getResponse()->setHeader('Content-type', 'text/plain');

    $this->view->sslEnabled = false;

    if(isset($_SERVER['HTTPS']))
      $this->view->sslEnabled = true;
    }
  }

Step 3 – Create the robots.txt view

In the controller above, I assigned a boolean to the view so that I could render a different robots.txt if the page is accessed via SSL. In this case, I want to completely remove the HTTPS site from Google’s index to avoid duplicate content issues.

<!--?php if($this->sslEnabled):?-->
User-agent: *
Disallow: /
<!--?php else:?-->
User-agent: *
Allow: /
<!--?php endif;?-->

Published byJosh McGinnis

Josh is a software engineer, leader, startup advisor for the LA Chamber of Commerce and consultant residing in the Los Angeles area. Josh is passionate about helping entrepreneurs and businesses reach their maximum potential.

1 Comment

Post a Comment