How to speed up a site with the caching system

Overview

The powerful and flexible symfony cache system can speed up a website by saving chunks of generated HTML code, or even full pages, for future requests. Easy to set up thanks to YAML files, the cache can also be cleared with a simple command.

Introduction

The principle of HTML caching is simple: part or all of the HTML code that is sent to a user upon a request can be reused for a similar request, so this HTML code is stored in a special place (the cache folder in symfony), where the front controller will look for it before executing an action. If a cached version is found, it is sent without executing the action, thus greatly speeding up the process. If there is no cached version found, the action is executed, and its result (the view) is stored in the cache folder for future requests.

As all the pages may contain dynamic information, the HTML cache is disabled by default. It is up to the site administrator to enable it in order to improve performance.

Symfony handles four different types of HTML cache:

• cache of an action
• cache of a partial, a component, or of a component slot
• cache of an entire page
• cache of a fragment of a template

The three first types are handled with YAML configuration files, the last one is managed by calls to helper functions in the template.

The symfony cache system uses files stored on the web server hard disk. This keeps the cache simple and efficient, without any other prerequisites than the framework itself. It is not yet possible to cache in memory or in a database.

Global cache settings

For each application of a project, the HTML cache mechanism can be enabled or disabled completely per environment. In the settings.yml configuration file, notice the cache parameter:

prod:

dev:
  .settings:
    ...
    cache:                  off
    ...

The default value of this parameter is set to off, so you have to specifically set it to on to enable it.

It is set to off by default in the development environment. This means that if you decide to add caching to one of your apps, you will not be able to see the effect of it in the development environment with the default configuration.

Consequently, the boost given by HTML caching is only perceptible in the production environment - or in any other environment where cache: on.

One other cache parameter can be changed in the settings.yml file: The default cache lifetime i.e., the number of seconds after which a cached file is overwritten and the page processed again. The default default_cache_lifetime is set to one day, or 86400 seconds.

Caching an action

Actions that display static information (i.e. without any call to a database) or actions that read the information in a database but without modifying it (typically GET requests) are often good clients for caching.

For instance, imagine an action that returns the list of all the users of your website (user/list). Unless a user is modified, added or removed (and this matter will be discussed later), this list always displays the same information, so it is a good candidate for caching.

The result of the above-mentioned action is a processed template (listSuccess.php), and this is what is going to be cached.

To enable the cache on such actions, simply add a cache.yml file in the myproject/apps/myapp/modules/user/config/ directory, with the following content:

list:
  enabled     on
  type:       slot

all:
  lifetime:   86400

This configuration stipulates that the cache is on for the list action, and that it is of the slot type (i.e. caching an action, as opposed to caching the whole page, which will be described later). The lifetime is the time (in seconds) after which the page will be processed again and the cached version replaced.

Now, if you try to call this action from your browser (probably by requesting an URL like http://myapp.example.com/user/list), you will notice no difference the first time, but refreshing the page will probably show a notable boost in response time.

The caching system also works for pages with arguments. The user module may have, for instance, a show action that expects an id argument to display the details of a user. You would then just need to add the following lines at the top of the module cache.yml file:

show:
  enabled:    on
  type:       slot

Now, requests like http://myapp.example.com/user/show/id/12 will create new records in the cache folder and if you repeat this request, it will be much faster the second time.

Caching a partial, a component, or a component slot

The view chapter explains how to reuse code fragments across several templates, using the include_partial() helper.

<?php include_partial('mymodule/mypartial') ?>

A partial is as easy to cache as an action. To enable it, create a cache.yml in the partial module config/ directory (in the example above, in modules/mymodule/config/) and enable the cache for the partials by declaring their names with a leading underscore:

_mypartial:
  enabled: on

all:
  enabled: off

Now all the templates using this partial won't actually execute the PHP code of the partial, but use the cached version instead.

Note: The slot type cache is more powerful than the partial cache, since when an action is cached, the template is not even executed - and if the template contains calls to partials, theses calls are not performed. Therefore, the partial caching is useful only if you don't use action caching in the calling action.

Just like for actions, partial caching is also relevant when the result of the partial depends on parameters:

<?php include_partial('mymodule/my_other_partial', array('foo' => 'bar')) ?>

A component is a lightweight action put on top of a partial. A component slot is a component for which the action varies according to the calling actions. These two inclusion types are very similar to partials, and support caching the same way. For instance, if your global layout shows a:

<?php include_component('global/day') ?>

...in order to show the current date, then you can cache this component with such a cache.yml:

_day:
  enabled: on

Note: Global components (the ones located in the application templates/ directory) can be cached, provided you declare the activation in the application cache.yml file.

Caching an entire page

The final page is the combination of the template and the layout, according to the decorator design pattern. Until now, all that was written in the cache was the template, so every request ended up in a decoration process (putting the cached template into a processed layout). But if the layout has no dynamic element, you can cache the whole page instead of just the template.

Let's say this is the case for the test application described above: the layout simply contains navigation links, nothing dynamic, and can easily be cached. The cache can then be set to page type instead of slot. Modify the cache.yml as follows:

show:
  enabled:    on
  type:       page

list:
  enabled:    on
  type:       page

all:
  lifetime:   86400

If you request again the aforementioned pages:

http://myapp.example.com/user/list
http://myapp.example.com/user/show/id/12

...the full pages will be completely cached, and the second time you request them, the response will be even faster than with the action result cached.

But even a cached page involves some PHP code execution. For such a page, symfony still loads the configuration, builds the response, etc. If you are really sure that a page is not going to change for a while, you can bypass symfony completely by putting the resulting HTML code directly into the web/ folder. This works thanks to the Apache mod_rewrite settings, provided that your routing rule specifies a pattern ending with no suffix or with .html.

For instance, for a page accessible at:

 http://myapp.example.com/user/list

Create a user/ directory in your project web/ directory. Add in a list.html file containing the generated HTML code of the page, that you can get, for instance, by calling [curl](http://curl.haxx.se/.

$ cd web
$ mkdir user
$ cd user
$ curl http://myapp.example.com/user/list.html > list.html

Now, everytime that the user/list action is requested, Apache finds the corresponding web/user/list.html page and bypasses symfony completely. The trade-off is that you can't pilot the page cache with symfony anymore (lifetime, automatic deletion), but the speed gain is very important.

Unfortunately, the layout often contains some dynamic elements, including component slots. So the cases where the whole page can be cached are not very common. As a matter of fact, this type is often most used for RSS feeds, pop-ups, or pages with a layout that don't depend on cookies.

Caching a template fragment

Many times the slot and page types will be too large for the templates of an application. For instance, the list of users can show a link of the last accessed user, and this information is dynamic. Would that mean that nothing can be cached?

Thankfully not. Symfony allows you to cache fragments of a template with the cache() helper. The listSuccess.php template could be written as follows:

<!-- uncached HTML -->
<?php echo link_to('last accessed user', 'user/show?id='.$last_accessed_user_id) ?>
<!-- /uncached HTML -->
 
<?php if (!cache('users')): ?>
  <!-- cached HTML -->
  <?php foreach ($users as $user): ?>
    <?php echo $user->getName() ?>
    ...
  <?php end foreach ?>
  <!-- /cached HTML -->
  <?php cache_save() ?>
<?php endif ?>
 
<!-- uncached HTML -->
...
<!-- /uncached HTML -->...

Here's how it works: if a cached version of the fragment named 'users' is found, it is used to replace the code between the <?php if (!cache('unique_fragment_name')): ?> and the <?php endif ?> lines. If not, then the code between theses lines is processed and saved in the cache, identified with the unique fragment name ('users' in the example). The code not included between such lines is always processed and not cached.

Note that the list action must not have a slot- or page- type cache enabled, since this would bypass the whole template execution and ignore the fragment cache declaration. So either remove the list part of the cache.yml file, or write instead:

list:
  enabled:   off

After requesting twice the http://myapp.example.com/user/list URL, notice that the second execution is faster than the first. However, the answer is not as fast as when the cache was set to slot or page, since the template is partially processed and the decoration is made for every request.

You can declare additional fragments in the same template; however, you need to give each of them a unique name so that the symfony cache system can find them afterwards.

Like the slot and page caching, cached fragments can be given a lifetime in seconds:

<?php if (!cache('users', 43200)): ?>
...

The default cache lifetime (86400 seconds/one day) is used if no parameter is given to the cache() function.

Cache file structure

You don't need to know how the cache files are structured to make it work, so jump to the next part if you are not a what's-behind-the-curtain kind of person.

You probably noticed that each symfony project has a cache folder. The tree structure of its subdirectories is:

cache/[APP_NAME]/[ENV_NAME]/

... in which you can find three directories, config, i18n and template. As mentioned in the introduction, the HTML cache is stored in the template directory, so for the myapp application in the prod environment, focus on the myproject/cache/myapp/prod/template/ directory.

It contains a tree structure that matches the requested URLs. In the examples above, you requested:

http://myapp.example.com/user/list
http://myapp.example.com/user/show/id/12

So the tree structure of myproject/cache/myapp/prod/template/ is:

myapp.example.com/
  user/
    list/
    show/
      id/
        12/

That's as simple as it can be. The domain name is part of this path because you can host one app in different domains, and you don't want the cache of one domain to mess up with the others (imagine a multilingual website with different domain names like myapp.example.com and myapp.example.fr: they need distinct cache folders).

The files stored in these folders depend on the type of caching used:

The name given to a fragment in the cache() helper is concatenated after fragment_. Here, the second fragment was generated by:

<?php if (!cache('other_unique_name', SF_DEFAULT_CACHE_LIFETIME)): ?>
...

If you use component or component slots in the layout, remember that their cache files are found in a tree structure relative to the module/action of the slot, not the module/action initially called.

Feel free to browse the cache folder to look at the chunks of code that are saved; you might feel more comfortable with the cache if you see that it only saves what it is supposed to save.

Removing something from the cache

Clearing the cache for all actions

When you modify a template or an action, you will need to clean the cache to avoid errors. The symfony command line can launch a clear-cache process which will erase all the cache (config and i18n included, see below). From the root folder of a project, call:

$ symfony clear-cache
// or use the short syntax
$ symfony cc

In the production environment, once the cache is cleared, the first request will reprocess the configuration files and recreate the config cache. Then, if the HTML cache is set to on for some actions, the first call to these actions will regenerate the corresponding cache.

You may want to be more specific about what to clear:

$ symfony clear-cache myapp
// will erase only the cache of the myapp application
$ symfony clear-cache myapp template
// will erase only the HTML cache of the myapp application
$ symfony clear-cache myapp config
// will erase only the config cache of the myapp application

This is pretty straightforward, but maybe a little too drastic for some cases.

Clearing the cache for specific actions

When the model is updated, the cache of the actions related to this model have to be cleared. Imagine that the update action of the user module modifies the columns of the User object. The caching of the list and show action has to be cleared, or else the old version of the templates, with erroneous data, will be displayed. This is where the ->remove() method of the sfViewCacheManager object (a singleton) is useful.

In the update action of the user module, you need to add:

public function executeUpdate()
{
  ...
  $this->getContext()->getViewCacheManager()->remove('user/list');
  $this->getContext()->getViewCacheManager()->remove('user/show?id='.$this->getRequestParameter('id'));
  ...
}

The ->remove() method expects the same kind of target as you would provide a url_for() helper, so it is quite easy to figure out how to clear the cached files. As an optional second argument, it accepts a type precision to target specific files:

// removes all files
$this->getContext()->getViewCacheManager()->remove('user/list');
// removes only the page.cache file
$this->getContext()->getViewCacheManager()->remove('user/list', 'page');
// removes only the slot.cache file
$this->getContext()->getViewCacheManager()->remove('user/list', 'slot');
// removes only the fragment_users.cache files
$this->getContext()->getViewCacheManager()->remove('user/list', 'fragment_users');

The trickiest part is to determine which actions are influenced by a change in an object of the model. For instance, imagine that the current application has a publication module where publications are listed (list action) and described (show action), along with the details of their author (an instance of the User class). Modifying one User record will affect all the publication descriptions of which this user is the author. This means that you need to add to the update action of the user module something like:

$c = new Criteria();
$c->add(PublicationPeer::AUTHOR_ID, $this->getRequestParameter('id'));
$publications = PublicationPeer::doSelect($c);
 
foreach ($publications as $publication)
{
  $this->getContext()->getViewCacheManager()->remove('publication/show?id='.$publication->getId(''));
}
$this->getContext()->getViewCacheManager()->remove('publication/list');

When you start using the HTML cache, you need to keep a clear view of the dependencies of the actions, so that new errors don't appear because of a misunderstood relationship. Keep in mind that all the actions that modify the model should probably contain a bunch of calls to the ->remove() method if the HTML cache is used somewhere in the application.

Clearing the cache of another application

If you deal with several applications and do caching in at least one of them, you will probably end up with the problem of clearing the cache of another application.

Just picture a back-office application that modifies the details of a user: all the cached pages that display the information about this user in the front-office have to be cleared, but they are in another application.

What you should do is to setup an action in the front-office application that will behave like a web service. The back-office will call this action through HTTP, passing it the detail of the modification in the model. The web service will determine which pages need to be cleared according to the part of the model that was modified.

A tutorial will soon be published to describe this procedure.

Testing and monitoring the cache improvements

Staging environment

The caching system is prone to new errors in the production environment that can't be detected in the development environment, since the HTML cache is disabled by default in development. If you use the HTML cache, it is strongly recommended to add a new environment, that will be called staging here, which is a copy of the production environment (thus with cache enabled) but with the web debug set to on.

To set it up, edit the settings.yml of your application and add at the top:

staging:
  .settings:
    web_debug:              on
    cache:                  on
    no_script_name:         off

In addition, create a new front controller by copying the production one (probably named index.php in the myproject/web/ directory) to a new myapp_staging.php and edit it to change the SF_ENVIRONMENT value:

define('SF_ENVIRONMENT', 'staging');

That's it, you have a new environment. Use it by adding the front controller name after the domain name:

http://myapp.example.com/myapp_staging.php/user/list

Monitor the performance improvement

You will notice the familiar web debug box at the top right corner of the browser window, showing that the cache is set to on.

After a bunch of flags, this debug box also displays the process duration. Go ahead, clear the cache and make some tests: the second time you request a page, depending on the complexity of the action and template, can be several times faster with caching turned on.

To get more details about the caching impact, enable the debug mode by editing the myapp_staging.php and changing the SF_DEBUG value:

define('SF_DEBUG', true);

Don't forget to clear the cache before requesting the page again.

Beware that the debug mode greatly decreases the speed of your application, since a lot of information is logged and made available to the web debug box. So the processed time in debug mode is not representative of what it will be when the debug mode is turned off.

On the other hand, the web debug mode allows you to get additional information: the number of database requests needed to display the page, the ability to reload the page without caching (the middle button in the top part of the debug box) and full detail about the events encountered by the framework objects (the information bubble in the top part of the box).

Identifying cache parts

In pages that contain cached parts (slots, page or fragments), the web_debug mode allows you to display information about each part. By clicking on the 'cache information' link, you open a detail box showing:

• the internal URI of the fragment,
• the lifetime of the fragment,
• the moment when it was last modified

This will help you identify problems when dealing with out-of-context fragments, to see when the fragment was created and which parts of a template you can actually cache.

HTTP 1.1 and client-side caching

The HTTP 1.1 protocol defines a bunch of headers that can be of great use to speed further up an application by piloting the browser's cache system.

The HTTP 1.1 specifications of the W3C describe in detail these new headers. If a page is using the 'page' type cache in symfony, it can use one or more of the following mechanisms.

ETags

When ETag is enabled, the web server adds to the response a special header containing a md5 hash of the response itself.

ETag: "1A2Z3E4R5T6Y7U"

The user's browser will store this hash, and send it again together with the request the next time it needs the same page. If the new hash shows that the page didn't change since the first request, the browser doesn't sent the response back. Instead, it just sends a '304: not modified' header. It saves cpu time (gzipping) and bandwidth (page transfer) for the server, and cpu time (page rendering) for the client. Overall, pages in cache with an etag are even faster to load than pages in cache without etag.

In symfony, you enable the etags feature for the whole application in the settings.yml. Here is the default setting:

all:
  .settings:
    etag: on

When the server receives a request for a page containing an etag, it processes this page as it would usually. For a page of 'page' type cache, the response it directly taken from the cache. The server will compute a new md5 hash of the cached response and see that it's the same as the one sent by the browser. Instead of sending the response again, the server will send a 304 header only, and the browser will redisplay the page it keeps in its local cache.

Conditional GET

When the server sends the response to the browser, it can add a special header to specify when the data contained in the page was last changed:

Last-Modified: Sat, 23 Nov 2005 13:27:31 GMT

When the browser needs the page again, it adds to the request

If-Modified-Since: Sat, 23 Nov 2005 13:27:31 GMT

The server can then compare the value kept by the client and the one returned by its application. If they match, the server returns a '304: not modified' header, saving bandwidth and cpu time just like above.

In symfony, you can set the last_modified response header just like you would for another header. For instance, in an action:

$this->getResponse()->setHttpHeader('Last-Modified', $date);

This date can be the actual date of last update of the data used in the page, given from your database or your file system. If you use a 'page' type cache, you just need to set the last_modified header to the current time().

Vary

Another HTTP 1.1 header is Vary. It defines which parameters a page depends on, and is used by browsers to build cache keys. For example, if the content of a page depends on cookies, you can set its Vary header as follows:

Vary: Cookie

Most often, it is difficult to set the cache type to 'page' in symfony because the page may vary according to the cookie, the user language, or something else. If you don't mind expanding the size of your cache, you can use the 'page' type in these cases, providing you set the Vary header properly. This can be done for the whole application (for instance in a filter) or in a per action basis, using the Response related method. For instance, from an action:

$this->getResponse()->addVaryHttpHeader('Cookie');
$this->getResponse()->addVaryHttpHeader('User-Agent');
$this->getResponse()->addVaryHttpHeader('Accept-Language');

Symfony will store a different version of the page in the cache for each value of these parameters. This will increse the size of the cache, but whenever a request matching these headers is received by the server, it is taken from the cache instead of being processed. This is a great performance tool for pages that vary only according to request headers.

Cache-Control

Up to now, even by adding headers, the browser kept sending requests to the server even if it held a cached version of the page. There is a way to avoid that by adding Cache-Control and Expires headers to the response. These headers are disabled by default in PHP, but symfony can override this behaviour to avoid unecessary requests to your server.

Beware that the major consequence of turning this mechanism on is that your server logs won't show all the requests issued by the users, but only the ones received. If the performance gets better, the apparent popularity of the site may decrease in the statistics.

As usual, it's by calling a methods of the Response object that you can trigger this behaviour. In an action, define the maximum time a page should be cached (in seconds) as follows:

$this->getResponse()->addCacheControlHttpHeader('max_age=60');

It also allows you to specify under which conditions a page may be cached, to avoid that providers cache keep a copy of private data (like bank account numbers):

$this->getResponse()->addCacheControlHttpHeader('private=True');

Using Cache-Control HTTP directives, you get the ability to fine tune the various cache maechanisms between your server and the client's browser. For a detailed review of the Cache-Control directives, see the Cache-Control specifications at W3C.

One last header is ignored by PHP but can be set through symfony: the Expires header:

$this->getResponse()->setHttpHeader('Expires', $date);

When to use HTTP 1.1 cache?

If there is a slight chance that some of the browsers of your website's users may not support HTTP 1.1, there is no risk when activating the HTTP 1.1 cache features. A browser receiving headers that it doesn't understand simply ignores it, so you are advised to setup the HTTP 1.1 cache mechanisms whenever your web server supports them.

In addition, HTTP 1.1 headers are also understood by proxies and caching servers. Even if a user's browser doesn't understand it, there will probably be a device in the route of the request to take advantage of it.

Postscript

In addition to the HTML cache, symfony has two other cache mechanisms, which are completely automated and transparent to the developer. In the production environment, the configuration and the template translations are cached in files stored in the myproject/cache/config/ and myproject/cache/i18n/ directories without any intervention. This already speeds up the delivery of pages a lot, but the most powerful feature, the HTML cache, can not be fully automated. That's why it relies on configuration and additional code.

One last word about speeding up PHP applications: Eaccelerator also increases performance of symfony PHP scripts by caching them in compiled state, so that the overhead of compiling is almost completely eliminated. This is particularly true for the Propel libraries, which contain a great amount of code. Eaccelarator is compatible with symfony, and their effects can be combined.