This article is part of a series on WordPress Optimization
This area only applies if you have control over your server. If you're using shared hosting you will most likely have no influence over this, so you can skip straight to the next section.
DNS: If you host your DNS on external servers this will reduce the load on your primary web server. It's a simple change, but it will offload some traffic and cpu load.
Web Server: Your web server can be configured to increase performance. There are a range of techniques from web server caching to setting cache headers to reduce load per visitor. Search for your specific web server optimizations (for example, search for "apache optimization" for more info).
PHP: There are various PHP accelerators available which can dramatically improve performance of your PHP files. This will apply to all PHP files, not just your WordPress installation. Search for PHP optimization for more information, f.e. APC.
MySQL: MySQL optimization is a black art in itself. A few simple changes to the query cache settings can have a dramatic effect on WordPress performance because WordPress repeats a lot of queries on every request. Search for mysql optimization for more.
A great example of how WordPress has been optimized was presented by Iliya Polihronov at WordCamp San Francisco 2012. Iliya, among other things, does server optimization for WordPress.com. You can view his short presentation at the link below: