This article is part of a series on WordPress Optimization
This area only applies if you have control over your server. If you're using shared hosting you will most likely have no influence over this, so you can skip straight to the next section.
DNS: If you host your DNS on external servers this will reduce the load on your primary web server. It's a simple change, but it will offload some traffic and cpu load. (It's not very common to run your own DNS these days anyway.)
Web Server: Your web server can be configured to increase performance. There are a range of techniques including web server caching, enabling gzip compression and setting cache headers that help reduce load per visitor. Search for your specific web server optimizations (for example, search for "apache optimization" for more info). Consider learning the differences between Apache and Nginx.
PHP: There are various PHP accelerators available which can dramatically improve performance of your PHP files. This will apply to all PHP files, not just your WordPress installation. Search for PHP optimization for more information, f.e. APC. Since PHP 5.5, OPcache is bundled with PHP.
MySQL: MySQL optimization is a black art in itself. A few simple changes to the query cache settings can have a dramatic effect on WordPress performance because WordPress repeats a lot of queries on every request. Search for mysql optimization for more.
A great example of how WordPress has been optimized was presented by Iliya Polihronov at WordCamp San Francisco 2012. Iliya, among other things, does server optimization for WordPress.com. You can view his short presentation at the link below: