WordPress uses a customized implementation of the TinyMCE WYSIWYG editor created by Moxiecode. It is written in javascript. Like most things in WordPress, TinyMCE is can be customized through a plugin architecture.

TinyMCE's native plugin system works by dropping a directory of js files into the plugins directory and modifying the array of settings used to initialize TinyMCE. You can still do this but it is also possible to include a TinyMCE plugin using the WordPress API.

A WordPress plugin can insert javascript code into the TinyMCE script by utilizing the action hook 'tinymce_before_init'. This allows you to process the JS before sending it to the browser, such as for translating strings or customizing features. For instance, you might need to insert the site url into your script.

Because javascript allows you to redeclare functions, and the tinymce_before_init hook occurs immediately before TinyMCE is initialized, you have the freedom to modify any and every part of TinyMCE. If you want to replace the HTML filtering code, drop in your own version of the function. If you want to change something in the init array, you can.

After you have inserted your javascript functions, you will need to modify the TinyMCE plugin list. This is done with the filter hook 'tinymce_plugins'. Your filter should take an array as its only argument, add your plugin's name to it and return it.

If your TinyMCE plugin will create a button for the toolbar, you will have to add the button name to the toolbar array. Like modifying the plugin list, this is accomplished with the 'tinymce_buttons' filter. You can also add buttons to create a second or third row using 'tinymce_buttons_2' and 'tinymce_buttons_3'.

That covers the WP API for adding TinyMCE plugins. The TinyMCE API is discussed elsewhere. For sample javascript code, please see the subdirectories of wp-includes/js/tinymce/plugins.