Composite
This is the extension that allows all the fancy stuff to be done. It is essentially support for backing stores with an protocol for accessing and manipulating that store for presentation. The extension allows for both manual redirection and automatic redirection but only one connection and specify manual redirection (making it the composite manager). I believe this exclusive manual control per window but for practical reasons is usually specified on all windows if it is on any.
This extension does not need to be done on the same X connection as the WM. The first implementations, xcompmgr and glxcompmgr, were completely separate programs. This means that the Composite support can and should be developed first as a separate librep program+plugin.
I would suggest a development path of: separate instance -> loaded module (but separate X connection and event loop) -> Composite events handled in the same event loop as the rest of the X events for sawfish.
Damage and Fixes
RandR
Render
Not really needed but it may come in useful during testing of the Composite extension.
OpenGL
All that can be said is that a direct mapping of the C api may not be the best implantation.
XCB
This is a new C binding to the X protocols. It claims to fix many of the oddities and problems of the old Xlib design. This would be a major change with little to no difference from the user's perspective. Changing sawfish to be based of XCB may clean some things up and make development easier, but it would be a drastic change. This is definitely something to wait on.