1. Context switches
Callback File System consists of the kernel-mode filesystem driver and user-mode portion of code linked to your application. When the OS sends the filesystem request to the filesystem, the kernel-mode driver invokes the user-mode code, thus causing a context switch back to the user mode and one more context switch when the callback is handled back to the kernel mode. If your user-mode code calls a system function, chances are than you again switch to kernel mode and back, as shown on the diagram. This process involves several threads and several synchronization objects as well.
Extra context switches cause significant slowdown of all operations. This is especially true when copying or other filesystem operation is done in small chunks (eg. 64Kb as Explorer does for copying, or even several bytes as some applications do). Callback File System includes several caches including the cache for file operations. Having the cache enabled slightly improves operations, but context switching is inevitable.
2. Global locks
During Open and Close callbacks there exist certain disk-wide resources that need to be guarded. This requires use of global lock (i.e. the lock which blocks any operations on the drive until it's removed). Such locks of course slow down operations, but they are required to protect resources from concurrent modifications from different threads. So the faster your code handles OnCreateFile/OnOpenFile and OnCloseFile requests, the less time is wasted waiting for global locks to be released.