Holy cow, I wrote a book!
Windows 95 introduced (and Windows Vista removed)
the concept of ShellExecute hooks.
These are objects which
implemented the IShellExecuteHook interface.
That interface had just one method:
which took a SHELLEXECUTEINFO structure
and returned S_OK to indicate that the item was executed,
S_FALSE to allow processing to continue,
and an error code to halt processing.
The intended purpose of the shell execute hook was to allow you to
extend the set of strings that can be executed.
For example, Internet Explorer 1 used a shell execute hook
so that you could type http://www.microsoft.com/
into the Run dialog and invoke the Web browser.
This was necessary because the original version of Windows 95
had no special knowledge of URLs.
Without the hook, typing a URL into the Run dialog would have
generated an error.
Let's look at those return values again.
The simplest return value is S_FALSE,
which means that you didn't recognize the item and wish to continue
Returning S_OK means that you recognized the item
and successfully executed it.
Returning an error code means that you recognized the item
but an error occurred when you tried to execute it.
Returning an error stops further processing, which is important
in this case, because you do want to stop processing:
You recognized what the user was trying to do and you couldn't do it.
If you had returned S_FALSE, then the shell
(and other shell execute hooks) would take their shot at executing
Since it's something only you recognize, they would fail,
and the user would get some sort of error like "File not found"
instead of an error that accurately describes why you couldn't
execute the item.
That's the theory, but I've seen a few applications that use
shell execute hooks for completely different purposes.
And they are doomed to failure.
One application installs a shell execute hook in order to
implement some sort of weird concept of "security".
When their hook is called, they check whether the user is
"allowed" to run the program;
if allowed, they return S_FALSE and if blocked,
they return E_ACCESSDENIED.
Another application uses their hook to log all the programs
that the user runs,
presumably so the logs can be inspected for auditing purposes.
After logging the action, they return S_FALSE
to allow normal execution to continue.
Both of these methods are really an abuse of the shell execute hook
model since they aren't extending the set of items that can be
executed by the shell.
They're just using the hook model to sneak some code into the
But that's also the problem.
First, they are intercepting only actions that go through the
If somebody just calls CreateProcess directly,
then the shell execute hooks won't run,
and whoever it is just snuck past your "security" and "auditing" system.
Second, these hooks both assume that they are the only shell execute
hook in the system.
there can be more than one.)
Suppose there are three hooks installed,
the hook installed by Internet Explorer to handle URLs,
the "auditing" hook, and
the "security" hook,
and suppose that the shell decides to invoke them in that order.
(The order in which hooks are executed is unspecified because,
if you use them as intended,
the order doesn't matter.)
If the user types a URL into the Run dialog,
the URL hook launches the Web browser and returns S_OK to say,
"I took care of this one."
Your "auditing" hook never got to see the URL,
and your "security" hook never got a chance to reject the operation.
Hooks that sit ahead of the "security" hook not only
get to bypass security, but they also elude the "auditing" hook.
So much for "auditing" and "security".
Since shell execute hooks effectively become part of
the ShellExecute function,
they are unwittingly subject to all the application compatibility
constraints that ShellExecute must follow.
Welcome to the operating system.
Your world has just gotten a lot more complicated.
There is no "list of application compatibility constraints" (oh
how life would be simple if there were); basically, anything
that ShellExecute did in the past is a de facto
Some programs crash if you create a worker thread inside
other programs crash if you call PeekMessage
Some programs call ShellExecute and
which means that your shell execute hook had better not return
before its work is done (or at least before the work has been
given to another process to finish).
Some programs call ShellExecute on an unusually small stack,
so you have to watch your stack usage.
And even though it is called out in the documentation
that programs should not call
shell functions from inside DllMain,
that doesn't stop people from trying, anyway.
Your shell execute hook had better behave in a sane manner
when called under such "impossible" conditions.