Is it possible to create and export a base class dll, so that its lib is compatible with derived classes as long as there's no new declarations (and the declaration order is the same)?
Yes. The order of declaration in the derived classes doesn't matter, mostly.
I just wondered if it was possible to create an easily mod-able engine by having a folder of new dlls that add new components to the engine but by using core/base function calls.
Again yes. But don't create the instances directly in you program. Instead have a common way of instantiating your objects using a standardised Creator/Factory. That way, you don't even need to know about the derived classes, you only need the interfaces and factories.
And how would it be possible to link in these dlls with the program during runtime?
i.e. I have a folder where they can all go, this folder is known to the program, I just need to scan it for all filenames and load them in. (scan using boost library or whatever else I need, but that's easy enough to find out)