The software architecture of FS9 and FSX was designed as a simulator core platform with attached SDK and API. And that SDK and API are what you use to expand the core through modelling and/or programmatically.
Be careful with all those (paper) books out there, quite a few of them are nothing more than a reiteration of the SDK documation. Depending on what you want to do you need different skillsets and different experience. If you want to build aircraft and scenery, you should have some background in 3D modelling and a basic understanding of how computer game graphics generally work. But to create sophisticated addons like aircraft systems, external weather engines, ATC programs etc. you need some background in programming (C++ is a must, C# is optional, plus software architecture, coding and testing methods) on top of knowledge about the simulator API (which again is already described in the SDK documentation).
The first "book" to check out would be the SDK documentation. You will find it for FSX right in the root folder of the SDK. It contains quite a lot about everything you want to know, how the tools have to be used and what you generally need to do to achieve what you want, including examples. The rest comes with months/years of experience.
On a side note: changing the simulator core in any meaningful way is impossible, unless you have the source code files, compiler and project structure. You will only get these if you pay Microsoft for full development and distribution rights - like Lockheed Martin and Dovetail Games did.