either remote attacks or physical attack to a single UAV which can give control over a fleet of similar UAV's
If that's your only goal, then the security architecture of the UAV itself is irrelevant. All you need is to avoid class secrets, i.e. don't have anything that you both want to keep secret and include in every device. Any good security architecture avoids class secrets. This means don't rely on security by obscurity and, for any cryptography, use different keys in every device. Each UAV generates its own secret keys for internal use, and has its own private key(s) for communication with the outside.
However, if you're worried about attacks, then presumably the UAVs have assets (and they're probably assets in themselves). For that, the security architecture of the device obviously matters.
Against software attacks carried from the outside, a dedicated security module makes sense. Have it mediate all communication, and be the repository of cryptographic keys that are used for communication. Also make it responsible for ensuring the integrity and performing software updates on other components — perhaps it stores the firmware for all other components.
However, you'll still need to ensure that individual components are bug-free. The dedicated module acts as a firewall against malicious requests from the outside, but if the other modules misbehave on valid commands, the game is lost. If a bug in a servo controller causes the UAV to go down instead of up and it crashes, the security module can't help with that.
You need to think about your device's trusted base. What parts of the device are critical for its correct operation? As Philipp explains, the answer is lots. A lot of sensors, motors, etc. are necessary for the UAV to operate correctly (starting with: not crashing). If the controller module gets incorrect data, or if the motors don't turn at the speed requested by the controller, then the device will misbehave; thus all the sensors and motors are part of the trusted base.
The best you can hope for is an architecture where most components can only affect themselves, e.g. one misbehaving sensor can't modify another sensor's readings or prevent the other sensor from reporting. This example requires proper isolation on the data buses and proper authentication of each sensor in their controller.
Against hardware attacks, this model breaks down even more. If someone has physical access to the device, then they'll have fairly easy access to a lot of data buses. What the sensor reports is not necessarily what the controller receives: it could be data injected by the attacker. To avoid this, you need to bring cryptography into all the sensors. This raises the bar for an attacker, but it isn't a panacea. An attacker with physical access can always perturb the sensor's readings, causing the sensor to make accurate reports of useless information, e.g. by obscuring a camera, holding a heat source near a thermometer, etc.