The short answer: They are two entirely different kinds of systems. They're just different. Comparing them is a bit like comparing apples and oranges. They are so fundamentally different -- different in so many ways -- that no one-sentence comparison will really do them justice.
Oh, that's not good enough for you? OK, then, here's the long answer.
Background: mandatory access control. In my view, the entire notion of "mandatory access control" (as opposed to "discretionary access control") is incoherent. Access control systems are always mandatory; the whole point of an access control system is to enforce some access policy on subjects. Compliance is not voluntary; subjects are forced to abide by the policy, whether they like it or not. So the entire notion is confused; any access control system always include mandatory elements.
That said, many people use the phrase "mandatory access control" to refer to a class of approaches to security, based upon military-style classification and information flow control. Mandatory access control systems often introduce some hierarchy of labels -- a simple example might be "Public", "Secret", and "Top Secret", or "Low" and "Hi" -- and tag data with these labels. The system then enforces some rules to ensure that anything that's derived from confidential data will also be treated as confidential; these rules typically enforce how labels are assigned and updated as information propagates throughout the system. A typical goal would be to ensure that Top Secret-classified information can never leak out over an unclassified channel
- (Strictly speaking, this is a "multi-level security" (MLS) systems; while MLS systems are a form of mandatory access control, one could also imagine hypothetical mandatory access control systems that don't use MLS. However, because MLS systems are the classical example of a mandatory access control system, "multi-level security" often gets conflated in many people's minds with "mandatory access control". At risk of causing confusion, I'll continue to use the phrase "mandatory access control" to refer both to the general concept as well as to the particular notion of multi-level security, because this is how the term seems to be used in the literature.)
Bell-Lapadula is an example of a theoretical model that is often considered representative of the mandatory access control approach.
SELinux is sometimes classified as a deployed system that implements mandatory access control, but this may be a bit misleading. SELinux has at least two configurations. In the strict configuration, SELinux enforces information flow control rules to every process running on your system. However, this is too strict for general use. Therefore, this is not how SELinux is deployed in practice. In practice, the Linux distributions that have adopted SELinux have instead deployed it in so-called "targeted" mode. For most intents and purposes, targeted mode limits only certain applications (there are a few global rules that apply to all applications but they are very narrow). Targeted mode consists of a set of ad-hoc rules about what kinds of information flow and accesses those applications can do. It only places restrictions on a subset of applications, and those restrictions are partial. It does not try to provide comprehensive security. It does not try to ensure that, e.g., Top Secret-classified information can never leak. Therefore, SELinux -- as it tends to be deployed today -- is a watered-down version of mandatory access control that falls short of what the original proponents of mandatory access control envisioned.
Object-capability systems. Object-capability systems are an approach to enforcing authorization policies. The core precepts are: access to a particular resource is granted to those who hold a corresponding capability (and only if the capability is held); capabilities can be freely passed from one subject to another; and capabilities are the only means by which access can be obtained.
Object-capability systems are designed to solve a particular problem. The design goal is that the OS should provide a very limited, simple substrate for expressing access control policies. Thus, the OS provides a few very basic primitives (the ability to use a capability, the ability to pass a capability on to another subject), and nothing more. What about applications that need to enforce rich access control policies? Rather than building support into the OS for each individual little thing that some application might want, in an object-capability system the application is empowered to implement its access control needs using the basic primitives provided by the OS. This keeps the interface exposed by the OS simple, while still preserving flexibility and enabling applications to express and enforce complex access policies.
Beware: "POSIX capabilities" are not capabilities. If you'll allow me to take a tangent, I'd like to inject a note of caution. You may have heard of POSIX capabilities. This name is unfortunately and poorly chosen. They are not, in fact, capabilities at all. Rather, the OS identifies a set of a dozen or so special privileges that are typically granted to root in a conventional Unix system, and allows processes to be granted a subset of these privileges. However, they are not held by processes as a capability, and there is no notion of passing them from one process to another. In other words, despite the unfortunately-chosen name, they aren't capabilities at all. Therefore, to prevent confusion, I'd like to exclude them from this discussion entirely (unless you tell me that "POSIX capabilities" is actually what you were interested in learning about).
Comparisons. Let me draw some comparisons and contrasts between mandatory access control and capabilities.
Confidentiality? Traditionally, mandatory access control systems have focused on enforcing confidentiality restrictions: e.g., ensuring that classified-Secret data is never leaked to anyone without a Secret clearance. In contrast, object-capability systems usually focus more on integrity and access control.
One reason for this is covert channels: no access control system can eliminate covert channels, and covert channels allow two colluding parties to conspire to defeat the purposes of traditional mandatory access control systems. If Alice has clearance to access the secret data, and Alice is conspiring with Bob, then Alice can arrange to leak the data to Bob, no matter what any access control system tries to say about this. Object-capability systems recognize this and don't try to prevent it; they subscribe to the slogan "Don't prohibit what you can't prevent". Mandatory access control systems try to prohibit this, even though they can't prevent it, and published work on mandatory access control is often evasive about covert channels or ignores the problem entirely.
Feasibility? Unfortunately, traditional mandatory access control policies -- while appealing in their elegance -- have turned out to be not very practical.
The primary challenge is the phenomenom that "eventually everything floats to Top", meaning that eventually almost everything in your system gets labelled with the top-most confidential label: e.g., Top Secret. Why? Well, imagine a spreadsheet containing some Public data and some Top Secret data. It will get labelled Top Secret. In general, any time you perform any computation that depends upon both secret and non-secret data, mandatory access control systems will label the result as secret. If Alice sends a non-secret message to Bob, and Bob is labelled secret, and Bob returns an acknowledgement back to Alice saying that he received her message, then mandatory access control systems will often label Alice as secret (this is intended to capture the fact that the acknowledgement might convey secret information from Bob to Alice). So basically any time that secret data mixes with non-secret data, it all gets labelled secret, and any time that a subject with access to secret data interacts with a subject with only non-secret data, they both get labelled secret. This acts as a one-way ratchet that causes more and more data/subjects to be labelled as secret. People have observed that, over time, most or all of the system eventually gets labelled as secret, and then the system becomes useless to non-secret users (you might as well have just said that this system is only intended for Top Secret data). This renders mandatory access control systems less useful than one might expect. As a result, traditional-style mandatory access control has seen little deployment.
Modern mandatory access control systems do try to avoid this problematic phenomenom, by introducing various variations to the basic MLS concept listed above. (For instance, SELinux's targeted mode avoids the "everything floats to Top" phenomenom, by avoiding trying to provide comprehensive information flow protection for all confidential data.) However, these variations on the traditional mandatory access control concept tend to introduce their own problems:
Complexity? In principle, both mandatory access control and object-capability systems are built upon a simple and elegant theoretical foundation. However, in the real world, mandatory access control systems are typical forced to depart from this elegant foundation if they want to avoid the "everything floats to Top" phenomenom.
As a result, deployed mandatory access control systems -- like SELinux -- tend to be extremely complex. Even in its targeted mode, SELinux's access control policy specification runs to something like a hundred thousand lines of access rules. This makes the system hard to understand, when something goes wrong. SELinux is notorious for causing hard-to-debug failures in some use cases, and on Linux distributions that use SELinux, you'll often see people suggesting that users try shutting off SELinux if they run into a hard-to-diagnose system administration problem.
So far, object-capability systems as implemented retain much of the elegance and simplicity of their theoretical foundations, so they don't seem to suffer from this complexity problem as much. However, we don't have as much practical experience with object-capability systems, so it is hard to know for sure whether this early experience will hold up.
Who sets policy? In mandatory access control systems, usually it is envisioned that some omniscient and benevolent system administrator will establish the access control policy that everyone must abide by. However, in practice, this can get really annoying. I'm sure we've all had the experience where some firewall or security system prevents us from sharing data with someone else, or some company security policy makes it hard for us to grant access to someone we're working with. What do you do when that happens? Why, send it in an email attachment, of course. In other words, having a system administrator set the security policy for everyone often fails to reflect users' actual needs, and the result is that users work to bypass the security.
SELinux is similar. In SELinux's targeted mode, the idea is that omniscient and benevolent SELinux experts will establish an access policy for each application you might ever install, specifying what files and resources that application is permitted to access. These policies have to be based upon the SELinux experts' guesses and predictions about how you might use the application, and what you the application might consequently need to access. However, in real life, SELinux experts are not omniscient, and sometimes they fail to anticipate how people might use their applications -- causing the SELinux policy to occasionally block legitimate program behavior.
(In principle, end users can write their own SELinux policies and modify the default system policy. In practice, not so much. The level of complexity makes this way beyond the means of most end users. In practice, I would expect that 99.99% of SELinux distributions use the default policy, because modifying it in any substantive way is too difficult.)
In contrast, object-capability systems don't rely upon an expert and well-informed system administrator to set the policy for everyone. Instead, each application is empowered to enforce its own access policy, based upon the needs of that application. Thus, access control is not specified in advance by a single central party; instead, it is determined, on the fly, by the combined actions of application developers and users.
Least privilege? One of the benefits of object-capability systems is that they tend to guide application developers towards designs that respect the principle of least privilege. Because the default is that each application receives no access, except for whatever it capabilities it is able to acquire from other sources, this tends to lead towards designs where each application receives no more privilege than they need, and where the same is recursively true of components and subcomponents of the application.
Some mandatory access control systems also try to enforce the principle of least privilege. For instance, in SELinux's targeted mode, SELinux experts analyze each covered application to determine what actions it might legitimately perform and what security privileges it needs to be granted to perform those actions. Then, these experts write a system security policy which specifies, e.g., the list of files and network sockets and other resources that, e.g., sendmail is permitted to access. That policy is distributed with your Linux distribution and is the same for everyone. In principle, this can also move us closer to the principle of least privilege, but it scales poorly: it requires a handful of experts to write rules for every application covered, so they can cover only a subset of applications (hence the "targeted" mode); it is trying to restrict legacy applications, in whatever form they come, which often means that no useful restrictions can be applied (try thinking about how to write a set of SELinux rules to limit emacs; what set of files are you gonna allow it to access? the only sensible answer is "everything", which defeats the principle of least privilege), whereas object-capability systems can do better by guiding application developers to write their applications in different ways.
Deployment experience? We have more experience with deploying mandatory access control systems than with object-capability systems. The conventional wisdom says that traditional mandatory access control systems are usually too restrictive and often prevent people from achieving legitimate tasks, though this doesn't apply to SELinux's targeted mode. The conventional wisdom says that object-capability systems cannot support legacy code well and require us to start over from scratch and re-design and re-implement our entire system and all applications running on it; this is probably the biggest shortcoming of object-capability systems.
There is a lot more to be said, and there is a lot written on capability-based security, object-capability systems, and mandatory access control. This answer only scratches the surface. I suggest you do some background reading. Then, if there is any aspect you'd like to learn more about, try asking a more focused question. Tell us a little bit about your particular application domain, about your requirements/goals/concerns, and what you've read so far, and then we can give you some more detailed advice. Or, tell us more about what specifically you'd like to know, and we can give you some reading recommendations.
