What makes a secure element a secure element is what it's used for. A chip is a secure element if it's used to store and process confidential material
for the benefit of a larger device. So for example a SIM card¹ is a secure element, because it processes data belonging to the network operator for the benefit of a mobile phone. But a credit card is not a secure element, because it is used on its own and interacts with a variety of devices (payment terminals, ATM, etc.).
Not all smartcards are secure elements. Conversely, secure elements must be able to make computations based on the keys that they store, and they're supposed to tamper-resistant, so all secure elements are smart cards. Most secure elements are smaller than the credit card form factor, but they have the same chip physical interface.
Like other types of smart cards, secure elements store confidential data on behalf of remote entities and are supposed to resist physical attacks (preferring to self-destruct rather than reveal secrets). In terms of security, the most relevant difference between a typical smartcard (in banking, authentication, etc.) and a secure element is that a secure element is typically kept inside the device that it acts on behalf of, whereas typical smartcards spend most of their time in the user's pocket or wallet. Embedded secure elements are even soldered onto the device. Thus if a secure element is stolen, it's stolen inside the device that it's useful for.
¹ Strictly speaking, the card with the chip is a UICC, “SIM” refers to its functionality.