Short answer: No, it is still very important
Most people have posted answers that include social engineering, or attacks that are necessarily interactive (such as keyloggers or variants of the evil maid attack). These can be defeated using secure remote attestation techniques (typically involving Intel TXT or SGX). Because most of these are interactive, and many adversaries can't afford to wait until someone logs into the server, I'll provide a few examples that can be used to compromise an encrypted server at any time, as well as mitigations. Note that this will be Linux-centric, but the hardware points are OS-agnostic.
Directly recovering memory using cold boot attacks
The method everyone knows, a cold boot attack, can recover encryption keys on most hardware with moderate reliability with two methods. The first involves cooling down memory using a freezing spray, and physically taking it out and putting it into a new motherboard to read the contents. The second involves keeping the memory in, optionally cooling it to improve reliability, and rebooting the system into a malicious and lightweight operating system, bootloader, or, in at least one known case, custom BIOS, which reads and dumps the memory onto a non-volatile medium.
The cold boot attack works because modern high-capacity computer memory, called DRAM, which stores data in capacitors that must be refreshed at a high rate when they are storing a charge, or they'll lose their charge. They are typically refreshed every 64ms for reliability, but even without power, many of them retain their data for a short time.
There are three main types of memory in common use in servers today: DDR2, DDR3, and DDR4. DDR2 can sometimes retain data for 30 seconds or more without power. DDR3 and DDR4 will lose power in just a few seconds, making them much harder to mount cold boot attacks against. Additionally, DDR3 and DDR4 memory will scramble their contents to reduce electrical interference, but the algorithm it uses is an LFSR, which is not cryptographically secure and is trivial to break.
Mitigating the first method is as simple as ensuring the DIMMs cannot be removed. This can be done by proper use of anti-tamper epoxy resin. If used properly, any attempt to remove it will destroy the memory in the process. This may not be possible if you are using a server which you do not physically own. Mitigating the second method requires your BIOS erase memory before booting. Disabling fast boot can sometimes cause memory to be wiped as part of POST. ECC memory also often requires being initialized by the BIOS before use. However, at least one time, a cold boot attack of this type involved a customized version of cold boot. BootGuard, a feature on many new Intel systems, will prevent custom BIOSes from being installed, but a particularly advanced adversary will still be able to bypass it.
There is still one, final method to defeat the cold boot attack, which works against both types of cold boot attacks, and that is to never allow the encryption key to ever come into contact with memory. The TRESOR kernel patch for Linux does just this, by putting the keys in the x86 debug registers. This does not make your computer immune from all attacks involving physical access, it only prevents cold boot attacks from reading your encryption keys. Data from your disk which remains in filesystem buffers for example will still be recoverable.
Reading and writing to memory using DMA attacks
Any device attached to the PCH (PCI, PCIe, LPC) have the potential to directly read and write to memory through DMA, or Direct Memory Access, attacks, also called evil bus mastering (a device which is bus master is allowed to send DMA requests, among other things). DMA is a technique that allows devices to asynchronously read and write to memory, without needing to involve the CPU, for improved performance. Unfortunately, if such a device is hijacked or maliciously inserted, it can read and write to all memory and the CPU can do nothing about it. The CPU can tell if the device has DMA capabilities, but if it is enabled, the CPU is putting its faith in the device, trusting it not to betray it.
There are three types of DMA attacks which I am aware of. The first is PCI/PCIe hotplugging, involving putting a malicious device into a slot on the motherboard, and letting the operating system auto-configure it. This is trivial to mitigate, by disabling hotplugging. This may require a change to the kernel configuration. The second type of attack is to hijack an existing, trusted device which is bus master, and use it to read or write to memory. This can be done either through exploiting the firmware running on the device, or hijacking the hardware through exposed debug interfaces, reflashing firmware and triggering a cold reset of the device's CPU, etc. The final type, which is much lesser known, is an attack over the LPC. The LPC is the equivalent of the archaic ISA bus on modern hardware. It handles old, low-speed peripherals. However, it can also be made bus master by asserting the LDRQ# interrupt. Not all motherboards expose this, and while this is purely anecdotal, an Intel developer I met said he had never seen a laptop which exposed LDRQ#. It may be different for servers. Unfortunately, the Intel PCH datasheet specifies that the LPC's bus master bit is read-only, forced on. If your system supports LDRQ#, mitigating LPC-based DMA attacks must be done in other ways.
There are two ways to mitigate DMA attacks. The first takes advantage of the fact that, while DMA works independently of the CPU, turning DMA on still requires the CPU give it permission by enabling the bus master bit. A driver, which is just software, can refuse a device which wants this permission. If the device is not given permission, it cannot initiate DMA. For example, not using the Firewire driver, or using the more modern Firewire driver which has DMA disabled by default, the Firewire PCI device will have DMA disabled (the bus master bit will be unset). This technique has its downsides, because some devices require DMA to function. Network interface cards, graphics cards, etc. require DMA, or they would be so slow that they would be unusable. While you would not have to worry about a new, malicious one being plugged in, an exploit against the card, or an attack against the hardware (controlling it using debug interfaces, for example) can use this privilege against the host.
The other mitigation is to use a feature in most modern CPUs, the IOMMU (the feature is named VT-d on Intel processors). The IOMMU is capable of efficiently filtering all DMA access, without disabling it outright. This is called DMAR, or DMA Remapping. The ACPI specification specifies a blob of data called the DMAR table which is stored in the BIOS which the IOMMU will use to isolate individual DMA-capable devices, redirecting any direct memory accesses to a specific, safe area of memory. On many Linux systems, you have to enable this by booting with the parameter intel_iommu=on
(for Intel), or amd_iommu=force
(for AMD). These options are not enabled by default because many BIOSes have a broken DMAR table, preventing the system from booting, or causing problems. Check if your system is using the IOMMU to isolate devices if dmesg | grep -e IOMMU -e DMAR
shows Intel(R) Virtualization Technology for Directed I/O
(obviously Intel-specific) in the output, and multiple devices are referenced.
Hijacking the CPU itself using JTAG
JTAG is a group of standards specifying interfaces and protocols for hardware debugging. One standard, titled IEEE 1149.1, allows putting a CPU into a low-level debug mode just by attaching a JTAG probe onto a header on the motherboard. Intel systems use use a modified, proprietary header, called ITP-XDP. If someone wants to attack an encrypted server with JTAG, they need to connect the probe to the XDP header. Even if the header is not clearly visible, the traces will still be there, and will still be live and viable. As soon as the probe is attached, all CPU cores will halt, and the attacker will be able to read the contents of all registers, read all memory, write to any registers, write to any memory, step the CPU instruction by instruction, change the instruction pointer, unpause and pause the CPU, and much more. In short, JTAG allows an attacker to take complete control of the CPU, controlling it like a puppet, and it can do nothing about it.
There is no way to mitigate a JTAG attack in software. Essentially all motherboards have XDP headers. A possible way to mitigate it, if you have physical access to the server and are allowed to permanently modify it (e.g. colo, hosting at home, etc.), you can use a strong epoxy resin and put it over the XDP header. This technique can be improved further by putting a thick plate of metal over the epoxy, preventing fine drills from breaking holes in the epoxy. I've been told that there isn't a worry about an attacker drilling through the bottom, because there are so many layers of traces in a modern motherboard that that would destroy the system. I don't know whether or not that would cause the system to shut down or prevent JTAG from working, so it may be better to put epoxy and a metal sheet on both sides.
Exotic, extremely difficult, or theoretical attacks
There is the well-known problem of side-channel attacks through power analysis and thermal analysis. This can be possible when the program performing encryption/decryption does not use constant-time operations where they are critical, resulting in detectable delays or large power draws that occur at different times depending on the value of the key. Modern crypto drivers are unlikely to have this problem, and hardware acceleration like AES-NI makes this even less of an issue. Mitigations include using ciphers that support hardware acceleration, like AES on proper hardware, using ciphers which have small S-boxes and can be optimized using bit slicing, like Serpent, or using red/black separation for proper EMSEC (look up NATO SDIP-27 for the specifications. Usually you'd just need to buy an approved computer).
Logic bus analyzers can be connected directly to any exposed trace or electrical wire, without ever needing to break the circuit. This is possible to do with DRAM, but as speeds get higher and higher, it gets increasingly difficult. PCIe uses techniques to improve speed which result in very strange electrical phenomena that logic analyzers do not at all like (a PCIe lane does not have a clean separation between hi and lo, as data bleeds into adjacent lines and electrical signals echo back and forth). It may be possible, but it would be difficult to do. If your PCI root hub supports AER, error reporting, you could configure your system to immediately disable bus master from any device which reports correctable errors, rather than continuing, as is the default. This fail-fast technique may be able to detect an attempt at inserting a logic bus analyzer into a PCIe device. Because this is such a theoretical and exotic attack which very well may not be possible anyway, and the mitigation is untested (merely an idea which came about during a conversation I had with a researcher), I'm just explaining the concept, not the steps to do it. It's likely that you wouldn't need to worry about reading PCIe data anyway, since it is unlikely to be carrying the key, but I'm including it for completeness' sake.
Finally, and most unrealistically, an extremely advanced attacker could trigger bugs in the CPU by exploiting timing and power quirks. These attacks (called glitching attacks) are commonly used to exploit simple microcontrollers such as cheap Atmel 8051s, but those are many orders of magnitude less well-tested than Intel processors. Glitching can involve setting the clock or clocks to invalid or unstable values, putting voltages out of specs, violating timing specs, etc. In particularly advanced cases, glitching can involve a technique called laser fault injection, where the processor is decapped (the top is carefully dissolved or burnt off), and a precisely controlled laser is used to elicit electrical activity in specific areas, inducing faults. The end goal of glitching is to cause a fault which changes the internal state of the device to a more attacker-friendly state, such as one where bus master may be enabled, or IOMMU restrictions may be broken. Real world examples with 8051 microcontrollers often attempt to disable security lock bits, such as the firmware read bit. They are almost always successful on such simple devices.
Mitigating everything, at least in theory
Finally, there is one commercial mitigation which protects from everything but the JTAG attack, and that is PrivateCore vCage. It is a virtualization solution which encrypts all memory, and puts the kernel in the CPU cache. DMA attacks are prevented fundamentally, and the TCB (Trusted Computing Base) is reduced entirely to the CPU itself, meaning that no matter what is hijacked, only the CPU must be trusted. In theory, even JTAG attacks could be defeated if the system executed all untrusted code in an SGX enclave. This would work because probe mode, the CPU mode JTAG puts the system in, cannot be used in the context of an SGX enclave. Unfortunately, this implementation is closed source, and the service is very expensive, so it's more interesting from an academic standpoint.