189

I've got a CentOS server somewhere in the building; I can login into it remotely and VNC, etc. Now I've got to physically move it, and for that I need to physically locate the machine among the lookalikes around the office.

What can I do remotely to make the machine visibly or audibly identify itself?

msrd0
  • 240
  • 3
  • 13
Michael
  • 1,723
  • 2
  • 11
  • 7
  • 46
    Not silly at all, nor trivial. If you don't know what room a device is in, there's a small chance it would be almost impossible to find. – Todd Wilcox Oct 07 '16 at 04:00
  • 57
    Obligatory bash.org reference: http://bash.org/?5273. I remember reading a story of an old mail server running inside a closet that no longer had a door (became a wall in a remodel) too. This isn't too common, but it happens every once in a while! – Techwolf Oct 07 '16 at 05:21
  • 20
    Another story, more recent than that, and not actually drywalled, but still: https://www.reddit.com/r/talesfromtechsupport/comments/54ko52/the_tale_of_the_missing_comm_closet/ – Law29 Oct 07 '16 at 06:20
  • Take a look at this, maybe making it "beep" sound can help you find it --> http://unix.stackexchange.com/a/1979/102671 – Alberto Medina Oct 07 '16 at 03:19
  • 3
    So turning it off and then going and finding the one that's off isn't an option? – pay Oct 07 '16 at 12:38
  • 3
    "Among the lookalikes" makes it sound like you already know what type of a machine it is, and what it looks like from the outside! I'd say that's a rather good start compared to the horror stories you hear. :D – ilkkachu Oct 07 '16 at 15:09
  • @Techwolf Were you talking about [Server 54](http://www.informationweek.com/server-54-where-are-you/d/d-id/1010340?) at the University of North Carolina? [Reportedly](http://www.prnewswire.com/news-releases/server-54-where-arent-you-71897402.html), their [director of computing systems](https://www.linkedin.com/in/judsonknott) denied the story as an urban legend. It's a good one, though. – Engineer Toast Oct 07 '16 at 21:11
  • @EngineerToast [Yep, that's the one.](http://www.theregister.co.uk/2001/04/12/missing_novell_server_discovered_after/) Urban legend or no, it makes for a good story. – Techwolf Oct 07 '16 at 22:18
  • You can limit the subnet but that may be the whole building. Lookup the names of IP address around it and you may gate a name you know the location of. – paparazzo Oct 08 '16 at 00:12
  • 2
    You could perhaps to trace the network connection; run traceroute, go to the last listed router, find which port (routing table) and MAC address (ARP table) corresponds to the IP, go to the switch connected to that port, look up the MAC address to get another port, follow the cable connected to that port, repeat if you get another switch. Of course, this only works if your network wiring is well-documented. – user253751 Oct 08 '16 at 07:55
  • 2
    Many years ago my dad had this exact problem. They sent a sound sample loop to the speakers literally saying "PING" and walked around until they heard it. But that was a server with an audio card and a speaker... – Alex Feinman Oct 08 '16 at 14:16
  • 1
    I have a friend at work who was just telling me he has to find several servers and he *can't* ssh/ping them -- they're totally unresponsive. – iAdjunct Oct 08 '16 at 15:00
  • 22
    Are you positive its physical? Its very hard to find a VM when you're looking for physical boxes! – Criggie Oct 09 '16 at 00:11
  • 13
    Can you please give us an update when you do find it? Enquiring minds would like to know what worked for you. – Criggie Oct 10 '16 at 19:29
  • iAdjunct: At work I once had a server that didn't come back up after a reboot. I don't remember why iLo wasn't configured. I went to the data center, spend an hour looking for it, before I concluded that it must be mislabeled, and decided that it should just install a replacement the next day. Some years later, a new colleague attached a screen to a server in the datacenter to see what it did (nobody could tell from the label), and that's when we found it (and an explanation for why it hadn't booted properly). – Henrik supports the community Oct 13 '16 at 15:12
  • 3
    @Criggie: I haven't yet. The easiest answer, to make the server eject CD tray, won't work because ours don't have CD drives. I'll probably install impitool and try that; didn't do that yet because have to address a few more urgent matters that came up. There can be a shortcut in my case though: apparently, my machine is one of only a dozen or so in the office that have a graphics card with HDMI output, so it may be as easy as looking for unmarked machines with HDMI ports in the back. I'll tell as soon as find it. – Michael Oct 13 '16 at 16:17
  • 1
    Consider - managed switches would poinpoint its ethernet port. Even one managed core switch would give you an idea which port its on. – Criggie Oct 13 '16 at 19:18
  • 1
    @Criggie: silly me, somebody (not me) was smart enough to mark these machines' IP addresses on the boxes when placing them on the racks. That made the problem I though I had rather trivial. Immensely grateful for all the responses though, for the problem may come up in a settings without any prior marking. – Michael Oct 17 '16 at 20:56
  • @Michael Great work! Now is the time to think about "how could this have been done better?" and implement an improved process. Mostly its about documentation, creating and updating, vetting and verifying. – Criggie Oct 17 '16 at 21:48

12 Answers12

147

Use IPMI to trigger LEDs, increase fan RPMs or sound the beep/alarm. Take a look at the man page for ipmitool https://linux.die.net/man/1/ipmitool depending on the server you may be able to set the LEDs, LCD display, fan RPM offset(listen when nobody is in the office). Some other IPMI or BMC interfaces may allow you to sound the beep but this functionality is more platform specific.

a powerful workstation or server will sound like someone vacuuming with the fans turned up all the way.

EDIT: To use the Identifier lights as mentioned in comments, this will however require setting making sure that an appropriate IPMI interface is setup, there are several guides and tutorials available, and depending on the OEM there may be proprietary interfaces and management systems like Intel's Data Center Manager(http://www.intel.com/content/www/us/en/software/intel-dcm-product-detail.html). I have used this tutorial before but there are others https://www.thomas-krenn.com/en/wiki/Configuring_IPMI_under_Linux_using_ipmitool ipmitool -I <appropriate interface for system> -U<username> chassis identify force should force the ID to an on state, depending on the interface and configuration you may need to specify authentication type and other command line options.

Matt
  • 2,711
  • 1
  • 13
  • 20
  • 24
    I'm surprised I haven't seen this answer here already. ipmitool is the most sensible solution in most situations, as it activates blinking a blue locator LED that lights with the power of a thousand suns. – Spooler Oct 07 '16 at 03:39
  • 18
    ... if you have one physically installed. – Lightness Races in Orbit Oct 09 '16 at 10:40
  • 2
    ipmitool won't do much for you with a headless server though. And if you have a rack of twenty servers, flashing a light on one of them, or spinning up fans is a pain to do. You need something a lot more obvious and simple, which is why it is standard practice in my company to use a USB stick to find a metal server. Also.. not all IPMI units have identifier lights.. BUT.. if you do have an identifier light then this is most definitely on the list for simple ways to find a server. I mention the USB stick trick below because it works universally. – T. B. Oct 12 '16 at 17:19
101

If your server has a CD/DVD or similar drive, a simple solution is to run eject and look for the open tray. (Remember, just because the tray is open doesn't mean that you opened it; confirm a second time after you think you've found the server.) Failing that, Matt's answer to turn on an identifying LED or display a message on an LCD with ipmitool is the way to go.

You could also use traceroute to determine what router the server is connected to, and even play with the switch management interfaces to try to find the exact cable (or at least one end of it) on which the server is connected.

cjs
  • 1,355
  • 1
  • 12
  • 23
Techwolf
  • 1,113
  • 1
  • 6
  • 8
  • 13
    By *disk drive* I assume you mean *removeable optical media drive* (CD-ROM, DVD-ROM, or any of the writeable/rewriteable variants). +1 from me, since this is what I often do! – MadHatter Oct 07 '16 at 05:46
  • 1
    @MadHatter If there is a server that uses optical disks, probably someone will know where it is to change the disks. Why does a server use optical disks anyway? – v7d8dpo4 Oct 07 '16 at 09:16
  • 17
    I have no opinions on whether servers should have optical media drives. I noted merely that ejecting the **hard** disk drives from a machine just to find out where it was, was a fairly serious - some might say insane - undertaking, if indeed it is even possible. – MadHatter Oct 07 '16 at 10:02
  • 20
    Many servers have an optical drive even though it is not routinely used. Until fairly recently optical drives were the dominent method of installing the operating system on a new box. – Peter Green Oct 07 '16 at 14:07
  • 8
    @MadHatter I'd say you picked your handle well. Thanks for the mental image of a 1U forcibly ejecting its hard disk drive ;) – Konerak Oct 07 '16 at 14:10
  • Well in the days of mainframes, this wasn't such an uncommon idea ;-) 12" across and several platters - maybe 6-8" deep, these were **certainly not** "floppy"! – Julian Knight Oct 09 '16 at 13:59
54

An electrical method:

When the power consumption in your building is not going to change a lot (like at night):

  1. Run the computer at full cpu (like prime 95)
  2. Measure the current going through each circuit breaker in the circuit breaker panel, record these numbers. (this can be done with a AC clamp meter)
  3. Run the computer at low cpu
  4. Remeasure the currents. If any have dropped from before, the panel box normally states which room that breaker controls. Go to that location and find the computer.

Notes:

  • Take precautions to not electrocute yourself while measuring currents
  • If multiple breakers have a drop in current, rerun high CPU test and repeat until 1 breaker could be isolated.
Eric Johnson
  • 755
  • 4
  • 10
40

If you have a managed switch:

ping ghostserver

arp -a

Note the MAC address, and after check your switch ARP table per port.

Depending on the switch model you will have output like this:

enter image description here

** Notice that multiple MAC per port usually means an uplink to another switch.

Mast
  • 103
  • 5
yagmoth555
  • 16,300
  • 4
  • 26
  • 48
24

Use beep.

Assuming you have an internal speaker installed (often a small piezoelectric device), this will emit a beep on said speaker. If you are unsure whether there is a speaker inside the box in question, perhaps you can try with a different machine of the same type with known location.

You can keep it running in a loop while you wander through the house. Or you can encode your house-internal phone number into the melody somehow and wait for some colleague to call, asking you to please end that annoying noise next to their office.

Make sure you are executing beep as root, so that it uses the ioctl mechanism. Unless the binary is installed suid root on your system and your user is allowed to access it. Just emitting \a to your tty (which is the fallback in case of insufficient privileges) likely won't be enough since it will just reflect to your remote terminal emulator. Depending on your configuration, you also might have to load the pcspkr kernel module to make this work.

According to https://pkgs.org/download/beep, beep should be available as a package for CentOS, so just try yum install beep. Hope it's not modified in some way to make it not use the ioctl approach. This was a problem on Gentoo for some time.

Matt's answer mentioned beep (unclear whether the command or the concept) in passing, as did some comments, but I think this should be an answer in its own respect. Chances are you'll hear beeps through closed doors while checking led patterns may require opening the room and waiting in front of each machine to distinguish deliberate patterns from random noise.

MvG
  • 1,793
  • 14
  • 18
  • +1: Simple yet effective and does not require IPMI access. On older systems it is also quite loud (real speaker instead of the small ones used today). – user121391 Oct 12 '16 at 09:12
17

With managed network devices, you have a number of options: you can look for CDP/LLDP packets from the switch towards the hosts (which will tell the switch name and port number), or you can install LLDP agents on the server (so a 'show lldp neighbor' or similar run on the switch will show the port its on). Next best is to match up interface MAC addresses with the switch's bridging table. In both cases, you can trace the cable from the known switch end to the unknown server end.

If you don't have managed switches and have to rely on the host itself, then:

If the machine is from Dell or HP or similar, you can use 'dmidecode' to get the serial number/service tag, and match that up with the label on the device. At least on a Dell, you can also use OpenManage to change the text on the LCD display. You may also have some options to force the fans to run at full speed, which may be sufficiently audible to locate the machine.

ethtool as an --identify option, which will make the LED on a specific network interface blink. (Generally this is for telling which NIC is which on a system that has several, so this isn't ideal for this use case)

If it has an optical drive, you can eject the tray.

techieb0y
  • 4,161
  • 16
  • 17
10

Reboot and listen for the POST beep

Patrick
  • 461
  • 5
  • 7
6

If the machine has a wireless card but is using a cable (unlikely, but maybe the motherboard has it built in?) You can configure it as an access point then use a phone app like wi-eye to play "hot or cold" with it.

Sidney
  • 184
  • 1
  • 1
  • 7
4

Hard drive LEDs can be made to show a lot of activity, turning them into viable identifier lights, by reading with dd if=/dev/sdX of=/dev/null. No IPMI support needed.

Also see https://unix.stackexchange.com/questions/1974/how-do-i-make-my-pc-speaker-beep.

Also, if ethtool is available, messing with the interface speed settings can make the machine identifiable via the speed LEDs on the switch and/or NIC. Caveat: There is a risk of taking your machine effectively off line that way if the switch reacts badly to you turning autonegotiation off.

Community
  • 1
rackandboneman
  • 2,487
  • 10
  • 8
3

There are two easy ways to do this.. one has already been mentioned ( eject the cdrom tray )

The other is to login with a wireless connection and watch the USB messages. Then go around and plug in a USB stick with known data on it.

Either you will see a USB message when you do it on the right machine, or you can use any of the numerous USB interrogation tools (lsusb, etc) to see if it is now there.

Once you see a USB stick, mount it and look at it to see if its the right one.

Done.

Flashing lights and making things spin up and down isn't very reliable and requires a lot of attention. These two ways are dead simple, and don't require any sensory gymnastics.

T. B.
  • 313
  • 1
  • 2
  • 8
2

If you're going to move it, how about turning it off and looking for a switched off machine?

user6858980
  • 41
  • 1
  • This does not provide an answer to the question. Once you have sufficient [reputation](http://serverfault.com/help/whats-reputation) you will be able to [comment on any post](http://serverfault.com/help/privileges/comment); instead, [provide answers that don't require clarification from the asker](http://meta.stackexchange.com/questions/214173/why-do-i-need-50-reputation-to-comment-what-can-i-do-instead). - [From Review](/review/low-quality-posts/291096) – Tero Kilkanen Oct 13 '16 at 22:46
  • 6
    @TeroKilkanen How exactly is this not an answer? It looks like an answer to me. – Michael Hampton Oct 14 '16 at 20:33
  • 1
    But what if you still can't find it, and now can't turn it on again? – Eric Oct 15 '16 at 12:52
1

Various research has been done into how to exfiltrate data via EMF; for example, http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-577.pdf

If you can get the missing computer to produce a radio signal, you can in principle hunt for it by signal strength.

R.. GitHub STOP HELPING ICE
  • 707
  • 4
  • 14