What is the best resource for really understanding Linux deeply

Question

I have been a Linux user on and off for many years. Recently, I have been using it daily. I am looking for suggestions for media; commercial books, free information, online videos etc that will help me to understand the OS at a much deeper level. Your suggestions please.

Answer 1

The best way to understand Linux is to break it, badly, and then fix it.

Answer 2

I'm not sure, when you say a "deeper level", if you're talking about getting more into the command line (some people never leave the GUI that comes with their distro, if you can imagine that), or with really gritty low-level stuff like writing kernel drivers. I'm going to assume more of the former than the latter.

I'd advise you to set reasonable goals for things you'd like to do using Linux and applications running on Linux (run a web server, serve files to Windows hosts, serve DHCP, run a graphical desktop, etc). It's my firm belief that you only learn when you're "doing". Once you've got a goal in mind, pursue it. That means reading man pages (lots and lots of man pages), "HOWTO" documentation, mailing lists archies, random blog posts, and, of course, the documentation that comes with the various programs you're installing or compiling to run on your boxes.

Having taught classroom-based IT certification courses for several years, I believe I can say with some degree of authority that the students who I saw make the most progress were the ones that were doing crazy projects of their own design, and learning by the seats of their pants.

As I said before, man pages, "HOWTO" documentation (http://tldp.org/docs.html and in many, many other places on the 'net) and mailing list archives are your friends. I'd steer clear of any books that talk about kernel internals, at least at this stage. You don't need that kind of deep knowledge to get started.

Talk to other people. Server Fault looks like it's turning out to be a great place to get good advice. Ask questions here, and don't think that you're going to look stupid doing so. If you can get some "face time" with people who are familiar with Linux, go for it. (Though I'd recommend you try and separate opinion from fact. There are as many "holy wars" in the Linux community over differing opinions of how to do things as in any other community-- perhaps more, given the nature of the community.)

To get really "deep" knowledge of the Unix heritage of Linux, you might go for some older Unix-specific administration or reference manuals. I'd steer clear of these early on (at least until you can appreciate the historical nature of the "paths not taken" with Linux as compared to some of the Unix-derived operating sysems).

Not knowing what your overall skill-level is, I will throw a shout-out to a book that my or may not be heplful. I highly recommend "TCP/IP Illustrated" (http://www.amazon.com/Illustrated-Volumes-Addison-Wesley-Professional-Computing/dp/0201776316). It's not Linux-specific at all, but you will be doing so many things that deal with TCP/IP that knowing it "cold" is a no-brainer.

I guess, in short, I'm saying that there's no magic book or books, no super-secret videos that the "masters" learned from, but absolutely no limit to what you can learn if you stick to it and aren't afraid to get your hands dirty.

Answer 3

Only years of experience playing with individual components will give you a deep understanding. Having said that, the vast majority of Oreilly Press books are really well written and perfect if you're not a dummy and have more than 24 hours. :)

There is a project called Linux From Scratch, which shows you how to build a linux distribution from nothing. You may find it educational to do once. It'll be time consuming as you need to compile everything from source and you'll throw it all away at the end.

I find LWN.net's kernel page invaluable for keeping up with how things work in the kernel at a high level.

Answer 4

Other than using it everyday (Which I think is a great thing!), I recommend you start thinking about services that could be implemented for your network on a linux box. Start designing it based on Linux services, research and implement once your ready. There will be hopefully mistakes in the process, and that experience will help you understand the OS at a deeper level.

Answer 5

You will find several free guides at The Linux Documentation Project, guides page. This is a short list i pick from there, but you should scroll through the page to find what you need.

• Introduction to Linux - A Hands on Guide; Jun 2008
• Linux on the Road; Nov 2005
• The Linux System Administrators' Guide; July 2005
• Advanced Bash-Scripting Guide; Mar 2009 (yes, this is good for understanding linux too)

You should look for specific HOWTOs for topics you are interested in. Another date sorted HOWTO list.

There are some starters at the Linux reviews beginners page.

Finally, this is a small book available online -- Linux Kernel in a Nutshell. Its in PDF form of the 2007 edition.

You should use these references only as a feeler to start you own search for things that you need.

Answer 6

For general Unix philosophy and an excellent introduction to the command line, there is Brian Kernighan and Rob Pike's classic The Unix Programming Environment.

Also, IBM's DeveloperWorks web site has over 900 articles in its Linux section. You can browse the article list for topics you find interesting.

Finally, once you're ready to get into the nitty-gritty, go to the source. Grab the source code for the kernel version you're using at kernel.org (your distribution will also have kernel source packages you can install) and check out the Documentation directory. You'll find lots of reference material on kernel internals and configuration settings.

Answer 7

School. To really understand the Linux system you need a wide array of Computer Science systems backgrounds. Compilers, Computer Architecture and Operating Systems. And you need a guide who can show you what's important and correct any misunderstandings you have or form.

Once you understand how modern processors work, and how C compilers exploit them, you can dive into books like Minix 3:

This is the latest version of the book that Torvalds, author of the Linux kernel, (loosely) based his work on. You'll learn about the fundamental components and algorithms of an operating system, and how exactly to implement one: the text comes with a nearly complete printing of the source code to Minix for reference and instructional purposes. Check out the interrupt handler to get a complete understanding of how the system works and where control flows.

And before you scoff at schooling and Computer Science, it's important to note that the jobs Operating Systems are asked to do are generally NP-complete. So understanding a wide variety of available algorithms is critical to performance tuning, since there won't be a provably optimal algorithm.

In addition to the kernel, there's a number of other areas. Because Linux is open source, this is an academic gold mine. Systems like Debian and Ubuntu make their source available, and it's dead simple:

apt-get source package-name

Many upstreams also host their code in revision control, so you can read the most up to date version of a program's source, or even see how it was built years ago.

Answer 8

Just start building your own distro. I've done it for yourself and you know what? After 4 month I've spend on that I know about Linux internals more than guys around me who are using Linux more then 10 years.

Answer 9

The source.

That, of course, if you mean the kernel.

If you are trying to learn the system above the kernel and the API, I would start at learning old school Unix. Maybe try some FreeBSD/OpenBSD/NetBSD/DragonflyBSD. A little Minix could help too. Then move to Slackware, Arch, Debian and Gentoo. They are all different (and come with great documentation), and in those diferencies, your curiousity will make you search why it is so, you will gain a lot of technical, comercial, political and historical knowledge.

Another way would be go diggin' in /etc, old Unix manuals, and of course, Google.

Linux from scratch is a great way of learning GNU/Linux.

Answer 10

Learning in any of the *nix environments is a holistic approach. It involves a series of epiphanies and experiences. This is not by accident. It is the deliberate byproduct of its design, as the original environment was designed around and for computer programmers. It is also its greatest shortcoming, as the focus is on technical aspects and not user experience. It is "a house with a sturdy frame that will last 100 years without service, yet its siding is haphazard and the paint clearly neglected".

Contrast this with Windows, which is the mirror image of this philosphy - do something that gives the user a tangible experience, but shield them from the inner workings of things. It is "a house with beautiful trim and paint, but the foundation that has been built-over several times with multiple work-arounds".

To really start learning, I would suggest building custom kernel images that have options specific to your hardware, and installing them with the options you want. You should fully expect going into this that something, somewhere will break, and you may or may not be able to back things out to "normal". This is a normal part of that learning process and you should approach this as if things will break (i.e. don't make this a primary install, use a separate drive or virtual machine or something...)

I've done an answer on a similar question, so to reduce typing I'll crosslink it here. You'll also want to read the first few paragraphs on this page, which will give you a better feel for what you're in for.

Answer 11

Start reading/cat'ing files in /proc and /etc - you won't do any damage by simply reading the files (save maybe screwing up your terminal/ssh-connection if you cat a file that produces binary info), and much of it is human readable. 'sysctl -a' output is also a gold-mine. When you find something interesting, google or man for more information about it.

You can find some real gems about how the kernel and OS work this way.

A few quick pointers:

/etc/inittab
/etc/rcS.d
/etc/rc2.d
/etc/crontab
/proc/1/environ
/proc/filesystems
/proc/meminfo
/proc/cpuinfo
sysctl -a | grep vm.swappiness
/etc/default    # debian-based
/etc/sysconfig  # redhat-based

Quick tip - some output in /proc is NUL separated, and hence is hard to read. Use 'tr' to convert the NUL's to newlines, eg:

 sudo cat /proc/1/environ | tr '\0', '\n'

Answer 12

There is a lot to learn about Linux, or any operating system. One kind of learning is what I call "in depth" learning, which is finding out how the kernel operates, what assumptions it makes, how the various bits talk to each other, and how it deals with hardware. That's kernel stuff. Stuff like that is very useful in figuring out why the operating system broke in just that way. In fact, I learned it the same way Karl Katzke did. I broke stuff, and made it better.

For this kind of knowledge, Linux is pretty easy. It's all documented in many places. The same can't be quite said for Windows, though there is still a lot of doc out there for it.

Then there is the 'getting around in the OS' learning, which is where knowledge of bash/sed/awk/regex and all that other stuff comes in handy. Because the fact is, an operating system is much more than its kernel these days. You have vendor supplied driver blobs. You have how this particular distribution packages its startup scripts. You have various patch/update mechanisms. You have system daemons that everyone needs, but aren't kernel.

This is a much broader spectrum of knowledge than simple kernel-fu. It varies by distributer so how things work on Red Hat may not work the same on Slackware. Where files are kept can vary. As can what they picked to replaced 'vi'.

"Learning Linux" is more about learning an entire ecosystem than it is a simple operating system.

Answer 13

I had a book called "Linux Programming" that taught me more about the OS than any "Learn Linux" book. I wish I had the ISBN number, but it was amazing..partly because it went into the programming "why"s, not not just the "how"s.

Overall, most of my learning was honestly through breaking it and fixing it. It happened slowly, and I'm still learning new things after 12 years.

It's definitely a process. The key is to be very, very curious. When you encounter something that you're unfamiliar with, become familiar with it. Seek out knowledge about how and why it ticks, and learn the underlying technologies, too.

Answer 14

I found drifting through Linux, FreeBSD, OpenBSD, Solaris, and then even various other Linux distros a good way to learn a lot about Linux. About how it did things differently. You learn as much about your home town while visiting other places than you learn about where you are.

Answer 15

When it comes to books, I like the O'Reilly's, so "Essential System Administration Pocket Reference" and "Linux System Administration" are possible starting points.

My copy of TCP/IP Network Administration is an older one and I haven't used it in a while, but I found it very useful to get a more in-depth understanding of what's going on.

Answer 16

You're already making the first step by using Linux as your regular desktop.

If you want to understand how all the parts hook together and are configured, I would suggest running Gentoo. From a regular stage3 install, you have to compile your own kernel, install all your services (except SSH which is there out of the box), build and configure X yourself, etc. When you're doing that, you end up with a much greater understanding of all the bits that go into a binary linux installation. It also doesn't hurt that Gentoo has the best documentation of any Distro I've found, or that portage is a fantastically powerful and flexible package manager.

Answer 17

I would strongly recommend this as "required reading":

The Unix Programming Environment - Brian W. Kernighan / Rob Pike

ISBN 0-13-937681-X

http://www.amazon.com/Unix-Programming-Environment-Prentice-Hall-Software/dp/013937681X

Answer 18

Someone else mentioned this, and I think its the thing that helped me most with using various linux flavors. If you feel you have a reasonable handle on linux, start using open/net/freebsd.
As for why, most linux folks I've worked with may use one or two linux distros that aren't all that different from the other. Sure, most of them knew those distros like the back of their hand, but they could still get surprised on an alarming basis with certain things. I think (not to start a flame war) that the way the BSDs are assembled its a little easier to discover the "why" of the way things are done (plus, lurking on the bsd mailing lists they explain that "why" to death). It comes down to changing your point of view, it may never occur to you to ask why certain things are the way they are until someone does it in a completely different manner.

Answer 19

This posting on Stackoverflow has a large listing of Unix/Linux resources including most of the canonical works on the subject.

Another way to learn what makes Linux tick behind the scenes is to work through Linux From Scratch (http://www.linuxfromscratch.org/). It is essentially a set of exercises in manually installing and configuring the various components of a linux system and is a very good intro to learn the nuts and bolts from.