How do device driver instructions program the GPU?

Question

Say I am using a computer, any typical modern computer more or less. A device driver is responsible for everything I see on my screen, including right as I type this the screen is updated(this is, of course, within the scope of an OS, like Windows).

Let us take this down to the level where the hardware does not care if there's a kernel or not, and just takes instructions.

Basically, how do GPUs "know" exactly how to do everything? Does a driver basically control every minuscule, intrinsic, or hardware circuit-level function possible or needed to compute binary data for display using special instructions that a driver provides?

If so, does the GPU use a "special" language or "assembly" to understand instructions sent to it like a CPU does?

I have too many gaps in my knowledge, and there's this unpleasant "mystery" behind GPUs and accessing them directly via hardware.

For example, a GPU and CPU have to be able to connect somehow, so it is possible to access a GPU via CPU assembly and implementing the correct data/address buses. There's no way a device vendor can make it impossible to access the GPU outside of a driver, because a driver is also compiled code, and the GPU, as a programmable circuit, must work with binary instructions.

I have asked these types of questions and they get deleted violently quick with no answer why on many sites, so what's the big mystery/secret here? It's a piece of hardware on a motherboard that is accessed like any other.

So really ... assuming this is an "ok" question here concerning software and hardware, anyone can directly access a GPU without vendor specific files, because the files are not magical.

So how are GPUs "programmed" from the bare-metal, direct hardware perspective on the hardware-specific level?

Answer 1

Basically, how do GPUs "know" exactly how to do everything? Does a driver basically control every minuscule, intrinsic, or hardware circuit-level function possible or needed to compute binary data for display using special instructions that a driver provides?

No, the device has its own ICs which control all of the hardware-level aspects of the device’s functioning, and they interface with a central processor which coordinates the on-device aspects as well as provide a higher-level programming interface which can be used to access, read from, write to, and control the device.

If so, does the GPU use a "special" language or "assembly" to understand instructions sent to it like a CPU does?

Yes and no. Like most devices, video-cards use standards to simplifying matters. They use standards like VESA to access the regular functions like boot-time video, or CUDA to access GPU-programming functions. In addition, programs can access the non-standard, or device-dependent functions directly using machine code (often C compiled to assembler) in conjunction with the device’s programming documentations (which are available from the adapter’s web site).

I have too many gaps in my knowledge, and there's this unpleasant "mystery" behind GPUs and accessing them directly via hardware.

Drivers access the device directly, which is why they have special access (regular user-level programs are not allowed to access hardware directly in Vista+ for security and stability reasons). The access the device using ports, DMA (Direct Memory Access), and memory-mapping.

For example, a GPU and CPU have to be able to connect somehow, so it is possible to access a GPU via CPU assembly and implementing the correct data/address buses. There's no way a device vendor can make it impossible to access the GPU outside of a driver, because a driver is also compiled code, and the GPU, as a programmable circuit, must work with binary instructions.

A driver can read and write the device with regular CPU instructions by accessing memory-mapped ports and such. As an easy-to-understand example, you can put a character on screen in text mode by writing to “memory” at address B8000. This address was not RAM, but rather, it was mapped to correspond to the video-card’s memory, so writing to it would write to the screen. Likewise, you could write tothe screen by putting the character you want to write in the AL registry, 0x09 in the AH register, and then calling the BIOS interrupt 0x10. More advanced functions are not different; you can read and write the adapter’s memory, call the functions on its chip, and so on using whatever inetrface(s) that the device exposes and documents.

So really ... assuming this is an "ok" question here concerning software and hardware, anyone can directly access a GPU without vendor specific files, because the files are not magical.

Only for features that comply to standards, but everything else (i.e., the more advanced stuff), you’ll need to consult the manufacturer’s programming documents.

So how are GPUs "programmed" from the bare-metal, direct hardware perspective on the hardware-specific level?

Through multiple layers:

1. Solid-state components like transistors and such
2. Low-level ASICs
3. On-board processor(s) and chipsets
4. Low-level programming interfaces (assembly)
5. Higher-level programming interfaces (DirectX, OpenGL)
6. High-level programming language (C++, C#, Python, etc.)

Answer 2

I'm not exactly a genius in computer architecture, but I'll try to treat your question by points, at the best of my knowledge.

---

A device driver is responsible for everything I see on my screen, including right as I type this the screen is updated(this is, of course, within the scope of an OS, like Windows).

It is true, mainly because direct interaction with the GPU is not easy to implement. Hence the development of graphic frameworks like DirectX and OpenGL.

This handy definition from Wikipedia explains this in more detail:

A device driver simplifies programming by acting as translator between a hardware device and the applications or operating systems that use it. Programmers can write the higher-level application code independently of whatever specific hardware the end-user is using.

---

Basically, how do GPUs "know" exactly how to do everything? Does a driver basically control every minuscule, intrinsic, or hardware circuit-level function possible or needed to compute binary data for display using special instructions that a driver provides?

GPU's are microprocessors. As such, they process. They process data that is fed from the CPU (i.e. position of elements) to a proper display. They "know" how to do everything because they are manufactured that way and a large group of people agreed on the proper ways to send data to and from the GPU.

Think of the display driver as a buffer, or a proxy. Imagine you're a program window and the user moves you up. You ask the GPU to refresh the display. But as the GPU speaks Klingon and you speak Vulcan, the driver receives your command, translates it to the GPU that in turn does what you commanded.

If so, does the GPU use a "special" language or "assembly" to understand instructions sent to it like a CPU does?

---

From Wikipedia:

ARB assembly language is a low-level shading language, which can be characterized as an assembly language. It was created by the OpenGL Architecture Review Board (ARB) to standardize GPU instructions controlling the hardware graphics pipeline.

There's this example, and remember that individual manufacturers (NVIDIA, ATI) have their own instruction sets.

Also, there exists OpenCL and many other ways to, in a way, directly program instructions to the GPU.

All this should answer your question, to some degree.