Multiple Windows installers on a USB stick

Question

I've recently got my hands on one of those SanDisk Extreme 64GB sticks and started tinkering with it. During this story, please keep in mind that this is a model where removable bit is set to 0 (which means Windows sees all its partitons by default - each is shown as a single disk in explorer).

Goals:

1. To have many different bootable distributions on a single stick
2. Among those should be Windows 7 installer AND Windows 8 installer
3. Only one partition should be auto-mounted in Windows (and preferably other OSs) -the first one (storage for files). This is to prevent accidental alteration of bootable partitions or files on them. It should be of type NTFS.
4. Optionally, I would also like a directly bootable fully persistent linux (preferably Ubuntu) on this stick.
5. All of these features should be available from a single menu shown at the time of booting the machine from the USB stick.

Ramblings:

Now this can all be achieved pretty easily by using a combination of SYSLINUX and GRUB (you'll see below). However, there is one problem: all Linux-based bootable items (those that use create ramdisk -> load kernel -> have fun scheme: like PartEd Magic, Ubuntu Installer, etc) seem to have no problem whatsoever to run from their own folder on a single bootable partition. They get along fine when simply putting them in different folders and pointing single SYSLINUX menu entries at them (whatever it is they use after that, might just be another SYSLINUX :)).

The problem (as always :D) is in Windows installers. They (or should I say, bootmgr) refuse to recognize relative paths. If you put one Windows installer in a folder, say, Win7 so that it looks like /Win7/bootmgr, the darned thingie looks for \Boot\BCD when passed control (not \Win7\Boot\BCD - the location it is actually stored in). Having learned that, naturally, you can't put more than one Windows installer on a single partition (a single directory structure). This violates GOAL 2.

Current Scheme:

Need 10 reputation to post images :(. Here is the scheme visualized.

Description:

1. Normal NTFS partition. Do whatever you want with it - it won't affect those sensitive boot links :)
2. FAT32 boot partition. This is where control is transferred from MBR. This one has to be primary to be able to be transferred control from MBR. It contains main SYSLINUX - the one that shows menu (GOAL 5 and GOAL 1). All partition below are extended to lift max-4-primary-partitions per disk
3. Windows 7 installer. This partition contains bootsector which points to GRUB on it which points to /bootmgr (don't ask why so many bootloaders - it's a test setup :)). Works fine except the concern below (GOAL 2).
4. Windows 8 installer. This partition contains bootsector which points to GRUB on it which points to /bootmgr. Works fine except the concern below (GOAL 2).
5. Persistent Linux partition to which Ubuntu was installed using normal methods (from the same USB stick). It uses GRUB to manage its boot menu. Also has boot sector of course (GOAL 4).
6. Linux Swap. Used by Linux on previous partition usually but is not bound by any restrictions on who can use it.

Now it all seems right (and it does work).. except GOAL 3.

Problems:

1. I have tried using hidden flag to hide those partitions from Windows' sight and it worked well, even too well. It also hid them from Windows' installer's sight. When I tried to install Windows from a hidden partition, the installer basically did not find where did it boot from :O. This is CRITICAL issue.
2. I have used 2 additional partitions to house Windows installers. This is not really a great scheme as every time I wish to change something and it does not fit on these, I have to re-partition. This is NON-CRITICAL issue - I can live with that.

My thoughts:

1. There are things out there like ISOLINUX and GRUB4DOS (they come up when you search regarding this). From what I understood they can help me actually mount Windows installer's bootable .iso. Is that true? Some pages contain a warning about images mounted like this will only be accessible at BIOS boot stage - when Windows installer starts it will not be able to see the mounted image. Is that true? Can you point me in the right direction?
2. Maybe I could use some custom installer for Windows (because essentially, I fight bootmgr here 100% of the time) or chroot the bootmgr somehow so that it becomes sane? Can this be performed? I think this still leaves me with a problem that the partition is still hidden and installer probably won't see it
3. Maybe I could hack the path into bootmgr, although that is a pretty bold assumption :)

If there is any additional advice regarding this, you are very welcome to discuss it here. I am open to any suggestions. If there is something that needs to be clarified or additional info has to be provided - feel free to ask.

Thank you very much!

Answer 1

After a long and eventually successful journey, I have been able to achieve it all.

I have created a tutorial on how to perform this. Check it out.

Per request, I am also posting the whole thing below:

Software:

First of all: you need a PC with Windows 7/8 and administrative access to it (Vista may work, not sure). This is where the tutorial will be performed.

To perform all that we will need this software package, it is also mirrored on GitHub (click Download ZIP). [0] (from now on, references will be marked with []). To download it, you may need to register there (not sure why).

The software package includes:

• YUMI by pendrivelinux.com. [1]
• altmbr.bin from SYSLINUX package which allows to boot partition by number. [2]
• Multiple scripts and one executable (including vmount program - 32 and 64 bit versions by erwan.l and sleep 32/64 bit from this thread to make this all work together [3].

Additionally, you need to download the following:

• WinSetupFromUSB (later WSFUSB). This one is crucial for this scheme to work. Use .exe version if you don't have 7zip installed. This tutorial is written at the time when version 1.4 was the latest. [4]

Optionally (the software I recommend which will make the process easier and more flexible but is not required):

• Gparted live .iso for partitioning and setting partition flags (analogues include Parted Magic which decided to go paid recently or Partition Wizard which is smaller but keep in mind that it is not Gparted and you have to have some experience in order to operate it). [5]
• Notepad++ is a superior text editor for Windows with many plugins. [6]

And of course a few Windows installer .isos as well as few bootable Linux distributions.Those could be anything that YUMI and SYSLINUX support (you'll be able to check that out later in this tutorial).

When you download all this, you are ready for next steps.

Concept:

Here, the scheme of operation will be briefly discussed.

The scheme discussed here is what I use for my setup. It certainly is not the only way to achieve the same goal.I'll do my best to try and explain it so that you could customize the setup for your needs.

This process seems to be best described in picture. (sorry for Paint, graphics design isn't really my strong side):

That's pretty much it. I really do not know how to explain it any better. I'll be monitoring this thread, feel free to ask questions in replies.

Actions:

1. [ Configuring software ]

   1. Extract the software package [0] to some folder (all software required is listed above, look there if you haven't already).
   2. Extract WinSetupFromUSB into the same folder (so that it's beside YUMI.exe and all other stuff). Double click it and extract to the same folder.
   3. Now take the folder with WinSetupFromUSB and drag it over onto the drag_here.cmd file. Like this (folder structure should be like this too): If you did everything right, a window will be shown saying that the process was completed:

      WinSetupFromUSB was set up successfully, you can delete that folder.
      Now it's time to run 'start_admin.cmd'.
      Press any key to continue...

   4. Just like it says, press any key to close the window. Do not run start_admin.cmd yet. Go over to the next step to define partition structure.

2. [ Partitioning ]

   1. Decide on the partition structure: yours should include at least 1 storage partition and at least 1 boot partition. Boot partition has to be of type FAT32 (FAT32 has a restriction of biggest file - 4GB and a biggest volume - 32GB which is 32768MB. If you plan to use bigger files or volumes, you can't use NTFS but you can try exFAT as experiment - I did not try it) and storage can be anything but I recommend NTFS. Mine looks like this: Storage partition is 32GB NTFS and Boot partition is 15GB FAT32 (it does not have a letter like G: because it has hidden flag already set, you'll learn about that later on). If you are using big disk, it might be a good idea to put boot partition first as some BIOSes have trouble reading beyond certain bounds.
   2. Disk Management way is described here but you can use any other software you like to apply the partition structure (like Gparted [5] mentioned earlier or Bootice which can be launched from WinSetupFromUSB). Open Disk Management in Windows. This can either be done through searching in start menu (type in Disk Management) or in Windows Command Prompt: diskmgmt.
   3. Apply the desired partition structure. Right-click any partition to manage it. Right-click the unallocated space left after partition deletion to create new partition. Always choose to assign drive letter and you can freely format it right in the same wizard:
   4. Before exiting Disk Management, note the number of your boot partition and number of your disk in the system. This is how they look like in Disk Management: Here, the disk number is 2 and boot partition number is 1.
   5. When you've finished applying partition scheme, it's time to make this drive actually bootable.

3. [ Adding base boot environment (YUMI) ]

   1. Go to the folder from step 1. Start YUMI.exe.
   2. Check Show All Drives box and select your boot partition from the dropdown list. You may optionally choose to format it as well.
   3. Now, it's time to add some Linux-based distribution(s). I have chosen Partition Wizard as an example (you may choose any other distribution as well as add more than one, link to Partition Wizard is given in SOFTWARE section). It is recommended to add all Linux-based distributions at this point, because it will be harder do do later on after the disk is finalized.
   4. Select Browse and choose your .iso image. The screen will look like this:
   5. Click Create and follow the on-screen instructions. When it asks if you want to add more, you can choose yes and repeat the procedure (do not choose formatting option when repeating the procedure, obviously) to add another distribution.
   6. At this point, the disk is bootable. Disconnect the disk and reconnect it again (this is needed for Windows to detect the partition as active properly).
   7. Go to folder from step 1 and run start_admin.cmd as Administrator. This is crucial for its operation, right-click it and select Run as Administrator:
   8. It is going to ask you for disk number and bootable partition number (step 2.4). Re-open Disk Management and find those numbers like you did in step 2.4. Specify the respective numbers and press Enter.

   ---

   [ATTENTION] Again, I am not responsible for what happens to your disks (including those not mentioned in this tutorial) if you fail to specify the right disk and partition numbers in this step or in any other place.

   ---

   9. At the end, it's going to ask you if you want the executables compressed before being put on your USB drive. Size is rarely an issue nowadays so I recommend entering n here (unxepected side effects may surface later if you press y, only do so if you know what you are doing).

4. [ Adding Windows-based installers (WinSetupFromUSB) ]

   1. Back in the same folder, go inside WSFUSB folder (not WinSetupFromUSB folder) and run WinSetupFromUSB.exe appropriate for your system architecture.
   2. Choose your partition in the topmost dropdown menu. Make sure to do it carefully.
   3. Check the box next to Windows 7/8 installer and click the button on the right to select your Windows installer .iso.
   4. Select the Advanced options box. Make sure that Don't check for and install grub4dos MBR and Launch Q-Dir before Setup.
   5. Double-check your settings and make sure they look almost 100% identical (except disk and .iso path) to this example:
   6. Optionally, you may select Custom menu names for Vista/7/8/Server Source so that you are able to see custom names when selecting Windows distributions at boot time.
   7. When you are ready, press GO button. If you selected the box from step 4.7, it's going to ask for the names. After that, wait for the operation to complete.
   8. Repeat the steps for each Windows distribution you want to add.

5. [ Finalization ]

   1. At this point we have 2 bootable structures: one for Linux (starts from MBR) and one for Windows (starts from bootmgr on your boot partition). We need to link them up. Open your bootable partition in Windows explorer.
   2. Open file <drive root>\multiboot\syslinux.cfg. It contains information about the menu which is displayed when booting from your drive. More info about it can be found here: menu.c32 - syslinux.cfg syntax, chain.c32 chainloader command. For now, we need to make a menu entry which will allow us to boot our Windows installers. Add the following lines at the bottom (or in between other menu entries) to create a new menu entry:

      MENU LABEL Windows Installers/Recoveries >>>
      MENU INDENT 1 
      KERNEL chain.c32
      APPEND fs ntldr=/bootmgr

      Preferably, add an empty newline before and after this code. Adjust the menu entry label if you wish. Save and close the file.
   3. [OPTIONAL FROM HERE] Open WinSetupFromUSB again. We will delete the stale Back to Main Menu(Grub4dos) menu entry from Windows installers menu. Select Bootice tool.
   4. Go to BCD Edit. Select other BCD File and locate <drive root>\boot\bcd file on your boot partition:
   5. Press Easy mode to edit the BCD (boot configuration data) file: NOTE: If you only have 1 installer, it's going to be loaded right after you press the menu entry you added to syslinux.cfg withouth showing any menu at all. Also, you may want to press Set as default on any menu item you want to make default. Click Close and exit Bootice tool. Exit WinSetupFromUSB as well.

6. [ Cleanup (OPTIONAL)]

   1. Now we need to delete stale files. You can skip this section if you do not care about this sort of things. Open your bootable partition in Windows explorer. Also, open the first partition on your drive (they may be one and the same).
   2. The following files and folders are needed for this disk to function (if the folder is listed, all its content needs to be preserved):

      \boot
      \efi
      \multiboot
      \WINSETUP
      \bootmgr
      

   3. The following files and folders can be safely deleted:

      .disk
      \default
      \grldr
      \menu.lst
      \plpbt.bin
      \shifthd.bat
      \usbdrive.tag
      

      Remove them at will. Additionally, Windows may create its stupid System Volume Information - you can remove that too if you're lucky. Also, this is the point where you may rename your boot partition from MULTIBOOT to any other name (I prefer simply Boot). You can do that just like with any folder through Windows explorer.

7. [ Hiding the boot partition (OPTIONAL)]

   1. Now you may want to hide your boot partition to prevent accidental alteration of any files on it. This means that it will not be visible on most systems by default when inserting your disk, but you will still be able to boot from it. Open WinSetupFromUSB again.
   2. Open Bootice tool and select Physical Disk tab if it's not selected already. Choose your disk (carefully) in the dropdown menu.
   3. Click Parts Manage button. Partition list will be shown.
   4. Find your partition in the list and press Hide button. A window confirming success will pop up. The whole thing will look like this: That's it. Your partition is no longer auto-mounted (will not have a drive letter) on most systems and files needed to boot the drive will not be altered accidentally.

Also, I prefer to keep vmount program ([3] in SOFTWARE section) to be able to quickly mount hidden partition under Windows if I need to.

That's pretty much it, if you notice any misconception or error, feel free to post it here. Also, questions and feedback are welcome.

Credits:

Wonko, erwan.l, ilko, Steve - members of reboot.pro community who helped me tremendously in this thread. This is where the journey ended :)

Bob who initially hinted about ImDisk driver in this room.

Mario who educated about the BCD files, their purpose and means to edit them in this original question. This is where the journey started.

Ricky who fixed an error in indentation.