I’ve got a new whizzy-fast USB3 flash drive and one of the best bits about it is booting into a live Ubuntu in about 12 seconds flat. Carrying around a full desktop environment is great, especially one that takes up very little space and can persist its storage, but one of its potential pitfalls is that it only has a limited amount of disk space that it can write to.
Pitfalls
With the version of the 13.04 iso I downloaded, I had to take a couple of unusual steps to get Ubuntu working properly from USB.
The first was when using usb-creator-gtk to create a bootable USB drive from the Ubuntu iso, I had to run it as root and allow writing to system internal drives, otherwise it segfaulted after doing the bulk of the copying. (https://bugs.launchpad.net/ubuntu/+source/usb-creator/+bug/859539)
$ sudo usb-creator-gtk --allow-system-internal
It might not be obvious, but the bootable USB drive actually has more than one bootloader. When it boots using UEFI, it uses grub, when it uses the older style BIOS booting, then it uses syslinux. At the version I was running, I had to manually edit grub’s boot menu to pass in the ‘persistent’ kernel parameter, which is easy enough but not entirely obvious. (https://bugs.launchpad.net/ubuntu/+source/usb-creator/+bug/1159016)
Merging persistent storage
The root filesystem that boots into the live environment is a read only squashfs that takes up about 740Mb of space on the drive in a file called casper/filesystem.squashfs. squashfs is very good, but it is read-only, so to store your changes, Ubuntu uses overlayfs to layer a normal writable filesystem on top of the read-only one, that’s stored in /casper-rw.
This works really well for casual use, but it can use up writable space at a surprising rate, especially if you delete a big package from the read-only filesystem (such as LibreOffice) or patch your system up to date with an apt-get upgrade.
The good news is that you can use another Ubuntu installation to merge the changes from the writable filesystem back into the read-only squashfs, then start afresh.
This all seems entirely scriptable, but you wouldn’t just download a random script from a stranger on the internet and run it as root would you? So for now, this is how I did it:
Change to root, copy the files from USB to a temporary area and mount them as filesystems
$ sudo -s # cd /var/tmp # mkdir usblive # cd usblive # cp /media/dav/C35B-7E7E/casper-rw . # cp /media/dav/C35B-7E7E/casper/filesystem.squashfs . # mkdir readonly readwrite merged # mount filesystem.squashfs readonly -o loop,ro # mount casper-rw readwrite -o loop,rw # mount none merged -o lowerdir=readonly,upperdir=readwrite -t overlayfs
If this has all worked then you should now see something like this:
# df -h Filesystem Size Used Avail Use% Mounted on < snip > /dev/sde1 30G 4.8G 26G 17% /media/dav/C35B-7E7E /dev/loop0 738M 738M 0 100% /var/tmp/usblive/readonly /dev/loop1 4.0G 1.2G 2.6G 33% /var/tmp/usblive/readwrite none 4.0G 1.2G 2.6G 33% /var/tmp/usblive/merged
Create a new squashfs based on the merged filesystem, this can take a little time depending on your CPU.
# apt-get install squashfs-tools # mksquashfs merged newfilesystem.squashfs Creating 4.0 filesystem on newfilesystem.squashfs, block size 131072. [=========================================================\] 108641/108641 100% Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 131072 compressed data, compressed metadata, compressed fragments, compressed xattrs duplicates are removed Filesystem size 1069333.62 Kbytes (1044.27 Mbytes) 39.06% of uncompressed filesystem size (2737455.95 Kbytes) < snip >
If you’re patient, or have a very fast cpu (hello Core i7), you can use a higher compression technique to save some space, at the cost of it taking longer to create:
# mksquashfs merged newfilesystem.squashfs -comp xz Parallel mksquashfs: Using 8 processors Creating 4.0 filesystem on newfilesystem.squashfs, block size 131072. [=========================================================|] 108641/108641 100% Exportable Squashfs 4.0 filesystem, xz compressed, data block size 131072 compressed data, compressed metadata, compressed fragments, compressed xattrs duplicates are removed Filesystem size 934018.40 Kbytes (912.13 Mbytes) 34.12% of uncompressed filesystem size (2737455.95 Kbytes) < snip >
OK, we’re nearly there, we just need to regenerate the filesystem’s size and manifest just in case we want to install from this USB stick
# umount merged readwrite readonly
# mount newfilesystem.squashfs readonly -o loop,ro
# chroot readonly dpkg-query -W --showformat='${Package} ${Version}\n' > newfilesystem.manifest
# printf $(du -sx --block-size=1 readonly | cut -f1) > newfilesystem.size
Now we copy the new files back onto the USB flash drive and reset the writable overlay filesystem
# cp newfilesystem.squashfs /media/dav/C35B-7E7E/casper/filesystem.squashfs # cp newfilesystem.size /media/dav/C35B-7E7E/casper/filesystem.size # cp newfilesystem.manifest /media/dav/C35B-7E7E/casper/filesystem.manifest root@newt:/var/tmp/usblive# mkfs -t ext3 /media/dav/C35B-7E7E/casper-rw mke2fs 1.42.5 (29-Jul-2012) /media/dav/C35B-7E7E/casper-rw is not a block special device. Proceed anyway? (y,n) y Filesystem label= OS type: Linux < snip >
and remember to clean up after ourselves
# cd # rm -rf /var/tmp/usblive
then rebooting back into the live USB drive, we can see that we’ve now got the full size to use for writing again
ubuntu@ubuntu:~$ df -h Filesystem Size Used Avail Use% Mounted on /cow 4.0G 92M 3.7G 3% / /dev/sdi1 30G 5.0G 25G 17% /cdrom /dev/loop0 913M 913M 0 100% /rofs < snip >
References
I got most of the way there myself by starting with seeing that the usb environment used overlayfs, then had a look around the flash drive to see what file system images were where, but I wouldn’t have got the package manifest without the Ubuntu docs, the example mount for overlayfs saved some time from the Ubuntu forum post, and it was good to see that I could have googled a little bit harder when I started this afternoon and just used’s Jan Varho’s excellent method instead. Oops.
I’ve since started writing a quick bash script that runs through this process, it’s only an initial draft so is missing most of the required safety checks, but it works for me.
https://github.com/davstott/ubuntu-usb-merge