Ubuntu 13.04 on a USB flash drive, and merging its persistent storage

I’ve got a new whizzy-fast USB3 flash drive and one of the best bits about it is booting into a live Ubuntu in about 12 seconds flat. Carrying around a full desktop environment is great, especially one that takes up very little space and can persist its storage, but one of its potential pitfalls is that it only has a limited amount of disk space that it can write to.

Pitfalls

With the version of the 13.04 iso I downloaded, I had to take a couple of unusual steps to get Ubuntu working properly from USB.

The first was when using usb-creator-gtk to create a bootable USB drive from the Ubuntu iso, I had to run it as root and allow writing to system internal drives, otherwise it segfaulted after doing the bulk of the copying. (https://bugs.launchpad.net/ubuntu/+source/usb-creator/+bug/859539)

$ sudo usb-creator-gtk --allow-system-internal 

It might not be obvious, but the bootable USB drive actually has more than one bootloader. When it boots using UEFI, it uses grub, when it uses the older style BIOS booting, then it uses syslinux. At the version I was running, I had to manually edit grub’s boot menu to pass in the ‘persistent’ kernel parameter, which is easy enough but not entirely obvious. (https://bugs.launchpad.net/ubuntu/+source/usb-creator/+bug/1159016)

Merging persistent storage

The root filesystem that boots into the live environment is a read only squashfs that takes up about 740Mb of space on the drive in a file called casper/filesystem.squashfs. squashfs is very good, but it is read-only, so to store your changes, Ubuntu uses overlayfs to layer a normal writable filesystem on top of the read-only one, that’s stored in /casper-rw.

This works really well for casual use, but it can use up writable space at a surprising rate, especially if you delete a big package from the read-only filesystem (such as LibreOffice) or patch your system up to date with an apt-get upgrade.

The good news is that you can use another Ubuntu installation to merge the changes from the writable filesystem back into the read-only squashfs, then start afresh.

This all seems entirely scriptable, but you wouldn’t just download a random script from a stranger on the internet and run it as root would you? So for now, this is how I did it:

Change to root, copy the files from USB to a temporary area and mount them as filesystems

$ sudo -s
# cd /var/tmp
# mkdir usblive
# cd usblive
# cp /media/dav/C35B-7E7E/casper-rw .
# cp /media/dav/C35B-7E7E/casper/filesystem.squashfs .
# mkdir readonly readwrite merged
# mount filesystem.squashfs readonly -o loop,ro
# mount casper-rw readwrite -o loop,rw
# mount none merged -o lowerdir=readonly,upperdir=readwrite -t overlayfs

If this has all worked then you should now see something like this:

# df -h
Filesystem          Size  Used Avail Use% Mounted on
< snip >
/dev/sde1            30G  4.8G   26G  17% /media/dav/C35B-7E7E
/dev/loop0          738M  738M     0 100% /var/tmp/usblive/readonly
/dev/loop1          4.0G  1.2G  2.6G  33% /var/tmp/usblive/readwrite
none                4.0G  1.2G  2.6G  33% /var/tmp/usblive/merged

Create a new squashfs based on the merged filesystem, this can take a little time depending on your CPU.

# apt-get install squashfs-tools
# mksquashfs merged newfilesystem.squashfs
Creating 4.0 filesystem on newfilesystem.squashfs, block size 131072.
[=========================================================\] 108641/108641 100%
Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 131072
	compressed data, compressed metadata, compressed fragments, compressed xattrs
	duplicates are removed
Filesystem size 1069333.62 Kbytes (1044.27 Mbytes)
	39.06% of uncompressed filesystem size (2737455.95 Kbytes)
< snip >

If you’re patient, or have a very fast cpu (hello Core i7), you can use a higher compression technique to save some space, at the cost of it taking longer to create:

# mksquashfs merged newfilesystem.squashfs -comp xz
Parallel mksquashfs: Using 8 processors
Creating 4.0 filesystem on newfilesystem.squashfs, block size 131072.
[=========================================================|] 108641/108641 100%
Exportable Squashfs 4.0 filesystem, xz compressed, data block size 131072
	compressed data, compressed metadata, compressed fragments, compressed xattrs
	duplicates are removed
Filesystem size 934018.40 Kbytes (912.13 Mbytes)
	34.12% of uncompressed filesystem size (2737455.95 Kbytes)
< snip >

OK, we’re nearly there, we just need to regenerate the filesystem’s size and manifest just in case we want to install from this USB stick

# umount merged readwrite readonly
# mount newfilesystem.squashfs readonly -o loop,ro
# chroot readonly dpkg-query -W --showformat='${Package} ${Version}\n' > newfilesystem.manifest
# printf $(du -sx --block-size=1 readonly | cut -f1) > newfilesystem.size 

Now we copy the new files back onto the USB flash drive and reset the writable overlay filesystem

# cp newfilesystem.squashfs /media/dav/C35B-7E7E/casper/filesystem.squashfs 
# cp newfilesystem.size /media/dav/C35B-7E7E/casper/filesystem.size 
# cp newfilesystem.manifest /media/dav/C35B-7E7E/casper/filesystem.manifest
root@newt:/var/tmp/usblive# mkfs -t ext3 /media/dav/C35B-7E7E/casper-rw 
mke2fs 1.42.5 (29-Jul-2012)
/media/dav/C35B-7E7E/casper-rw is not a block special device.
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
< snip >

and remember to clean up after ourselves

# cd
# rm -rf /var/tmp/usblive

then rebooting back into the live USB drive, we can see that we’ve now got the full size to use for writing again

ubuntu@ubuntu:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/cow            4.0G   92M  3.7G   3% /
/dev/sdi1        30G  5.0G   25G  17% /cdrom
/dev/loop0      913M  913M     0 100% /rofs
< snip >

References

I got most of the way there myself by starting with seeing that the usb environment used overlayfs, then had a look around the flash drive to see what file system images were where, but I wouldn’t have got the package manifest without the Ubuntu docs, the example mount for overlayfs saved some time from the Ubuntu forum post, and it was good to see that I could have googled a little bit harder when I started this afternoon and just used’s Jan Varho’s excellent method instead. Oops.

One thought on “Ubuntu 13.04 on a USB flash drive, and merging its persistent storage

Comments are closed.