Nuke and pave of pfSense on the SG-2440

I may not be the first to deal with the fallout of filesystem corruption on an SG-2240 running pfSense 2.2 due to sudden power loss, but I might be the first to put the cliff notes of the recovery process in one place.

The first obvious symptom of trouble is the web admin throwing http 500 and 503 errors. Research reveals that this problem is not completely rare, and is often caused by unclean shutdowns resulting in filesystem damage. If you’re feeling unsettled about why a tiny fan-less network appliance such as this would be so grumpy about power loss, be advised that pfSense has another mode where things can lose power safely because the ‘non-volatile’ file systems like / are mounted r/o, and volatile ones like /var/log, /tmp, /var/run are on ramdisk. The SG-2240 does not use this mode by default, it uses the ‘full install’ mode, which behaves much more like a standard FreeBSD system, so you’re supposed to shut it down like a nice person. Read up on the difference between the pfSense “full install” and “nanobsd” configurations.

Since the web admin is dead, to diagnose this further we’ll use the console port.

Accessing pfSense console port from OS X

• Download and install CP-210x driver from: http://www.silabs.com/products/mcu/Pages/USBtoUARTBridgeVCPDrivers.aspx
• Connect USB cable between the mac and pfsense’s console port

• sudo cu -l /dev/cu.SLAB_USBtoUART -s 115200

• Press enter. You should have a root shell.

Accessing pfSense console port from Windows

• Connect mini-usb cable between pfsense console port and windows machine.
• install USB to COM bridge driver found here: http://www.silabs.com/products/mcu/Pages/USBtoUARTBridgeVCPDrivers.aspx
• open Device Manager -> Ports
• locate Silicon Labs USB bridge COM listing. Note the number after COM, e.g. COM3
• boot firewall
• fire up Putty, make a new serial connection with a speed of 115200 using the COM port discovered previously
• press enter. You should have a root shell.

Once consoled in, I ran /etc/rc.initial to use the ‘Restart PHP-FPM’ command to try to reboot the web stuff, as I read this worked for some folks. It emitted some nonsense about not knowing what the wheel group means. A cursory glance around town shows that /etc/group, /etc/passwd, and /etc/master.passwd are all munged. Not good.

Reinstall pfSense

• Download a memstick image from pfsense. Choose the ‘netgate’ option from the Computer Architecture menu, since apparently the SG-2440 is a netgate.
• Prepare a USB stick with install media.
  • Insert a USB stick (into your workstation, in this case a mac) that you don’t mind erasing.
  • If any filesystems on the usb stick are mounted, unmount them (but do not eject the device) – you can do this with Disk Utility by selecting the volumes and clicking “Unmount”.
  • Find the USB stick device number with: diskutil list
  • Wipe the partition table on the USB stick with dd:

    sudo dd if=/dev/zero of=/dev/disk3 bs=1m count=1

    (assuming the USB stick is /dev/disk3)

  • Copy the image to the device:

    gzcat pfSense-memstick-ADI-2.2.2-RELEASE-amd64.img.gz | \
    sudo dd of=/dev/disk3 bs=16k

• Eject the USB stick and insert it into one of the pfSense USB ports
• Boot the pfSense box.
• Shortly after boot, you are prompted to press F12 if you want a boot menu. Do it.
• You should now see a list of storage devices; select the USB stick.
• Let the next menu pass you by; don’t choose anything:
• After a bit more booting, you will be given the chance to press ‘i’ to run the installer. Do that.
• From the next menu, accept the console settings.
• Choose “Custom Install”
• Select the Generic Ultra HS-Combo Disk as the target for the installation
• Choose “Format this Disk”
• Choose “Use this Geometery”
• Format da1
• Skip the custom partitioning step
• Accept and install Bootblocks
• Select the internal drive
• Accept and Create
• Watch the progress window
• Embedded Kernel
• Reboot
• No VLANs
• Name each of the four network interfaces igb0, igb1, igb2, igb3 for WAN, LAN, OPT1, OPT2 respectively.
• Type ‘y’ to finish.

...
pfSense (pfSense) 2.2.2-RELEASE amd64 Mon Apr 13 20:10:22 CDT 2015
Bootup complete
FreeBSD/amd64 (pfSense.localdomain) (ttyu1)
*** Welcome to pfSense 2.2.2-RELEASE-pfSense (amd64) on pfSense **
 WAN (wan) -> igb0 -> 
 LAN (lan) -> igb1 -> v4: 192.168.1.1/24
 OPT1 (opt1) -> igb2 -> 
 OPT2 (opt2) -> igb3 -> 
 0) Logout (SSH only)               9) pfTop
 1) Assign Interfaces              10) Filter Logs
 2) Set interface(s) IP address    11) Restart webConfigurator
 3) Reset webConfigurator password 12) pfSense Developer Shell
 4) Reset to factory defaults      13) Upgrade from console
 5) Reboot system                  14) Enable Secure Shell (sshd)
 6) Halt system                   15) Restore recent configuration
 7) Ping host 16) Restart PHP-FPM
 8) Shell
 
Enter an option: