A while ago I manged to screw up my ZFS pool by having it use files on other ZFS pools as cache and slog devices. After 6 different versions/distrobution/releases,  I have given up, and recreated pools that are much saner in layout.

The filesystem now runs Nexenta community edition

With the following pools
250GB root pool or syspool as Nexenta calls it
3x 500GB in a mirrored layout in a pool called tank I may break it into a 2 way mirror later. And then create another 2x 500GB pool or zdev on the pool, currently this pool is mostly being used for ESX nfs storage. I wanted to go 3 drive wide mirror for read speed and improved write performance over raidz.

I was going to use SXCE on the system but during the install it hung at 18%, so I gave up and went with Nexenta since I had the disc sitting next to the machine anyway. I am enjoying it mostly did have to add another repository to get bind9 build for it, but that plus about 15 minutes of configuring it I was able to get nexenta to be a DNS server, thus making NFS happy.

Future:

Well I don?t expect to be using this fileserver long, since I want to move to a all in one box solution for ESXi+ZFS using an HP ML350g6, but this will allow me to add more storage to my existing ESXi 3.5 box, its built in 60GB is showing its age with its SCSI drives that are expensive for anything larger than 36GB and use lots of power in return. I will be picking up components to go in the ml350 that can live in this box till the main (expensive) bits are ready.
I will probably get 4x 2TB drives and put them in this box perhaps even the ssd?s for l2arc/slog can go in this box as well.