Ben’s Blog

Acroread doesn’t work by default on OpenSolaris.  I got it to work on the b134 preview by giving it the privilege that it requires.

By default acroread (version 9.x on x86) will do the following.

terminate called after throwing an instance of 'RSException'

That doesn’t tell us much.  There are similar error messages reported for Linux also if you do a search.  To find the actual problem use the ppriv command like so,

% ppriv -e -D acroread

acroread[6810]: missing privilege "proc_priocntl" (euid = XXXX, syscall = 112) needed at secpolicy_setpriority+0x24

terminate called after throwing an instance of 'RSException'

So that tells us that it needs the proc_priocntl permission.  Now there are some pages that explain how to add that privilege for a user so they can run acroread.  That doesn’t fit my security model and has other problems.  So instead I gave the acroread command the privilege so any user can run it.  I couldn’t find this documented anywhere which is why I’m writing this now.

I modeled my solution after the way cdrecord is given permission to work.  I added the following line to /etc/security/exec_attr

Basic Solaris User:solaris:cmd:::/usr/local/bin/acroread.bin:privs=proc_priocntl

The acroread.bin file is actually a script that runs acroread.  We are giving it the proc_priocntl privilege.  /usr/local/bin/acroread is just a script that looks like the following:

#!/bin/sh

exec pfexec /usr/local/bin/acroread.bin "$@"

So that’s how to give acroread permission to work correctly so you don’t need to give users a privilege that they don’t need.

I recently investigated setting up Sun Rays using kiosk mode using the Windows Sun Ray connector.  Using the connector from Solaris is pretty easy as the client program utssc is basically an RDP client.  Actually, rdesktop could also be used from Solaris in the same way.  However, I had to set up kiosk mode so the Sun Ray would go directly to a Windows server.  I followed the docs on setting up kiosk mode (utconfig -k, it kept failing until I ran it under truss then it succeeded…).  Most of the docs talk about using a GUI, which I do not use and setting up kiosk for smart cards or non smart card users, etc.  What I need to do was set up one specific Sun Ray to use kiosk mode, which was a little harder to find.  To do that I ran the following:

/opt/SUNWut/sbin/utkioskoverride -s kiosk -r pseudo.TOKEN -c uttsc

Where TOKEN is the mac address for the specific Sun Ray.  I also had to run ‘utkiosk -i uttsc -f uttsc.conf’  In uttsc.conf I had to add KIOSK_SESSION=uttsc and of course change KIOSK_SESSION_ARGS to set the Windows server to connect to.  This took me a little while to get used to, but it worked well.  I found this web site useful.

Then I was tasked to come up with a grub-like menu so users could select Solaris (or Linux) or Windows.  This was a bit more work.  Most of the solutions seem to talk about using the GUI and having the Sun Ray admin put things in to change what the Sun Ray does for different tokens.  I ended up taking ideas from two implementations to come up with a solution that provides a grub-like menu for Sun Rays.

One site that is recommended is http://blogs.sun.com/danielc/entry/meta_kiosk_how_to_run on “Meta Kiosk”  which I looked at and it gave me the idea of using Xephyr.  I also found http://blogs.sun.com/mplona/entry/customized_sun_ray_kiosk_sessions which gave me the idea of using zenity and gave some code to follow (I looked at menu2).  Neither solution did what I wanted so I wrote my own kiosk session script.  My proof-of-concept so far is much smaller than those solutions, but seems to work for me.  Here is my script:

#!/bin/sh
# install this as /etc/opt/SUNWkio/sessions/menu/menu

# Add a line to set the background image, then display the menu
choice=`/usr/bin/zenity  --width=200 --height=200 --list --column  "Select  Operating System" "Solaris" "Windows"` 

case "${choice}" in
    "Solaris")
     # This part was inspired by meta-kiosk-session's run-X11-session
     # set servername of course
     display=0
     while [ $display -ne 256 ]
     do
       display=`expr $display + 1`
       /usr/X11/bin/Xephyr :${display} -once -query ${servername} -fullscreen
       if [ $? -ne 1 ]
       then
         exit
       fi
     done
     ;;
    "Windows")
    # Need KIOSK_SESSION_ARGS set correctly for server to connect to
     . /etc/opt/SUNWkio/sessions/uttsc/uttsc
     exit
    ;;
 esac

This can be used to connect to any Solaris or Linux system that would accept the X11 connection.  So instead of listing Operating systems maybe you would want to list several hosts for example.

I called this the menu config and then changed the Sun Ray from using uttsc to menu with  utkioskoverride.  The script goes in  /etc/opt/SUNWkio/sessions/menu/ and the following goes in /etc/opt/SUNWkio/sessions/menu.conf

KIOSK_SESSION=menu

KIOSK_SESSION_LIMIT_VMSIZE=2000000

KIOSK_SESSION_ARGS=-t 1800 -- -r sound:low SERVERNAME

KIOSK_SESSION_EXEC=$KIOSK_SESSION_DIR/menu

I also loaded that file with ‘utkiosk -i menu -f menu.conf’.  And it works.  It presents a little menu where you can select and then what is put in the case statement runs.  I then specified some other Sun Rays to run in kiosk mode using this session config and they are being tested.  One bug found so far is that sound doesn’t work.  Also, USB devices do not really work using this approach.  For example, plugging in a USB key drive the drive gets mounted by the random untrusted kiosk user and the user that logs on through Xephyr doesn’t have permission to it (maybe a work around exists for that?).  But if those things are not important and you and have the need for a grub-like menu to select the OS on a Sun Ray it seems to do the job.

I tested this on a Sun Ray server runing SXCE b111 with version 4.1 of the SunRay server software and version 2.1 of the windows connector (SRWC).

I just did a small speed test with gtar on an x4540 (thor) system that will become our replication server.  I wanted to see how much adding SSDs for the zil would help.  The system has 500GB disks arranged in four raidz2 vdevs of 10 disks each.  For my test I unpacked mysql-5.1.36.tar.gz twice in each configuration.

My first test was to do the test locally on the system to get a baseline.  I used ptime to time how long ‘gtar xzf’ on mysql-5.1.36.tar.gz took.  Locally with no ssd the times were 1.62 seconds and 1.56 seconds.  I then added two ssds as mirrored slogs and the local times were 1.49 and 1.50.  So I have baseline performance numbers to compare against and I then removed the log devices.  This test is using OpenSolaris b134, btw.

I then mounted the filesystem on the test x4600 with NFS over a 10 gigabit ethernet link.  The times to unpack the file with no ssd configured were 7:05.86 and 6:53.76 (around 7 minutes).  I then configured the mirrored slogs again and the times went down to 2:26.73 and 2:28.62, so under 2.5 minutes.  That’s a pretty good improvement.    I just did the same test again after adding part of the SSDs as L2ARC read caches.  I didn’t expect much change and there wasn’t as the times were 2:19.22 and 2:24.29.

The SSDs used were Crucial 256GB RealSSD C300.  They are MLC so they should be good for L2ARC, but they also improved the synchronous writes quite a bit.  I’d like to test a couple of SLC SSDs like the Intel X25-E which are supposed to be better for slogs.  I plan to use dedup on this system so the L2ARC should help.  If this system ends up serving out iSCSI the log devices for zil also should help.

Turns out the workaround I did for the ntpd problem in dom0 when running the xVM hypervisor doesn’t work consistently (or since we when to DST it is acting differently).  There is a combination of using ‘rtc -z US/Eastern’ and ‘rtc -z UTC’ where I can get ntpdate to set the time correctly and have ntpd keep it correctly.  However, it seems non-deterministic if it will work on reboot or not.  So now that we are in DST the clock may jump ahead 4 or 8 hours on reboot and need to be fixed.  The open bug for this is http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6908973.

Also, there is another thing I changed from the defaults on OpenSolaris.  Network automagic (nwam) is cool for a laptop where you may go back and forth between wired and wireless (I’ve used it some for my laptop), but I don’t see why that is the default and it’s recommended to keep it turned on.  For servers of course I want to use static IPs rather than dhcp.  NWAM does have a way to set static IPs (through /etc/nwam/llp), but I much prefer the old way of using /etc/hostname.INTERFACE files.  So I disabled the network/physical:nwam service and enabled network/physical:default to get the previous behavior.  I also have to put files first for hosts in /etc/nsswitch.conf for an nfs mount in /etc/vfstab to work which I didn’t need to before.

Zones (or containers) were first introduced in Solaris 10.  Once ZFS came out (using a zfs dataset per zone is much better than using UFS with zones) deploying multiple zones became a great way to provide several virtual machines on servers.  One of the features that I liked best about zones Sun called “sparse root zones” where most software is shared between the global (bare metal) zone and the non-global zones.  This was done by having read-only loopback mounts for /lib, /platform, /sbin and /usr (I also added /usr/local to the list, others may want /opt that way), which saved lots of disk space and makes maintaining zones a snap.  Using sparse root zones is optional and you could also do the other extreme where every zone has local copies of those mount points which is called a whole root zone.  Or maybe just /usr needs to be local and keep the others as read-only loopback mounts, the administrator could decide.

The OpenSolaris distribution does not support sparse root zones!  All zones need to be whole root zones.  This is because of the IPS package manager (pkg command).  See my previous blog entry for what I think of package managers.  This is a major defect and has been since IPS was added.  The pkg command cannot handle managing software in all the zones on a system from the global zone is the reason sparse root zones are not supported.  Well, I don’t want it to.  I just want the software that is installed to be available with read-only loopback mounts in the zones.  The supported way is to maintain packages in each zone (treating each one like a little Linux machine essentially).  That would be fine if you have a need for such a setup, but I want to have 20-30 zones on a system with all the same /lib, /sbin, /platform and /usr, so there is no need to waste time managing #!$*&^# packages in each one.

ZFS dedup could be used to save space, when having several zones, but that’s not the same as each zone still needs to be managed and updated separately.  Have I mentioned I hate package managers?  Sparse zones work great, were a great idea and it goes against the Sun way to remove things.  One of the great things about Solaris is backwards compatibility, but with OpenSolaris the idea is to be more like Linux (in good and bad ways) by breaking what used to work great!

Oracle listen up.  Sparse root zones are a great feature and should not be dropped from the production release that follows Solaris 10.  They should be added back to OpenSolaris.  Do a quick web search for “opensolaris sparse zones not supported”  or check the zones mailing list and see how people have been complaining about this for quite awhile.

There is a work around.  Simply edit the xml config file (the one with the dire warning to not edit – I have the tendency to like vi better than configuration editors and edit files like that often) and add in the sparse root zone entries (inherited-pkg-dir).  The zonecfg config will not allow you to add  inherited-pkg-dir entries anymore (it even refused to add my /usr/local entry which pkg doesn’t need to be concerned with.  I could have added /usr/local a different way though using ‘filesystem special=…’.).  However, the backend will still parse the entries and the directories get mounted as desired.  Then shutdown the zone and remove the files from /lib, /platform, /sbin and /usr to save lots of space, reboot the zone and test that things work.

I did some research on this first and other people suggested the same thing.  People then reply saying that it is not supported and that pkg will not work and unpredictable things will happen and…  Well, if I never run pkg in the zone what will break?  I do not need to run pkg in the zones.  I know how to manage things correctly myself better.  As long as the backend doesn’t get broken it should be fine.

You say that pkg needs to be ran to update things in /etc and /var or what ever.  Yeah, whatever.  I never used the SVR4 package commands either and I could update systems running over 20 zones without a problem.  The only things I found that I ever had to update (between several different upgrades) were SMF things in /var/svc/ and /etc/svc/.   I think that will continue to work without running pkg (I much prefer sdist or rsync to keep files updated and in sync).

Another thing I found out that was different in OpenSolaris was that zones require two ZFS datasets instead of just one as I was used to (the root directory gets its own dataset now).  It took me a little while to figure out how to bring up a new zone by hand.  I prefer to know how things work and be able to do it by hand rather than just run some command to magically do everything.  It turned out that the zfs dataset for the root filesystem need a couple of special entries.

One of the strangest things about this new root filesystem for zones was that it a legacy mount.  I was curious how they get mounted.  I unmounted the root filesystem from my test zone, disabled the zone service and then re-enabled it to find the filesystem was mounted.  I looked at the SMF startup script for the zones service and found that zoneadm was being called with the sysboot option for each zone.  That option isn’t documented in the help for zoneadm or the man page.  So I unmounted a zone root and ran zoneadm with the sysboot option for the zone to find the filesystem gets mounted.  That’s cool I thought.  Since the sysboot option is undocumented it probably shouldn’t be depended on as it may change.  I checked the zoneadm source and a comment says this is an undocumented interface for use from SMF.

Here’s an example of what a zone looks like now.

[/zones/template/]# df -h . root
Filesystem             size   used  avail capacity  Mounted on
pool1/zones/template   47G   113M   911M    12%    /zones/template
pool1/zones/template/ROOT/zbe 47G   115M    45G     1%    /zones/template/root

And here are the local zfs entries.

# zfs get all pool1/zones/template pool1/zones/template/ROOT/zbe | grep local
pool1/zones/template/ROOT/zbe  canmount                        noauto                                local
pool1/zones/template/ROOT/zbe  org.opensolaris.libbe:active    on                                    local
pool1/zones/template/ROOT/zbe  org.opensolaris.libbe:parentbe  someUUID  local

Notice the zbe dataset that gets mounted as the zone root now.  The someUUID part is important (otherwise the zone will not boot).  To get that value install a zone with zonecfg the supported way and use the value it creates.  So the way I’m going to deploy zones on OpenSolaris systems is install one the supported way, wait awhile so it can stupidly download all the packages from a repository, set it up, then halt, setup the sparse root and reboot.  After setting up one then create others by hand by creating clones of the zfs datasets, etc.

I also tested bring up a zone root from SXCE b130 on OpenSolaris b134 and it worked fine.  I updated the SMF stuff so it was in line with b134.  I had to mess with the zfs parameters I mentioned above for it to boot after creating a data set for root.

Anyway, I think I have found a way that I can support sparse root zones on OpenSolaris even if they are not officially supported.  I think using dedup may be useful for things that need to be local for zones, but I prefer the read-only loopback mounts for software in most cases.  The IPS package system needs to grow up and learn to deal with sparse root zones or just don’t use it!

I recently upgraded mpich2 to version 1.2.1 on Solaris systems (sparc and x86).  There were some problems initially where some header files were not found.  I eventually fixed the problem by fixing a bashism that was in one of the configure scripts.  In src/mpid/ch3/configure I changed the line

export MY_EXPORTED_LIBS="$LIBS"

to be the two lines

MY_EXPORTED_LIBS="$LIBS"
export MY_EXPORTED_LIBS

I found this out after doing some web searches which led me to

https://lists.mcs.anl.gov/mailman/htdig/mpich-discuss/2010-January/006388.html

which offered that fix.   After that fix it compiled fine.  I use the Sun Studio compilers and here is the configure line:

./configure --prefix=/usr/local/mpich2 --enable-sharedlibs=solaris-cc --enable-threads --enable-cxx --disable-f77 --disable-f90

I tested that “hello world” worked on multiple clients for all three platforms (Solaris sparc and x86 plus Linux x86_64) I compiled it on.