Tuesday, January 13, 2015

Introducing UpdateDay patching

In the school district where I work, we have a large number of laptop carts. Day to day usage of these carts varies greatly. Most are used often at some locations while at the others they will sit in the carts for days or weeks at a time. We are also dealing with a legacy carryover - generic accounts. The transition away from these generic accounts has been slow, mainly due to support issues (i.e. we have a small support staff). Since the laptops are often used with the same account, users find it easy to just close the lid when they put them away at the end of their session. The result of this is that we have laptops that are often not logged-out or shutdown for months at a time. Also, they often leave browsers and other applications open. Patching these computers has become a nightmare. We’ve been using Casper for a few years to deploy software, but one of the drawbacks to using it as it comes out of box its that it will, if you’re not careful, install software while a user is logged in and using the software. This has caused us to receive numerous angry emails and phone calls telling us how evil we are for disrupting their work and ruining their computer. Casper does have the ability to install on startup/shutdown or on login/logout but as I’ve stated above, neither of these happen consistently in our environment. To combat this, we were looking for a method to deliver updates to our users.

Additionally, we will be taking part in the upcoming PARCC online testing later in the school year. As a part of the preparation for these tests we have had to verify that each computer has the prerequisites for the test.  In order to prove this, at a glance, we create reports in the JSS. We have Extension Attributes that display the requirements along with other information.  We periodically distill this information into reports that we provide to upper management to calm their fears that we will not be ready in time.

After watching Greg Neagle demonstrate AutoPkg last summer at the Penn State MacAdmins conference, we began using AutoPkg with Alister Banks’ JSS-autopkg-addon last summer. For those that are unfamiliar with it AutoPkg is a software package that can be used to download commonly used software updates and package them into an easily deployable format. So now I have in our JSS the latest versions of the software that we use daily. I just needed to find a way to deploy it in a consistent manner.

I discovered the final piece to the puzzle when Rich Trouton released his First-Boot-Package-Install package. This packages, after a restart, will iterate through a series of folders and install the software packages that it finds.

The Solution
Phase 1: Staging the Update Packages

The first part of the solution we needed to find a method to copy the software packages into a “secret location” on the computer and store them in folders to be used by the first boot packages. In order for this to work properly, the packages need to be stored in a numbered folder system. To accomplish this I developed the script below:

#!/bin/bash

pkgname=$4
echo "Package is "$4
basedir=‘/secretlocation/'
waitingroom='/Library/Application Support/JAMF/Waiting Room/'
source=$waitingroom$pkgname

echo "Cached file is "$source

for tens in {0..9}
do
    for ones in {0..9}
    do
        foldernum=$tens$ones
        update_dir=$basedir$foldernum
        #echo $foldernum,$update_dir
        if [ ! -d "${update_dir}" ]; then
            mkdir -p ${update_dir}
            break 2
        fi
    done
done

destination=$update_dir/$pkgname
echo "The destination is "$destination

# Move the cached package into the appropriate folder
mv "$source" "$destination"

# Remove the Casper receipt (XML) file
echo "Cleaning Casper receipt" $source".cache.xml"
rm "$source".cache.xml
exit 0

A Casper policy is used to cache the package to the computer. A numbered folder gets created and the packages get moved into the next folder. Then we remove the Casper "receipt" file so that it doesn't think that something is there to install. In the below picture you can see that I have it set to install via a custom trigger during testing. This trigger will be maintained during deployment as you will see later. Pre-deployment we will use the recurring trigger to stage the packages prior to the UpdateDay.




  1. A numbering scheme is used to ensure that the packages get installed in the proper order.
  2. The script checks for the existence of a numbered folder in the $secretlocation and increments its counters until if finds an available one to use (00-99)
  3. The script then moves the package into the numbered folder
  4. The name of the package is passed to the script as $4. This setting is set in Casper Admin.

By setting the parameter name here it is passed to the web interface when setting up the policy. This serves as a reminder to the creator of the policy that they are supposed to enter something here.

The script then moves the package to the secretlocation. The package name is used here : (1) so that we will only move only the intended package (packages cached by any other policies are not disturbed;

The receipt file that Casper creates is deleted so that if a “install all cached packages” policy is run, it will not look for this package. This is repeated for all of the packages that we will be installing.

The last package that we install is a payload-free package that has script that runs a recon. This ensures that the inventory in the JSS has up to date information that we can use to verify the process has completed successfully.

Phase 2: Installing the Update Packages

After the packages are downloaded, we can then move to installing them. Here we’ll be using another policy and script combo called via a custom event "trigger". At deployment time we set the policy to be active at startup so that we can tell the user to simply restart the computer to get the updates. The process can also be started when the trigger is called via a "jamf policy -event updatekickoff" statement via script, command-line, or remote command. To give you a better idea, here is how the policy looks:


Here’s the update_kick.sh script that we’re using to install of the First-Boot-Package-Install packages:

#!/bin/bash
jamf='/usr/sbin/jamf'
secretloc=/secretlocation/'

# Check for FBI packages

if [ ! -f "${secretloc}fbi.pkg" ]; then
            $jamf policy -event fbi-install
     else
          if [ ! -f "${secretloc}fbiswu.pkg" ]; then
            $jamf policy -event fbi-install
        fi
fi

/usr/sbin/installer -dumplog -verbose -pkg "${secretloc}fbiswu.pkg" -target /
$jamf policy -event patching
exit 0

This script checks the $secretlocation for the First-Boot-Package-Install packages. If they are missing it uses a custom trigger, “fbi-install”, to download them to the $secretlocation and then installs the desired version. (We’re installing both versions, with and without Apple Software Updates, onto the computer so that we have them in the future.)

Then, we re-check for any packages waiting to be cached via the trigger “patching". As I’ve stated above, sometimes the laptops sit in the carts for weeks, so they won’t have an opportunity to download the updates. This way they’ll have the ability to pull down the packages. The other computers will also then be able get any last minute packages. Then, once everything is in place, the policy reboots the computer.

One of the reasons why we wanted to move away from Casper's install at startup process is that it doesn't provide any feedback to the user - the computer just sits there and appears to be frozen. Initially we wanted to provide some kind of feedback to the user during the process, but it seems that Apple has removed some of that ability in 10.9. Indeed, when we ran versions of this script using CocoaDialog, JamfHelper and system Notifications we had errors related to WindowServer not being able to start at every line of code. We're using Casper's built-in notifications to give a warning to the user if someone is logged in.

The First Boot Packages use Per Olofsson’s LoginLog  application to prevent the user from logging in by displaying a log window overtop of the login window as seen below.  This fulfills our desire to provide feedback to the user. It also prevents them from logging in until the installations have finished. After the packages have been installed, the computer restarts and is ready for use.


Phase 3:  Adding Self Service to the Process

At the outset, we knew that we would have to come up with a way of updating the staff computers as well. Given the number of and location of the computers we have, there is no way that we would be able to touch all of them. After allowing the above procedure to run for a couple of weeks successfully, we decided to see if we could extend the procedure to be run from Casper's Self Service application. We could then use this to allow our teachers and administrators to install updates at their convenience. It turned out to be easier than we initially thought, a two-step process in fact. All we had to do was add the additional computers to the update package scopes. We then duplicated the "Update Kickoff" policy, gave it an cool official sounding name (2015 Software Update 01), set it for Self Service and released it to our staff.

Lessons Learned and Future Additions

Initially we had a “/sbin/reboot” command at the end of the script, but we quickly learned that it caused the computer to enter a continuous rebooting loop because the policy would never finish.

The first question that was asked after I sent out the instructions for how to update the staff computers was "what about software updates?" We have a Self Service item for installing system updates, but other than a progress bar with minimal text, again there is no feedback to the user. Also, the user has to continually run it to make sure that everything has been installed. As mentioned above, there is a version of the First-Boot-Package-Install packages that installs Apple Software Updates (First Boot Package Install With Automated Apple Software Update). This package will continually check for updates until there are none remaining. We will be creating a Self Service policy, based upon the one described above, to do this.  We decided to keep them separate because users sometimes become impatient when an installation seems to take "forever" to finish.  This way, they can choose what they will be updating and when it'll be done. 

We quickly discovered that missing from this process was the ability to monitor progress. Because this process is done outside of Casper, we couldn't easily use its tools to troubleshoot. We could wait until the next Recon ran, but that wouldn't give us immediate feedback, or tell us what went wrong. Fortunately the process keeps a log file. We could query the log remotely if needed, but that wouldn't give us something that we could capture and analyze easily. Since there is a log file, we're thinking that we should be able to send them to a syslog server.