Introducing UpdateDay patching

In the school district where I work, we have a large number of laptop carts. Day to day usage of these carts varies greatly. Most are used often at some locations while at the others they will sit in the carts for days or weeks at a time. We are also dealing with a legacy carryover - generic accounts. The transition away from these generic accounts has been slow, mainly due to support issues (i.e. we have a small support staff). Since the laptops are often used with the same account, users find it easy to just close the lid when they put them away at the end of their session. The result of this is that we have laptops that are often not logged-out or shutdown for months at a time. Also, they often leave browsers and other applications open. Patching these computers has become a nightmare. We’ve been using Casper for a few years to deploy software, but one of the drawbacks to using it as it comes out of box its that it will, if you’re not careful, install software while a user is logged in and using the software. This has caused us to receive numerous angry emails and phone calls telling us how evil we are for disrupting their work and ruining their computer. Casper does have the ability to install on startup/shutdown or on login/logout but as I’ve stated above, neither of these happen consistently in our environment. To combat this, we were looking for a method to deliver updates to our users.

Additionally, we will be taking part in the upcoming PARCC online testing later in the school year. As a part of the preparation for these tests we have had to verify that each computer has the prerequisites for the test.  In order to prove this, at a glance, we create reports in the JSS. We have Extension Attributes that display the requirements along with other information.  We periodically distill this information into reports that we provide to upper management to calm their fears that we will not be ready in time.

After watching Greg Neagle demonstrate AutoPkg last summer at the Penn State MacAdmins conference, we began using AutoPkg with Alister Banks’ JSS-autopkg-addon last summer. For those that are unfamiliar with it AutoPkg is a software package that can be used to download commonly used software updates and package them into an easily deployable format. So now I have in our JSS the latest versions of the software that we use daily. I just needed to find a way to deploy it in a consistent manner.

I discovered the final piece to the puzzle when Rich Trouton released his First-Boot-Package-Install package. This packages, after a restart, will iterate through a series of folders and install the software packages that it finds.

The Solution

Phase 1: Staging the Update Packages

The first part of the solution we needed to find a method to copy the software packages into a “secret location” on the computer and store them in folders to be used by the first boot packages. In order for this to work properly, the packages need to be stored in a numbered folder system. To accomplish this I developed the script below:

#!/bin/bash

pkgname=$4

echo "Package is "$4

basedir=‘/secretlocation/'

waitingroom='/Library/Application Support/JAMF/Waiting Room/'

source=$waitingroom$pkgname

echo "Cached file is "$source

for tens in {0..9}

do

    for ones in {0..9}

    do

        foldernum=$tens$ones

        update_dir=$basedir$foldernum

        #echo $foldernum,$update_dir

        if [ ! -d "${update_dir}" ]; then

            mkdir -p ${update_dir}

            break 2

        fi

    done

done

destination=$update_dir/$pkgname

echo "The destination is "$destination

# Move the cached package into the appropriate folder

mv "$source" "$destination"

# Remove the Casper receipt (XML) file

echo "Cleaning Casper receipt" $source".cache.xml"

rm "$source".cache.xml

exit 0

A Casper policy is used to cache the package to the computer. A numbered folder gets created and the packages get moved into the next folder. Then we remove the Casper "receipt" file so that it doesn't think that something is there to install. In the below picture you can see that I have it set to install via a custom trigger during testing. This trigger will be maintained during deployment as you will see later. Pre-deployment we will use the recurring trigger to stage the packages prior to the UpdateDay.

1. A numbering scheme is used to ensure that the packages get installed in the proper order.
2. The script checks for the existence of a numbered folder in the $secretlocation and increments its counters until if finds an available one to use (00-99)
3. The script then moves the package into the numbered folder
4. The name of the package is passed to the script as $4. This setting is set in Casper Admin.

By setting the parameter name here it is passed to the web interface when setting up the policy. This serves as a reminder to the creator of the policy that they are supposed to enter something here.

The script then moves the package to the secretlocation. The package name is used here : (1) so that we will only move only the intended package (packages cached by any other policies are not disturbed;

The receipt file that Casper creates is deleted so that if a “install all cached packages” policy is run, it will not look for this package. This is repeated for all of the packages that we will be installing.

The last package that we install is a payload-free package that has script that runs a recon. This ensures that the inventory in the JSS has up to date information that we can use to verify the process has completed successfully.

Phase 2: Installing the Update Packages

After the packages are downloaded, we can then move to installing them. Here we’ll be using another policy and script combo called via a custom event "trigger". At deployment time we set the policy to be active at startup so that we can tell the user to simply restart the computer to get the updates. The process can also be started when the trigger is called via a "jamf policy -event updatekickoff" statement via script, command-line, or remote command. To give you a better idea, here is how the policy looks:

Here’s the update_kick.sh script that we’re using to install of the First-Boot-Package-Install packages:

#!/bin/bash

jamf='/usr/sbin/jamf'

secretloc=‘/secretlocation/'

# Check for FBI packages

if [ ! -f "${secretloc}fbi.pkg" ]; then

            $jamf policy -event fbi-install

     else

          if [ ! -f "${secretloc}fbiswu.pkg" ]; then

            $jamf policy -event fbi-install

        fi

fi

/usr/sbin/installer -dumplog -verbose -pkg "${secretloc}fbiswu.pkg" -target /

$jamf policy -event patching

exit 0

This script checks the $secretlocation for the First-Boot-Package-Install packages. If they are missing it uses a custom trigger, “fbi-install”, to download them to the $secretlocation and then installs the desired version. (We’re installing both versions, with and without Apple Software Updates, onto the computer so that we have them in the future.)

Then, we re-check for any packages waiting to be cached via the trigger “patching". As I’ve stated above, sometimes the laptops sit in the carts for weeks, so they won’t have an opportunity to download the updates. This way they’ll have the ability to pull down the packages. The other computers will also then be able get any last minute packages. Then, once everything is in place, the policy reboots the computer.

One of the reasons why we wanted to move away from Casper's install at startup process is that it doesn't provide any feedback to the user - the computer just sits there and appears to be frozen. Initially we wanted to provide some kind of feedback to the user during the process, but it seems that Apple has removed some of that ability in 10.9. Indeed, when we ran versions of this script using CocoaDialog, JamfHelper and system Notifications we had errors related to WindowServer not being able to start at every line of code. We're using Casper's built-in notifications to give a warning to the user if someone is logged in.

The First Boot Packages use Per Olofsson’s LoginLog  application to prevent the user from logging in by displaying a log window overtop of the login window as seen below.  This fulfills our desire to provide feedback to the user. It also prevents them from logging in until the installations have finished. After the packages have been installed, the computer restarts and is ready for use.

Phase 3:  Adding Self Service to the Process

At the outset, we knew that we would have to come up with a way of updating the staff computers as well. Given the number of and location of the computers we have, there is no way that we would be able to touch all of them. After allowing the above procedure to run for a couple of weeks successfully, we decided to see if we could extend the procedure to be run from Casper's Self Service application. We could then use this to allow our teachers and administrators to install updates at their convenience. It turned out to be easier than we initially thought, a two-step process in fact. All we had to do was add the additional computers to the update package scopes. We then duplicated the "Update Kickoff" policy, gave it an cool official sounding name (2015 Software Update 01), set it for Self Service and released it to our staff.

Lessons Learned and Future Additions

Initially we had a “/sbin/reboot” command at the end of the script, but we quickly learned that it caused the computer to enter a continuous rebooting loop because the policy would never finish.

The first question that was asked after I sent out the instructions for how to update the staff computers was "what about software updates?" We have a Self Service item for installing system updates, but other than a progress bar with minimal text, again there is no feedback to the user. Also, the user has to continually run it to make sure that everything has been installed. As mentioned above, there is a version of the First-Boot-Package-Install packages that installs Apple Software Updates (First Boot Package Install With Automated Apple Software Update). This package will continually check for updates until there are none remaining. We will be creating a Self Service policy, based upon the one described above, to do this.  We decided to keep them separate because users sometimes become impatient when an installation seems to take "forever" to finish.  This way, they can choose what they will be updating and when it'll be done. 

We quickly discovered that missing from this process was the ability to monitor progress. Because this process is done outside of Casper, we couldn't easily use its tools to troubleshoot. We could wait until the next Recon ran, but that wouldn't give us immediate feedback, or tell us what went wrong. Fortunately the process keeps a log file. We could query the log remotely if needed, but that wouldn't give us something that we could capture and analyze easily. Since there is a log file, we're thinking that we should be able to send them to a syslog server.

The road to Reposado

One of the projects that I've had in the back of my head for a while was to ditch Apple's Software Update Service (SUS) and use Reposado in its place.  We currently have several (20) servers in place. During our ongoing evaluation process we've begun to question the necessity of maintaining these servers in their current capacity. We are using them for 3 purposes: Workgroup Manager (MCX), a Casper (JSS) Distribution Point, and Software Update Services.

• MCX is being replaced by profiles.  As we move more of our computers to 10.8/10.9, it is no longer necessary to continue this service.
• We could use (virtually) any server to host our Distribution Points.
• Software Update Services only (officially) hosts updates for the same level of the host's operating system or lower. Since our servers are currently at 10.7.5 this obviously creates a problem.  We were able to use a kludge to get it to host 10.8.X updates, but this didn't work when 10.9 was released.

So, as you can see, we can possibly do without maintaining Apple branded servers in our environment. As an experiment, on a lonely Sunday afternoon, I decided to install Jamf's Netboot/SUS appliance in our virtual infrastructure. A couple hours and a few hairs later I was servering updates back through 10.4.X.  I quietly put this into production and began servicing our 10.9 clients. This would normally be fine, except that we have relatively slow links to our remote sites

Thus this past week I launched my master plan to replace Apple's SUS with Reposado.  Below is the process that I used. (Warning: This is the process that I used.  I'm sure that there are many other ways to do what I've done.)

Resources used:

• Reposado GitHub site (https://github.com/wdas/reposado)
• Margarita GitHub site (https://github.com/jessepeterson/margarita)
• Bye Bye Apple SUS (http://www.nbalonso.com/bye-bye-apple-sus/)
• Reposado Apple Software Update Server (http://signalboxes.net/mac2linux/reposado-apple-software-update-server/)
• Setting up Reposado and Margarita on Linux – Part 1 (http://macadmincorner.com/setting-up-reposado-and-margarita-on-linux-part-1/)

Preparation:

• Clone (download) Reposado and Margarita repositories.
• Download X-Code command line tools

Process:
Create Reposado working directory:
mkdir /usr/local/reposado

Copy items in reposado-master/code directory to this folder.

Create Reposado repository folders:
mkdir -p /Volumes/Data/reposado/html
mkdir /Volumes/Data/reposado/metadata

Change to the Reposado working directory:
cd /usr/local/reposado

Configure Reposado and answer the questions that come up. (Answers in bold) In this example I’ll be using port 80 to serve updates:
./repoutil --configure

Path to store replicated catalogs and updates [None]: /Volumes/Data/reposado/html
Path to store Reposado metadata [None]: /Volumes/Data/reposado/metadata/
Base URL for your local Software Update Service (Example: http://su.your.org -- leave empty if you are not replicating updates) [None]: http://myupdateserver.com 

Clone the Apple Software Update library:
./repo_sync

The initial sync will take a couple of hours. While this is running, you should see it creating files and indexes and downloading the updates into it’s file structure.  Since this is going to take a while, we can begin to prep our web server.

Open the Apache2 configuration file (/etc/apache2/httpd.conf) to make sure it’s setup properly:

•Make sure it’s listening on port 80 (around line 50?):
#Listen 12.34.56.78:80
Listen 80

•The rewrite module is enabled: (Uncomment if necessary)
LoadModule rewrite_module libexec/apache2/mod_rewrite.so

•Change the default directory. Comment out the default line and add your new directory so that it looks like below:

Change: Directory "/Library/WebServer/Documents"
to: Directory "/Volumes/Data/reposado"

•Some people have found it necessary to make sure that Overrides are enabled (I haven’t found it necessary, but ymmv)
Change: AllowOverride None
to: AllowOverride All

Close the file and edit the default port 80 configuration (/etc/apache2/sites/0000_any_80_.conf) :

Change the default directory here as well:
DocumentRoot "/Volumes/Data/reposado/html”

(This step is probably not necessary, but this is the combination was required in order for me to get this working.)

Create a .htaccess file in your Reposado repository with the following items (check the Reposado wiki for the latest). This will allow you to point all of your clients to [repositoryURL]/index.sucatalog and not worry about making changes for every operating system:

1:  RewriteEngine On  
2:  Options FollowSymLinks  
3:  RewriteBase /  
4:  RewriteCond %{HTTP_USER_AGENT} Darwin/8  
5:  RewriteRule ^index(.*)\.sucatalog$ content/catalogs/index$1.sucatalog [L]  
6:  RewriteCond %{HTTP_USER_AGENT} Darwin/9  
7:  RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-leopard.merged-1$1.sucatalog [L]  
8:  RewriteCond %{HTTP_USER_AGENT} Darwin/10  
9:  RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-leopard-snowleopard.merged-1$1.sucatalog [L]  
10:  RewriteCond %{HTTP_USER_AGENT} Darwin/11  
11:  RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-lion-snowleopard-leopard.merged-1$1.sucatalog [L]  
12:  RewriteCond %{HTTP_USER_AGENT} Darwin/12  
13:  RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog [L]  
14:  RewriteCond %{HTTP_USER_AGENT} Darwin/13  
15:  RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-10.9-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog [L]  

Let’s start the web server and  check to make everything is OK:
sudo apachectl restart

Check the log files for errors:
tail /var/log/apache2/error_log
tail /var/log/apache2/access_log


After the repo_sync has finished. point one or more of your clients to the new repo and test, test, test:
sudo defaults write /Library/Preferences/com.apple.SoftwareUpdate CatalogURL “http://myupdateserver.com/index.sucatalog”

Verify that your entry was done properly:
defaults read /Library/Preferences/com.apple.SoftwareUpdate CatalogURL
http://myupdateserver.com/index.sucatalog

Test your client:
sudo softwareupdate -l

You should see the computer checking similar to below:

Software Update Tool
Copyright 2002-2010 Apple


Software Update found the following new or updated software:
   * SecUpdSrvr2014-003Lion-1.0
     Security Update 2014-003 (1.0), 180667K [recommended] [restart]
   * SecUpdSrvr2014-002Lion-1.0
     Security Update 2014-002 (1.0), 173813K [recommended] [restart]

If you see any errors, check your CatalogURL and try again.

At the server, check the log files to ensure that no errors have occurred. I like to go back 50 or 100 lines or so to make sure I’m seeing the beginning of the process:

tail -n50 /var/log/apache2/error_log
tail -n50 /var/log/apache2/access_log

Congratulations!! You now have a functioning update server!

Now let’s schedule it to sync nightly:

Create a plist file (reposync.plist) in the Reposado working directory with the following contents:

1:  <?xml version="1.0" encoding="UTF-8"?>  
2:  <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">  
3:  <plist version="1.0">  
4:  <dict>  
5:    <key>Label</key>  
6:    <string>com.yourcompany.reposync</string>  
7:    <key>OnDemand</key>  
8:    <true/>  
9:    <key>RunAtLoad</key>  
10:    <false/>  
11:    <key>Program</key>  
12:    <string>/PATH/TO/SCRIPT</string>  
13:    <key>ProgramArguments</key>  
14:    <array>  
15:      <string>/usr/local/reposado/repo_sync</string>  
16:    </array>  
17:    <key>StartCalendarInterval</key>  
18:    <dict>  
19:      <key>Hour</key>  
20:      <integer>02</integer>  
21:      <key>Minute</key>  
22:      <integer>00</integer>  
23:    </dict>  
24:  </dict>  
25:  </plist>  

This plist will run repo_sync every night at 2:00 am. Copy it to /Library/LaunchDaemons/com.mycompany.reposync.plist:
cp reposync.plist /Library/LaunchDaemons/com.mycompany.reposync.plist

and launch it:
sudo launchctl load -w /Library/LaunchDaemons/com.mycompany.reposync.plist

Next steps:

• Installing Margarita (https://github.com/jessepeterson/margarita)
• Duplicating this process across all remaining site servers. (Might turn out to be an excuse to start using Puppet (http://puppetlabs.com/) to manage my servers)
• Implement Sean Kaiser’s Multi-Site solution (http://seankaiser.com/blog/2013/05/23/multi-site-reposado/)
• Changing the server configurations to pull from my master repo. (Just an idea, only if bandwidth utilization gets out of hand.)

Cleaning up the network monitoring

Last week I had the pleasure/honor of being able to attend the Penn State Mac Admins Conference. But, before i get to implementing what I learned there are a free things that I need to clean up.  One of the issues that I was having prior to leaving was figuring out why our servers weren't recovering properly from power outages. (Before anybody starts screaming "you should have a UPS!", we have them but you can only maintain power for so long). We tend to have seasonal power outages and brown outs at this time of year as well as people being helpful and unplugging the server racks. While things usually recover nicely, we've been having some timing issues. The servers are coming back up before the network. And since they don't have the network available, DNS and DHCP epithet don't or sort of start up. I need to come up d with a way of monitoring this with my current console. I've been using Opsview core for a couple of years to do general system monitoring. It's based on Nagios and had a semi useful web interface, so I can pretty much bend it to my will.

Post PSUMac ToDo List

Post Conference ToDo List

Last week I had the honor/pleasure of attending the MacAdministrator's Conference at Penn State University. It was a fun filled and brain filling week of techy goodness.  Below is the short version of the todo list that I came up with:

1. Cleaning up and finishing some of my deployment scripts.
2. Reposado (https://github.com/wdas/reposado) 
1. Multi-Site alterations (http://seankaiser.com/blog/2013/05/23/multi-site-reposado/) 
3. Autopkg (https://github.com/autopkg/autopkg) 
1. JSS addon (https://github.com/arubdesu/jss-autopkg-addon) 
2. Autopkg change notifications (http://seankaiser.com/blog/2013/12/16/autopkg-change-notifications/) 
4. Git (https://about.gitlab.com/gitlab-ce/) 
1. Workstation client (Sourcetree?) 
2. Server solution (Git lab??)
5. Changes to computer preference management.
1. Change "defaults write" and plist packages to profiles (http://talkingmoose.net/2014/07/12/posting-my-psu-macadmins-conference-presentation/)

Deep Thoughts for the New Year

I started this blog a few years ago as a place to put some of my thought and ideas and to document some aspects of my life. Well, this has been a major fail.  I hope to fix this this year. Now, on to today's topic.

A lot of people use the new year as a time for reflection and planning goals for the upcoming year.  I guess I'm no exception to this. I usually don't make resolutions, but look at things in my life that I could change - part of my never ending quest to be a "better" person.

Since I've been somewhat of a loner most of my life, I don't have a lot of friends. A close friend once told me that I have more friends than I'm aware of and I need to be more receptive to their friendships. Words I've tried to take to heart. I've met some interesting people in the past year, a few I plan to keep around for a while.  The others, well, they serve their purpose. Fighting the urge to retire into my shell is something that I constantly fight. While I can fake being comfortable in social situations, its not what I like to do.  Mind you, public speaking and performing is not a problem for me. The time that I spent in church during my teen years is responsible for that.

One of my flaws is that I tend to throw myself into my work. For me, work is an extension of some of my interests. Creating something functional from a pile of parts; Finding a unique solution to a problem that I didn't even know that I had; Well, you get the idea.

Over the past few years I've forced myself to get involved with a couple organizations that I didn't know existed prior.  Like minded individuals that I can bounce a few of my ideas off of. Many of them are very well known nationally worldwide, at least in some circles.  Some of them are published authors and/or renowned speakers as well. I've learned a ton from them. Although I have no idea how they can do so much with the same amount of hours in a day.  A lot of it is time management. A lot of it is sheer knowledge and recall ability.

My engineering background, as well as being raised by a basement mad scientist complete with lab coat, has given me a need to learn as much about as many different things as possible. Somewhere along the line I developed the ability to get myself up to speed quickly on new ideas and procedures. There is a lot of depth to go with the breadth mind you. I've learned through the years that others may find my knowledge intimidating, or they don't believe that I know all that I claim to.  Through talking to people, I've learned that that is a side affect of successfully working in the IT department of an educational organization.

I remember a quote that I once read that said something along the lines of the truly great minds are not afraid of mistakes as their part of the learning experience. I also know that there are others that have more knowledge, experience and resources than I have.  One of the great things about the internet is that people are more than happy to share their experiences and solutions to the problems they've faced. To this end, I've discovered a ton of blogs that I can use for my research and ideas. Why reinvent the wheel when I can take the opportunity to learn from them.

Those that I work with are completely clueless about a lot of the ideas, processes, and products that I bring forward. But, as evidenced by these blogs, they are indeed in use and practiced by many in the field. I try to not be critical of them, but they can be quite frustrating at times.

A few years ago, a former boss tried to impress upon me that I needed to stop creating my own solutions.  He was also convinced that any software that is put into production should come out of a box and be backed by a "real" company.  He equated open source software to the shareware of the 80's - unpolished, unreliable and unbacked. While this may be true of some products, it is not true for all. The internet was built, and continues to run on, mostly open source software.  Many commercial products are based on open source. In fact, many of the more successful open source products have been spun into real companies, while others have been acquired. I know someone who took a leave from work to start such a company with the ultimate goal of acquisition in mind, although 9/11 brought that to a crashing end for him. (No pun or offense intended.) What I've learned through my new contacts is that these products are in use in many of the companies that we're familiar with.

If you're unfamiliar with it, the concept of open source is very successful. It goes something like this:  I have an idea, or I have a need or problem that is not solved by off the shelf products. Or, those products don't work the way I need them to and I can't change that.  So, I'll come up with my own solution and share it with the world.  The only stipulation is that if you come up with new ways to expand it, or solve problems that are in my code, then you must give those changes back to the "community." While some may scoff at the concept of giving away their work, again, it's highly successful.

As my work responsibilities have grown and changed, I've had to take his thoughts to heart. I have had to begin using products that I can turn over to others (with little technical experience) to operate. But, the same crowd-sourced efforts that are used to further open source products can be applied to many of these products. I've tried to walk the balance between commercial products with "real" support yet give me the ability to craft my own customizations.  This usually gives me the excuse to keep my programming skills sharp as well as learning more about the systems that I support. However, I simply don't have the time to do it all myself.

I struggle to comprehend how those that I work with do not want to learn about the internals. They're stuck (barely) in maintenance mode. No thought is given to how or why something happens. Or, why does the solution work. Sometimes I take it personally. They're a continual source of aggravation for me. Both their limited skill sets and their unwillingness to grow and expand their own knowledge are baffling. Yet, they seem to be happy this way. How can they be happy not knowing all they could about their chosen profession? While I have technically passed day to day supervision of the underlings to someone else, they don't have the skills to assist them. Or the supervisory skills to keep them on task.  While they do need a task master, I still find myself getting too involved with them on a daily basis. This definitely interferes with my ability to get my own tasks done. Yes, another source of frustration.

Regardless of how I try to mask my frustration when I talk to others, it shows in some manner. Some people may think that I'm a mean, cold, uncaring person. My true friends know that this is not true. Still others tell me that I need to calm down because "it's not that important." What? If you don't think that getting your problem (that you've complained to those above me about) resolved is important enough for me to get upset over - you're part of the problem as well. More stress! I'm not proud of this. If anything I sometimes find it embarrassing. I try to be a gentlemen at all times. Well, most times. I have my moments like anyone. I believe that people have many sides. That's what shapes our personalities. Some people verbalize all of their thoughts. I internalize mine. Others don't seem to have any thoughts at all.

It's something that I struggle with daily. How can I find balance. There is part of most people that wants to be around others. Our species did not succeed by being solitary. I can sometimes tolerate being around others for short periods. Unless it becomes an excuse to badger or harass me. Then I can become quite grizzly and off putting.  Again, it's not something that I'm proud of and I know that it's a coping mechanism. Sometimes these people need to be put in their place and I may be more than willing to be the one to do it. Usually, afterwards they're more cooperative.

In closing, please be patient with me. I'm a work in progress. Learning to deal with these stressors is something that I need to get a handle on.

Here's to a new year! And, to my true friends:  Thanks for being you.

(Mis)Adventures in Networking

EDIT:  Whoops! I never published this blog post.

Wow, it's been a while since my last post.  Anyway, if you were looking for an example of Murphy's law, I've got one for you.

We had 2 simple goals:  upgrade our internet service and bring up our new WAN connections.  This adventure starts on Tuesday, April 19, 2011.

Background:

In the city where I work, we are really locked into our choice of providers.  Because of the city's infrastructure, we have 2 options: the incumbent telecom provider and the local cable company who has no interest in providing service to us.

This install is an upgrade to our existing service and also a relocation to another building.  It was originally supposed to take place last November. As we were ramping up for the install, we discovered that my former boss didn't place the order. (This after I received a couple of phone calls asking why we were requesting service at 2 sites during the summer.) After a couple of angry phone calls to the telco rep, the order was placed.  We were given an install date in early February.  A week before the date arrived I sent my weekly pester the project manager email and was told that they were having "unforseen issues and would be unable to meet the date." (According to the rep, they ran out of capacity) The date they gave me was in June.  Called the rep, explained to him how this was unacceptable and that I was going to recommend that we take action to terminate our existing contracts. He was then able to get us a March date. As we got close to the March date, we were told that date was only for our end of the connection and it would take a few weeks more to get the other end installed. Finally in late March, we were told that everything was in place and we looked at the calendar to find a suitable date that wouldn't cause too much disruption.  We decided on spring break week since the majority of our employees were off anyway.

Tuesday Afternoon:
I met with an engineers from my ISP and my integrator to go over the connections and site prep prior to the installation. He checked the configuration on the router as it was shipped and made some changes/corrections.  He made a call to try to bring up the connection, even though it wouldn't be routing any traffic at that point - just to make sure everything was good on the telco end.  After spending some time on hold and talking to several people, he was told that our appointment wasn't until 9:00 the next morning and there was nothing that could be done until then. I found this rather entertaining since he works for the same company.  While he was doing this the integrator and I busied ourselves with racking switches, connecting cables, checking connections, etc. A couple of hours later we went over our game plan for installation and left for the day.

Wednesday Morning:
We met at the appointed time and set out to make magic!! Our enthusiasm was soon dashed when we couldn't get a link to the demark. We did the usual switch the polarity of the cables, trying new/different cables, trying different SFP modules, nothing was working.  At some point during we noticed that we didn't have a link light on the WAN router either - but the connection was up and passing traffic. The ISP engineer continued to work with the telco side to try to troubleshoot the issue.

During a trip to the demark to bang my head against the wall I looked at the fiber box that went to the MDF - it read "SM Fiber to MDF". I thought SM - as in Single Mode Fiber? This should be MultiMode Fiber.  I disbelief I opened the box and looked inside.  Sure enough, it was Single Mode. WTF! (Bang Head on wall). Funny thing is the vendor that installed the fiber was hired by the telco on the premise that they wouldn't install the line unless dedicated fiber was in place. Now, the demark happens to also be in one of the IDF's in the building. Couldn't we just use that? We were told no.  After we put our heads together for a few minutes, I said "F it, this is the only opening we have for over a month that we can do this during the day." I then connected to the fiber that was in the IDF, ran upstairs, moved the cross-connects and bingo, the WAN came up with a link light.

While this was going on, the telco engineer had moved the router to the demark to try to get a connection there. After a while, he was able to get the line up and brought the router back upstairs.  We re-racked the router and fired it up. He was unable to pass any traffic across the connection.  He gets on the phone again and calls me over after about 15 minutes and puts his cell on speaker.  The person on the other line says "Oh, you want to switch your internet service to that line? I asked him why the heck he thought we were going through all of this? His answer (I wish I was making this up): "Oh, I thought you were just testing the line." As Bill Engvall says "I didn't start out the day wanting to be a jackass, but you just pushed my jackass button." Needless to say, we had internet in a couple of minutes. Elapsed time: 3.5 hours.

I wish this was the end of the story, but it's not.

While we did have internet, traffic was sporadic. Our initial troubleshooting lead us to believe it was a DNS issue.  And, indeed, we were having trouble with DNS resolution.  But, when we tried to use a public DNS server, it only marginally better. It worked fine from the DMZ and from outside.  Made a few changes to the routers and firewall and we were able to get access at-first. After about 10 minutes, it stopped working again. Undid the router and firewall changes and it worked again for about 10 minutes.

Hours of troubleshooting later, we were able find the issue, but not a clear solution. When we starting routing traffic through the new WAN, it introduced routing loops at almost every point in the network.  Considering we have 22 locations, the problem was quite massive.

Making corrections after deployment (or whoops!)

Like clockwork, almost as soon as we deployed the new workstations, we began receiving complaints about things that weren't right.  Most of the initial complaint were "I need the administrator password so that I can install software and make changes."  This was met with a resounding "NO!"  It's been an uphill battle and we admittedly have to do some good work to repair our department's reputation. It was most entertaining to get a phone call with a complaint and to be able to tell them "Look at the computer, I'm logged into it right now. And, that program that you said doesn't work does."

The last week or so has been pretty quiet.  After they began to see printers and software magically appear on their computers, they stopped complaining.

There was one item that slipped by us: Some of the laptops would be going home with teachers and administrators and they would need to be able to connect to their home wireless networks.  We had this locked down to prevent students from changing to an open network that they can see from the school.

After hours of digging and false starts, I found the preference file that needed to be changed:

/Library/Preferences/SystemConfiguration/preferences.plist

My first reaction was it's a plist, it should be easy to script the change.  Wrong! It doesn't act like any of the other plist files.  Using a "defaults write" command would put the setting into the wrong place in the file.  I tried to open it in several plist editors, and it only found some of the settings, none of which I needed to change.

After I got home I spent a couple of hours trying to use sed and awk to change the file, but I couldn't get the regex correct.  I was about to give up, when I decided to give Google another try and found my answer:

/usr/libexec/airportd

I ran it with the -h flag to see the options available and I found what I needed:


RequireAdmin (Boolean)
RequireAdminIBSS (Boolean)
RequireAdminNetworkChange (Boolean)
RequireAdminPowerToggle (Boolean)

I could use "/usr/libexec/airportd en1 prefs RequireAdmin=NO" to change all of the settings at once or I could use the others to get more granular if I chose. I wrapped this in a quick BASH script and tested - success!! Here's the code that I created.


#!/bin/sh
#
#  Script to allow Users to change the wireless network without requiring a password
#
#

# Turn off all settings
/usr/libexec/airportd en1 prefs RequireAdmin=NO

# To allow granular settings you could do the following:
#/usr/libexec/airportd en1 prefs RequireAdminIBSS=
#/usr/libexec/airportd en1 prefs RequireAdminNetworkChange=
#/usr/libexec/airportd en1 prefs RequireAdminPowerToggle=

exit 0


Tomorrow I get to see if it works with Lion.