Before NIMBioS was created many of the core team of researchers were a part of an older project known as The Institute for Environmental Modeling (TIEM). Like many academic organizations, TIEM operated on a shoestring budget, often making do with tools freely available from the open-source community. It worked, and it worked well. We weren't just making do, we were making; we were productive. And not only were we successful at what we turned our hands to, but we did a better much job of it than most – dare I say, than anyone else in our field. (See Jane Comiskey for an exhaustive list of the less than stellar contenders with which we must share our academic field…)

Many speak of “The Year Of The Linux Desktop” as the punchline to a joke, as everyone knows Linux is for servers, not for desktops. But for TIEM, the year of the Linux desktop came sometime back in the late 1990's, born out of a need to migrate away from an aging and no longer financially viable Solaris desktop environment. And we've been doing very well with Linux desktops for quite a number of years now, thank you very much.

When funding for NIMBioS became available, it was decided that – now that funding was possible – the new institute would make use of the best commercial tools available. With more than fifteen years of legacy Unix tools, data, and products to support, Apple's OSX looked to be the best choice for our new institute. OSX, at least according to the sales propaganda, was the best-fit between continuing support for our unix-based legacy and providing an easy-to-use desktop experience for our institute's users and visitors alike. (And with a unix-based legacy, the idea of migrating to Windows was off the table straight away.)

All along it had been assumed that if we were productive and resourceful while “making do” with open-source tools, that we would in turn be even more productive with commercial tools – tools backed by commercial technical support; allowing us to stand on the shoulders of giants, as it were. Alas, this turned out not to be the case. At the risk of sounding socialistic, the move away from open-source and to commercialism turned out to be a mistake. Productivity dropped for both the end users and especially for the IT staff, who became so bogged down in the quagmire of putting out the fires and fixing the broken that very little time was left over to actually provide support services to end users. As an institution who's existence relies on the high-volume and high-turnover of visiting academics, the service of IT support for the end user is important. (Over the course of 'The Year of OSX', the IT manager wound up delegating end user support service to other IT staff who, quite frankly, have too many other things on their plates already.)

Detailed below is a (nearly exhaustive) list of problems and complications that arose as a result of our attempt to migrate our desktop environment from Linux to OSX.

As a stand-alone desktop computer, we at NIMBioS have no problems with OSX. If all the reader wants is a computer to take home and slap on their desk, the IT management will still recommend OSX as a pleasant, viable contender. If the reader wishes to purchase an OSX machine, the IT manager would say, “Go for it.” If the reader wishes to use an OSX machine in conjunction with another machine running some other OS, the IT manager would say, “Great. Do it.” Most of the problems with OSX that the NIMBioS IT staff have experienced are almost exclusively in the realm of integration with a heterogeneous unix networking environment – a feature touted by Apple pre-sales propaganda as being utterly within the realm of reason.

When the first NIMBioS-purchased iMac was unboxed in Q4 of 2009, over a year before this document was written, managing OSX systems was still a new area of expertise, and IT management will make no attempt to hide that fact. The only claims the IT management make in regards to system administration capabilities are that the findings documented below are the result of following all available Apple-branded technical, managerial, and policy documentation as laid out in Apple manuals, both online and in printed form; and from online support forums and mailing lists used to communicate with other Apple administrators. It is fully expected that an experienced Apple engineer or administrator might look on the following and instantly fall into a fit of spasmodic balking, uttering phrases such as, “What the…”, “Well why in the world…”, and “That would only happen if…”. In response to such persons, the official statement presented by the NIMBioS IT management and staff is as follows:

“We have done the best with what we currently have, and we are left seriously wanting. To this end we have reached an impasse. We the IT management have come to the conclusion that in order to stay with OSX, we will need to replace our Linux server infrastructure with Apple servers, reconfigure the services provided by those servers, purchase very expensive Apple support contracts, and purchase quite a lot of expensive commercial software. Either we foot a very large bill, or we can first try moving back to Linux for free.”

Given the options, it was decided to at least try Linux before shelling out enough money to buy a very nice luxury automobile…

With more than 20 years of unix and Linux server experience, and since Apple claims easy integration into a heterogeneous unix environment, it was decided to keep the Linux servers and use OSX on the workstations.

On an all-Apple network, user home areas are provided via AFP from an HFS-formatted filesystem, and support the full range of filesystem capabilities that HFS has to offer. Unfortunately, since Linux only speaks the AFP protocol with limitations (at the time of this writing, Linux can serve AFP, but can only mount AFP read-only), for network filesystem support only NFS could be used, as it's the greatest common denominator for both OSes. Fortunately, Apple is able to store this HFS file metadata on non-HFS filessytems in dot-underscore files, meaning that user home areas could be stored on any of the modern Linux-supported filesystems. Ext3 was chosen for familiarity, stability, and reliability.

Under TIEM, each workstation was also an NFS server, providing network filesystem support for the automounting of a user's home directory to any other TIEM machine that the user might log into. OSX does not support this feature (at least, not out of the box). Therefore all user home areas are stored on a Linux server, and the OSX workstations mount the user's home area when the user logs in.

Microsoft Office (MSO) is a vital and required piece of software. Whereas TIEM rarely had to exchange documents with the outside world, NIMBioS must do so daily – neigh, hourly. And since it is a Microsoft world after all, (insert grumbling here,) the proper tools were needed if the NIMBioS staff were to expect to be able to communicate and cooperate effectively with both visitors and other organizations. Many tout OpenOffice (OO) as a viable solution, but experiences at NIMBioS have shown that OO is only viable if the user does not have to exchange documents with MSO users. Many MSO-created documents do not translate well when opened by OO, and this poses a stumbling block to NIMBioS users. Any time a user has to spend time repairing the format of a document that doesn't show up quite right in OO is lost productivity.

MSO 2008 on OSX turned out to be a stripped-down version of it's Windows cousin. One of the services required by our staff is access to shared data kept in a set of Access databases. The OSX version of MSO had no Access program, and open-source Access replacements were unable to handle the required file formats to meet the institute's needs.

MSO on OSX was found to be buggy. While the words “bugs” and “Microsoft” are often uttered in the same sentence, our experience has been, at least for some, worse than the average Windows user. To present an extreme case: one user's MSO experience became so bad that she was forced to abandon the use of MSO all together. MSO programs would crash consistently on every MSO-format file she had in her account. MSO was reinstalled, all MSO-related configuration files were deleted from the user's account, and, in the end, the user's machine was even reinstalled completely. All attempts at diagnosing and repairing the problem yielded no improvement. In the end the only diagnosis left to the IT staff was that of bugs in the OSX implementation of MSO. Since the user's needs were few, the user was given a copy of Apple's iWorks. While this did solve the user's problem, it still meant that there was one user out there who's computing experience and computing requirements were no longer homogeneous with that of the rest of the user base.

As a result of problems with MSO, most users have abandoned the OSX version of these office tools in favor of creating a Windows VirtualBox, simply so that they can “get things done”.

The iMac machines (models iMac7,1's, iMac8,1's, and iMac9,1's) do not show expected network speeds on a brand new gigabit network, despite having hardware that claims to support gigabit connectivity. Preliminary tests with Ubuntu 9.10 installed on the same iMac hardware show that Linux is more efficient at network operations than OSX. A typical example can be shown while copying files. A test file was created that was 16.14GB in size. This file was made available on computer A, which was a Linux server hosting the file on an NFS-mountable, non-HFS, POSIX-compliant filesystem. Tests were performed with a regular user account. The test user was logged into a second computer, computer B. The user's home area was NFS-mounted from computer C, another Linux server hosting the user's home area via an NFS-mounted, non-HFS, POSIX-compliant filesystem. The file was then copied from computer A to computer C via computer B. Computer B is the variable. In all tests computer B was the same hardware, but in half of the tests computer B had OSX 10.5.6 installed, and in the other half of the tests computer B had Ubuntu 9.10 installed. Below are the test results. (All tests were performed individually under identical circumstances. All times have been rounded up to the nearest minute. Test results are sorted in order of decreasing times. Tests were repeated multiple times, with the results varying by less than 4% of the values listed.)

In fact, a series of other tests performed by phoronix have concluded that Ubuntu 9.10 will beat OSX 10.6.x when running on the same hardware.

Spotlight doesn't work correctly for NFS-mounted, non-HFS filesystems. Spotlight will consistently fail to find all occurrences of a search term, even if the search term appears in easy-to-find file and directory names right on the user's desktop. The ~/.Spotlight-V100 directory was removed to force Spotlight to re-create it's index, but this had no effect.

NFS and automounted filesystem support on OSX is quirky. Often when navigating in Finder to an NFS-automounted directory, the first attempt fails without an error, giving the confused user an empty Finder window and leaving the user wondering where their files are. This is due to a simple race condition where Finder times out waiting for automount to finish mounting the NFS directory.

NFS and automounted filesystem are buggy. Often NFS traffic throughput would bottleneck somewhere in the OS, causing either specific application windows or the whole workstation to freeze intermittently and display a spinning beach ball. NFS throughput to the other Linux servers would be unaffected, and bandwidth on the NFS server would be well within acceptable usage. Heavy NFS usage will cause the OS to hang, with no opportunity for recovery.

Addendum: Further experience with OSX versus Linux seems to corroborate this diagnosis. After converting several iMacs into Linux workstations, the workstations still running OSX continue to have problems with NFS support. The OSX log files show the following error:

kernel[0]: nfs server home.nimbios.org:/export/a/Users: not responding

Meanwhile their Linux-running iMac brethren show no problems at all.

Addendum: Some iMacs running Linux continue to show networking problems. When reviewing the system log files, the following errors were seen under OSX:

AppleYukon2: 00000000,00000000 sky2 - RX ring overflow -- dropped a packet
AppleYukon2: 00000000,00000005 sk98osx sky2 -  - sk98osx_sky2::replaceOrCopyPacket tried N times

And the following errors continue to be seen under Linux:

kernel: [169461.632554] sky2 eth0: rx length error: status 0xfa0100 length 1642
kernel: [169467.010307]  [<ffffffffa013048b>] sky2_status_intr+0x55b/0x610 [sky2]
kernel: [169467.010332]  [<ffffffffa013059d>] sky2_poll+0x5d/0xc0 [sky2]

The content of the error messages seems related despite the use of a different operating system and device driver. Both operating systems refer to an overflow of data that exceeds the valid length of a standard ethernet packet. Online research concerning this error has turned up evidence that suggests that a run of Marvell Yukon 88E8058 ethernet chips were produced and shipped with a logic bug in the chip itself. (However, the link to this research has since been lost.) The fact that all of our iMacs that experience networking problems contain this chip seems to corroborate the story. There may simply be no fix for this problem.

Finder would, under some mysterious circumstances, occasionally exhibit a bug on NFS-mounted, non-HFS filesystems, where files and directories would either disappear or appear in the wrong location. For instance, on one occasion a subdirectory containing a space in it's name would disappear. In it's place would be a duplicate listing of some other subdirectory. The only fix for this seemed to be to open a Terminal window and rename one of the two subdirectories.

On another occasion Finder would show the incorrect files in a subdirectory. The image below shows an example of this Finder NFS bug. In the example image below, the subdirectory “Animals” contains a list of image files. The subdirectory “Spaces” is empty, but Finder is showing that the “Spaces” subdirectory contains the same files as the “Animals” subdirectory.

When inspecting files and directories from Terminal everything is found to be normal. All directories and files are as they were expected to be, so this bug only seems to affect Finder. This bug only seems to be present on NFS-mounted, non-HFS filesystems.

OSX would not allow users to change file group ownership or permissions if said file was located on an NFS-mounted, non-HFS filesystem. After creating a file or directory, in order to ensure that the proper access permissions were in place so that the correct group of users were given correct access permissions, the user would have to open Terminal and use the chgrp and chmod commands. In a multi-user environment where staff share a network filesystem with visitors this causes lost time as the user takes the necessary extra steps.

OSX would not allow users to set a umask for default permissions on files and directories that the user creates. The default permissions on created directories allows any user to read and enter the directory. The default permissions on created files allows them to be read by anyone. The inability to set a user's umask presents a security risk. For instance, our business manager's files, many of which hold financial and personnel information, should never be accessible to a visiting academic. (Luckily our business manager does not have to handle social security numbers, and this security hole was caught early.)

Instead, the OSX method is to use Access Control Lists. This requires additional metadata (presumably stored in the dot-underscore files). The control of ACL was inaccessible on OSX 10.5.6 for files and directories housed on NFS-mounted, non-HFS, POSIX-compliant filesystems.

The fact that Finder had controls in place to allow the user to manipulate permissions, but had them greyed out, suggests that this aspect of Finder was either bugged or incomplete.

Over the course of a single 365-day year, a total of 35 iMacs have suffered three failed hard drives and at least two failed DVD drives (that is, they can read but writing will produce a corrupted DVD). This is a failure rate of greater than 10% for the year. Meanwhile, cheap Dell PCs running Linux on the TIEM network have suffered two hardware failures over the course of the last six years. (I have my suspicions that these iMacs simply run too hot.)

Addendum: Now in Q3 of 2011, NIMBioS is in possession of five dead iMacs out of the initial investment purchase of 35 machines. That is a failure rate of 14% in 2.5yrs. Meanwhile, the PCs are still going strong, with only a single hard drive failure reported.

Apple uses a unique implementation of the Remote Desktop Protocol. Neither Linux nor Windows users were able to connect to the VNC or RDP services of an OSX workstation. (Tutorials are available that instruct the user to use SSH tunneling to provide remote access, but this involves more steps than my users feel comfortable with. These tutorials also did not work for the OSX 10.5.6 VNC server when used with Windows and Linux VNC viewers.)

Users can gain access to an OSX workstation via SSH, but since most of the programs they need require an Aqua interface this method of access is of little or no use.

ARD would fail constantly, and need to be restarted. The online Mac administration community abounds with posts regarding ARD failure and recovery. Recovery involves logging in via SSH and issuing a series of commands.

Often when ARD fails while logged into an IT manager account, the console is left open for any user happening by to take control and begin wreaking havoc on the machine. This is a security risk.

It is possible, using the Apple Remote Desktop software package, to “curtain” the machine while the remote IT manager is working. However, occasionally, when ARD fails, the remote IT manager cannot regain control, in which case the only known method to regain control of the system would be to issue a reboot command via SSH. Unfortunately, if connection was lost in the middle of applying an OS patch or installing software, while the local screen was curtained, then there would be no way to monitor the progress of the task to know when it would be safe to perform a restart. The only solution left in this case is to reboot the machine and hope for the best. This is risky (see: Problems With OS Updates, Problems With Software Management).

Best methods for remote IT management involve initiating an SSH connection to the machine and becoming root before connecting via ARD, so that in the event of ARD failure the remote IT manager can quickly attempt to regain access. Often the only method of regaining access is to reboot the machine. When performing IT managerial tasks this is risky (see: Problems With OS Updates, Problems With Software Management). Particularly when using ARD to install OS updates, the remote IT manager risks losing connectivity not only with ARD but with SSH as well, leaving the workstation in an unknown state and possibly open to security risks.

The Apple-endorsed method for fixing problems involves distributing patches. This approach is similar to Windows. Unfortunately this method of OS upgrades presents some problems.

Many updates require rebooting. Even updating the web browser requires a reboot. This means either interrupting the user to kick them off of the console. This would suggest the need for after-hours work.

Sometimes when a machine installs updates that require rebooting the machine will fail to boot up properly. The machine will hang during boot at a blank blue screen. The fix for this seems to be to power-cycle the machine, which appears to cause the hung patch to be re-applied on reboot. This makes remote administration of updates risky. After-hour updates must be done on-site in the event that the machine does not boot up properly, so that a human can take action if the machine does return to proper functioning order. This would suggest the need for on-site work. Taken together with the need for after-hours work, this would make for unhappy IT staff.

The Apple-endorsed method for managing the installation and upgrade of multiple workstations involves using a template machine to build a disk image containing all of the software and configuration settings desired for an OSX workstation. This disk image is then post-processed to produce a new disk image that is suitable for network streaming. The installation process involves booting the install client workstation from a netboot image that contains an installer program to format the client's drive and download and install the workstation disk image. Finally customizations are added such as configuring LDAP binding and running admin-provided post-install scripts.

The tools purchased with OSX Server for installing and upgrading workstations are buggy and unreliable. The template disk image is created with a tool called System Imaging Utility. When this tool works properly it will take between 4 and 5 hours to image the template machine and post-process the image for network streaming. When not working properly, this tool will usually run for 4 to 5 hours and then crash. Occasionally SIU would simply hang, never finishing even if left running for 12 or more hours. The failure rate for SIU was found to be at least 50%. It would be often that the IT manager would have the time to run SIU only twice in one average workday, have SIU fail both times, and leave at the end of the workday having accomplished nothing. After successfully completing an installation image, subsequent tasks using SIU would fail until SIU had been restarted.

The program hdiutil, the automated disk utility program used by the netboot install image to partition and format an install client's drive, can sometimes fail. Occasionally when hdiutil fails, it also leaves the client's drive with a corrupted partition such that future invocations of hdiutil will also fail (presumably due to a bug in hdiutil). The only known fix is to manually open the Disk Utility application and repartition the drive by hand. This makes automated installation unreliable.

The installer embedded in the netboot install image uses ASR. ASR is finicky and unreliable. It's ability to successfully complete an installation is dependent on the speed of the network and the CPU load of the server providing the template disk image file. There are no tools to monitor or measure the health of the ASR install utility or the ASR server. When the installation fails, there are no helpful error messages in the log files of the server. The client will report an unhelpful error message that reads “XSTA fail” error or a codec overrun/underrun error. The solutions given by other Mac administrators vary wildly, implying that the causes of the error and how to really fix/prevent it consist of an unhealthy dose of speculation. This makes automated installation unreliable. Sometimes, when ASR is being particularly difficult, using uncompressed disk image files will appease it, but this makes installation times longer, and it does not always work.

Due to the buggy and unreliable nature of the installation management tools included in OSX Server many administrators have turned elsewhere. Many use free tools such as DeployStudio. Whole companies have arisen who's sole method of revenue is to provide replacement tools for the ones provided by OSX Server. The clear winner in the realm of corporate solution seems to be JAMF Software, who's installation management tools cost $3600 plus $100 per year per workstation managed. This makes automated installation very expensive.

Since the Apple method of installation management involves building template images, and building template images involve applying OS patches that do not always apply properly, it is possible to create an install image with errors that are then propagated to all of the institution's workstations. This has been seen at least once, when a patch caused problems with the wireless ethernet driver. At the time of this writing none of the Apple iMac workstations installed from a template image can access the UTK wireless network. Other Macs installed by hand are able to, as well as other operating systems.

Generating template images is a lengthy and time-consuming process that cannot be automated (see also: Problems With Software Management, Problems With OS Updates). Because of this, template images often contain human errors and oversights.

Since the Apple method of installation management involves building template images, any changes, fixes, or updates to the template require new images to be generated, which is a lengthy, time-consuming, and error-prone process (see also: Problems With Software Management, Problems With OS Updates).

The best approach to minimize error and maximize productivity is to keep the template image small. Only include the operating system and the configuration necessary to allow the remaining software, packages, and configuration to be installed after reboot. This way, any changes needed may be able to be made to a separate package file that can be distributed separately after installation, instead of requiring that a new set of images be built. This approach presents it's own set of problems (see: Problems With Software Management, Problems With OS Updates).

Software provided as an Apple *.pkg package:

• Pro: This is the preferred method of installing software. It involves creating an Apple package file (flat *.pkg files). Apple *.pkg files can be installed from the command line remotely via an SSH connection, or using the Apple Remote Desktop tool. In many cases this can be done without requiring console access or rebooting, so that software can be upgraded without interrupting the console user.
• Con: Apple package management tools lack the ability to report the version number of either *.pkg files or of installed software in a consistent and reliable manor. This means that external methods must be employed when attempting to keep track of what version of a package has already been installed, what version is available, and how to compare the two version numbers in a meaningful way.
• Con: Apple package management tools and packaging policies lack a method to either encourage or enforce a consistent (re)configure/uninstall interface. The ability to perform these tasks is left to the package creator. Often package creators fail to include an uninstall feature at all.
• Con: Apple package management tools and packaging policies lack a method to either encourage or enforce idempotency.
  • If package removal is possible, then installing and removing a package may not leave the machine in the same state as it was before installation.
  • Reinstalling a package that is already installed may change the state of the machine. (Ex: One should never reinstall Microsoft Office 2008, as reinstalling over an existing installation may break the software.)

Some software providers only provide an application file:

• Pro: This type of software delivery method is the easiest to manage, and very easy to repackage into a flat Apple *.pkg file.
• Con: There is no consistent method to determine the version of an installed application. This means that the IT manager must employ external methods to track which workstations have which version(s) of the application.
• Con: Sometimes software removal is as simple as deleting the application. This is the Apple-endorsed method for software removal, and it does not work consistently or for all cases. Often an application will leave files behind, taking up drive space. For example, the open-source math package “R” can be installed by merely dragging the application to the /Applications/ folder. However, in the process of running R, many addons may be downloaded and installed. These addons are stored in the /Library/ folder, not in R's application directory. This means that if R is later removed from the system by deleting the application, there is a significant amount of data left behind in the /Library/ folder taking up drive space.

Some software providers only provide a proprietary installer program:

• Con: The installer program usually requires console access and the Aqua GUI interface. Such software cannot be installed from the command line remotely, and installation requires that the console user be interrupted to relinquish control of the console to the IT manager.
• Con: The software provider may not provide a method for software removal.
• Con: When a method for software removal is available, the method is inconsistent from one software provider to the next.
• Con: Software providers may not provide tools for automatically checking for and applying updates and upgrades.
• Con: When an updater utility is available, it usually requires console access and the Aqua interface, meaning that the console user must be interrupted to relinquish control of the console to the IT manager.
• Con: When an updater is available, the methods of checking for and applying updates is inconsistent from one software provider to the next.
• Con: Software provided in this fashion is usually not idempotent:
  • If software removal is possible, then installing and removing the software may not leave the machine in the same state as it was before installation.
  • Reinstalling software that is already installed may change the state of the machine, and may in fact corrupt the software.

Some software is provided via Fink and MacPorts

• Pro: Easy to install, easy to upgrade, easy to remove
• Con: Only useful for installing unix-related applications (console or X11-based)
• Con: Many of the tools we have needed are only available from MacPorts, which does not provide pre-compiled binaries. Installation involves compiling on each workstation. Features exist in MacPorts to build pre-compiled binaries for private distribution, however the tools and methods were unreliable and failed often.
• Con: Access to MacPorts source repositories were often interrupted when software in the repository was being updated. Updates to software often broke dependency chains or compile requirements for other software. This has the result of causing some workstations to have working-but-out-of-date versions of software while other workstations had no version of the software at all due to dependency/compile errors.

Third-party and open-source software:

• Pro: Free
• Pro: Most local and visiting academics are used to using free software and are already familiar
• Con: When there is a significant change in OSX, or when OSX delivers a new release (ex: when Show Leopard was released), third-party and open-source software makers must scramble to update their software so that it will work again. In the meantime, anyone who has already updated their version of OSX must wait and do without their favored software.

Given two servers, an LDAP server and an LDAP replica server: When the server went down, the IT management team followed the instructions from the OSX Server documentation to promote the replica server to become the new master server. The task failed. Horrendously. Every user was offline, and remained offline, until repairs to the original master LDAP server were complete.

The above complications all add up to diminished productivity for users and IT staff alike. To summarize:

1. Currently supporting three operating systems: OSX (workstations), Linux (servers), and Windows (virtual machines), incurring the cost of extra time from the IT support staff. Less time spent on support for operating systems means more time for user support.
2. OSX workstations
   1. Difficult to install and upgrade
      1. Apple package management does not include the ability to uninstall outdated or replaced software; for some software the only means of removal is to reformat and reinstall without it
      2. Apple software installation has many different forms, only two methods allow installation from the command line
      3. Most software installation methods require Aqua, which required interrupting the console user
      4. Many users rely on free and open-source software (i.e. gimp, imagemagic, jgraph, gnuplot, etc.)
         1. When significant updates are made to OSX, or a new version of OSX is released, third-party software makers must scramble to update their software to work with it. (Ex: When OSX 10.6.0 came out, many of the open-source tools we rely on were incompatible for several months, meaning that we could not have upgraded even if we wanted to.)
         2. Updates to software repositories (such as that managed by the MacPorts project) often suffer from broken dependency and/or compile chains when upgrading software. (By contrast, modern Linux distributions maintain three separate repository trees: one for unstable, one for stable, and one for legacy. By the time software moves from unstable to stable dependency and compile tree issues have already been resolved.)
   2. Require constant care to keep administration tools and services running
      1. A lack of command-line administration tools means that administrative tasks must be performed via ARD
      2. Using ARD means interrupting the console user or performing admin tasks after hours
      3. ARD, required for remote admin tasks, is buggy and fails regularly, needs constant care
      4. OSX Server tools for building and installing workstation images is buggy and has a failure rate of at least 50%
         1. Creating workstation images is a task that takes 4-5 hours
         2. Apple server tools may go for hours without a hiccup, and then fail, resulting in hours of lost productivity
         3. On average, 40% of admin time is spent fixing OSX Server administration tools instead of actually using those tools productively.
      5. Have bugs that cannot be fixed
         1. Finder and NFS and missing files/directories
         2. MSOffice is buggy, most users have abandoned it's use for Windows on VirtualBox
         3. VirtualBox shared folders on OSX suffer from a 10-second delay, this is not seen on VirtualBox under Linux
3. Windows
   1. Can't get away from it (Tried. Tried real hard too.)
   2. More users have needed Windows and Windows tools than expected
   3. More users have turned away from OSX tools to MS tools running under VirtualBox

1. To properly support Apple OSX will require more resources:
   • Q: What are other admins doing to make it work?
     A: Although Apple's pre-sales propaganda promises the necessary tools, those tools fail to deliver. Other OSX admins:
     • Buy Apple server hardware
       • Our $2,700 Dell Linux servers would cost $4,000+ from Apple
       • Our $6,000 Linux storage servers would cost $20,000 from Apple, and still have less capacity
     • Buy Apple support contracts
       • $6,000/yr for 3 incidents
       • $20,000/yr for 20 incidents
     • Buy expensive third-party management tools to replace the buggy and broken tools from OSX Server
       • $3,000 for software to replace installation tools bought with $500 OSX Server, plus
       • $100/yr/workstation for licenses – this means with our current workstation count a yearly cost of $3,000
     • Still does not resolve issue of inadequate MSOffice tools, users will still rely on VirtualBox, so why do it?
2. To move to Microsoft
   • Will require more costly server licenses ($1500 for a terminal server license to allow remote login)
   • Cost of training
3. To move to Linux
   1. Can offer a faster, better, and cleaner user interface experience:
      1. On the same hardware, Linux out-performs Leopard
      2. Wine can run MSOffice 2007 tools: Word, Excel, PowerPoint, Access, Publisher – some users will not need VirtualBox anymore
      3. For users that need/prefer VirtualBox, no more 10-second delay when using shared folders
      4. User home areas can be located on the user's primary workstation (on their desk), increasing response time while decreasing network traffic (this is the paradigm that TIEM systems are built around, and it's a time-proven method).
   2. Can offer useful methods of remote access
      1. SSH and X11 tools are freely available for Windows, and come with OSX
      2. With SSH and X11, anyone can log into a Linux machine remotely and run any application. (Note: VirtualBox does not run well over SSH+X11, but VirtualBox does offer authenticated VNC access that works quite well.)
   3. Ease of installation and upgrade:
      1. Instead of building install images, all installations involve using Linux package management tools to install packages
      2. Linux packages enforce the ability to uninstall
      3. Linux packages are idempotent
      4. Linux packages are discrete, installing or upgrading a package does not interfere with the files from another package unless special previsions have been made to ensure compatibility
      5. Linux packages can be made available in a secure, private repository
      6. A workstation knows what software it should have installed
      7. A workstation knows where to find software that should be installed but is not
      8. A workstation knows the versions of all software and whether or not the repository contains a newer version
      9. A workstation can be made to inform the IT manager if software needs upgrading
      10. A workstation can be upgraded via SSH by the command line

See also:

• OSX versus Linux Benchmarks – How does Linux compare to OSX on the same hardware?

• A Plea To Software Vendors From Sysadmins - 10 Do's And Don'ts – Many of the same points concerning software installation, upgrade, and management are mirrored in this article.

• Take This GUI And Shove It – More 'Why the command line is superior' propaganda.

• 12 Reasons Why Every Linux System Administrator Should Be Lazy – Why it's better to work smarter and not harder.

• Apparently Apple is scaling back their support for the server market, and subsequently, their support for an all-Apple enterprise environment:
  • Apple Kills Xserve Server Line – Suggesting that Apple may be positioning itself to abandon the server market.
  • 5 Things About Apple You Won't Hear From Steve Jobs – More 'Apple Abandoning Server' rhetoric.
  • Xserve Transition Guide – Apple's official 'How To Ditch Your Apple Servers' guide.