Home

Using Lab Computers at HMS

Posted in

Network File Storage and Backups

SFTP Access to Crystal

If you have an outside user that needs to download or upload a file to our site, it's better not to give the person a network account, as the principle of least privilege applies. So we can provide 'anonymous' access to our site via a chrooted user account. It is protected by rssh to only allow SFTP (no SSH shell) and because it is chrooted to its home directory, the remote user will not have access to any other files on our network.

The remote user should use their preferred SFTP client to connect; you should contact help@sbgrid.org for the required username and password.

Once connected, the user can place files into the UPLOADS directory, where you can copy the files out of later. Should the remote user need to download files, you can place them in the DOWNLOADS folder for them to access.

Backups

SBGrid provides backups for local HMS labs, which include the Harrison, Chou, Walker, Hogle, Rapoport and Walz Labs. All files in /nfs are backed up nightly to an on-site file server and then off-site to tapes (except previous years' synchrotron data, which is archived at the end of each respective year). Some data (/nfs/home for example) is also backed up to a third backup server for additional redundancy.

Unless you've been told otherwise, local files on your computer are not backed up. If you have any files stored on your local hard drive, please contact help@sbgrid.org to inquire about options for backing it up.

Details

  • Participating labs share all hardware costs for the backup server. This includes initial purchase and maintenance.
  • As part of this service, SBGrid will keep a single copy of your data on RAID-5 backup servers at HMS.
  • We attempt to run backups on daily basis, but please note:
    • In the unlikely event that both your local filesystem and our backup RAIDs fail at the same time you might lose your data.
    • Local data corruption may propagate to the backup server
    • Any data deleted on your site will periodically be deleted from the backup server during the nightly backup procedure.
  • To increase your data security we suggest that you make periodic archives of your important data.

   

Mounting Your Network Directory on Your Home Computer with SSHFS

If you would like to browse your files in the Finder OS X user interface on your home computer, you can easily do so with a program called Macfusion. To mount your home directory on crystal, please use the following steps.

  • Request SSH access from help@sbgrid.org, if you have not done so already.
  • Download MacFUSE from here: http://code.google.com/p/macfuse/downloads/list
  • Install MacFUSE by double clicking the .dmg file, and then opening the MacFUSE.pkg file.
  • Restart your computer.
  • Download Macfusion from here: http://www.macfusionapp.org/
  • Unzip the Macfusion app and place it into your applications folder.
  • Double-click the Macfusion app, and click the '+' sign to create a new mount. Select SSHFS from the popup menu.
 
 
  • Name the mount in the first text box, then enter the server name, your username, and password. All of the other values should be okay as Default.
 
 
  • Mount the share by clicking 'Mount' in the Macfusion window. Your share should be located at /Volumes/<mountname> which can be located with the Finder.
Network Account Quotas

In general, you have two file system quotas limiting your network account. One controls the space in your home directory, and the second controls the amount of space in your data directory. 

Home Directory Quotas
There is a default quota of 5GBs for your files in /nfs/userdocs/<lab>. Providing you haven't been given an exception, this means that you cannot have more than 5GBs of files in your home folder at a given time. If you exceed this number, you will be given a grace period of 7 days to remove files that put you over the quota. If you do not remove files within this grace period, you will be locked out of your account.

Please also note, that if you exceed 7GBs of files at any time, you will be locked out of your account immediately.

If you get locked out of your /nfs/home directory, you will not be able to receive new mail or log in to the Linux workstations.


Data Directory Quotas
Your data folder in /nfs/xray is also limited by a quota. By default, your quota is 110GBs. Much like with your home directory, you will be given 7 days to clear space if you exceed this number. If you exceed 130GBs of space at any point, you will be locked out immediately.

Checking Quota Usage
To check your quota usage, you can run the following command:

quota (use quota -s on Linux computers for a more human-readable output)

If you become locked out of your account at any point, please send an email to help@sbgrid.org.

Copying Data Between Sites

If you need to transfer data from a remote location (such as a synchrotron site), you can take advantage of the UNIX command line to do so.  In order to do this, you will need SSH access to your lab. This is not enabled by default at all research sites. If you need SSH access, please submit a ticket to help@sbgrid.org. The recommended way that you transfer your data is by using the rsync command.

EXAMPLE

rsync -av /path/to/folder remote-host:/path/to/data/folder

This example shows how to copy a whole folder from your local computer, to the remote computer. The local and remote filepaths may be reversed in order to copy a file from a remote host to your local computer.

If you are uncomfortable using the command line for moving files, you can also do so with a desktop SFTP (SSH File Transfer Protocol) Application. For OS X, a recommended SFTP client is Cyberduck (http://cyberduck.ch/). For Windows, you can download WinSCP (http://winscp.net/eng/index.php). Both applications are free.

 

Like with rsync, You will need SSH access enabled. To connect to your lab, just enter your server's fully qualified domain name (i.e. lab.university.edu) in the server field and use your network account username and password to log in.

Understanding Your Network Account

Portable Home Directories

Overview

Your PHD is a local account on your iMac, which synchronizes data to your CMCD network account. This gives you the benefits of using your account locally (application compatibility, independence from the network) while still placing your files where they are easily accessible to others in your group, or from somewhere else on the network, such as from one of your lab's linux workstations.

 

Your network account files are stored on our file servers. Typically, these are located in /nfs/userdocs/<PI Initials>/<username>. Your PHD is located on your mac, and is located at /Users/<username>.

 

Synchronization takes place every hour in the background when you are logged in. The Sync is bi-directional, meaning you can make changes in either your network home directory or your local home folder, and the changes will be reflected on both sources.

 

You can ensure that checks are happening by clicking the double-house icon in the menubar. The time and date following the 'Last Home Sync' item is the time of your last sync.

 

 

Sync's only happen once every hour, so if you make a change that you need on the network immediately, you can initiate a sync, by selecting the 'Sync Home Now' option shown in the picture above.

 

Sync Failures

Syncs can fail for various reasons, including corrupted files, unresolved sync conflicts, or exceeding your quota. If you notice that you have not had a recent sync, please contact help@sbgrid.org as soon as possible.

 

Conflicts

Sync conflicts usually arise when you have a particular file that was modified in two different places between the current time and the last sync. You can usually resolve the problem yourself in this case. When the sync encounters a conflict, you will be greeted with a window asking you to select the correct version of the file.

 

 

There are a few options to choose from:

  • Use the local version
  • Use the network version
  • Keep the older version of the file
  • Keep the newer version of the file

 

The options are self-explanatory, but be sure that you understand what they mean before choosing one. If you do most of your work on your mac, you can usually choose the local version of the file. If you log into other computers frequently, it may be a good idea to select the most recently modified version of the file. Again, be sure to choose the option that best fits your recent usage, as selecting one version of the file will overwrite the other version.

If you need help deciphering sync conflicts, you may contact help@sbgrid.org for support.

 

File Issues

Syncs can often fail if you have a corrupt version of a file, or if the filename has any special characters in it. In most cases, you won't receive a notification if this happens, so it is important to monitor your sync status via the PHD menu bar icon (as described above). If you notice a sync hasn't completed recently, please email help@sbgrid.org

 

Exceeding Your Quota

PHD syncs will fail silently if you attempt to sync a file that will exceed your storage quota on the file server. If you notice that syncs are failing, you can check your quota with the `quota` command, and be sure that none of the files you are trying to sync will fill your available space.

 

Sync Exclusions & Utilizing the Local Hard Drive 

The following items are NOT included in the PHD syncs:

 

  • Trash folder
  • Local email folders from Apple Mail, Thunderbird, and Entourage
  • Music folder, and any *.mp3 files.
  • iPhoto Library
  • Downloads folder
  • Virtual Machine files (VMWare, Virtualbox, Parallels)
  • Dropbox folder

 

If anything important to you is included in that list, it is recommended that you back it up to another media (such as an external hard drive)

 

If you would like to utilize the local storage on your mac, or you would intentionally like to prevent a file from being synced, and it isn't on that list, you can keep the file in the /Users/Shared folder. However, please note that the /Users/Shared is not backed up or maintained by SBGrid. In many cases, the contents of this folder could be erased if your operating system needs to be reinstalled. You can also utilize an external hard drive to store things outside of your home folder. In any case, it is recommended that you keep another copy of important items that you have placed in the /Users/Shared folder.

Changing Your Password

If you would like to change your password, you can do so with the web-based tool, located at: http://cmcd.hms.harvard.edu/changepwd

Please note the following restrictions. Your new password must:

  • Differ from your account name
  • Contain at least one letter
  • Contain at least one number
  • Contain at least 8 characters

 

If you have forgotten your password, or are otherwise unable to log in, please email help@sbgrid.org for support.
 

Network Account Creation and Deletion

Account Creation
 
Requests for user accounts must come from a PI or from someone a PI has authorized to perform requests of this nature. In order for SBGrid to create your account, please have your PI send the request to help@sbgrid.org with the following information:
 
  • Your first and last name.
  • Your primary email address

 

Account Deletion

  The PI must email help@sbgrid.org to notify us when a user leaves the lab. At termination, any SSH access is disabled. You may request 90-day extension prior to expiration by email (we'll CC: PI). You must set up forwarding email address if account is extended

Troubleshooting Firefox on a Network Account

The popular Firefox web Browser isn't written for concurrency within a user profile. That means that you can have only one open instance of the program per user. If your account is not local to your machine, you must close all previous instances before trying to start a new browsing session. On occassion, Firefox will not start, because previous sessions were not closed properly (because of a crash or network interruption.)

On Mac OS X

To fix this issue on OS X, please log out, and log back in.

You may also need to delete your parentlock file, located at ~/Library/Application Support/Firefox/Profiles/*.default/.parentlock

On Linux

Before attempting the following steps, please make sure that you are currently logged into only one computer.

Here are some steps to use Firefox, if it becomes locked by mistake:

  • Remove any application lock files. Typically, these lock files are found in the browser home directory. Just run the following to delete the locks:

    rm -f ~/.mozilla/firefox/*.default/.parentlock

    After you do so, try to run firefox one more time.

  • In the worst case scenario, you could do this: mv ~/.mozilla/firefox ~/.mozilla/firefox_hold

    (Note: this will completely reset Firefox.) Then start your browser, which will create a new profile. After you do this, you may wish to restore your bookmarks. You can do this via the Firefox menu path Bookmarks/Organize Bookmarks/File/Import (your old bookmarks would be in ~/.mozilla/firefox_hold/…)

HPC / Cluster computing

Running Frealign on the Clathrin cluster at CMCD

Preparation

The first step towards submitting jobs using Grid Engine is to ensure that you have passwordless SSH access to each of the cluster nodes.

If you don't already have an SSH key installed on your account, you can create one using the following command while logged into your network account:

ssh-keygen -q -t dsa -f ~/.ssh/id_dsa -N ''

Then, to set up passwordless authentication, issue the command:

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

 

Job Submission

  1. SSH to the headnode of the cluster
  2. change to the directory where your particle stack is
  3. run the command '' /programs/share/frealign/i386-linux/cluster_scripts/bin/frealign_cluster.sh'' to copy the scripts to your dir
  4. edit the mparameters file and take note of the different variable names and comments
    • be sure to at least edit the path to the frealign binary you want to use, which cycles you want to run and what increment to use
    • the last tested version of frealign was 7.05. Please let us know if you need to use another version
  5. run ''mrefine_sge.sh'' to start the job submission process

If you want to monitor your job, open another shell on the headnode and run:

qstat -f -u \*

To use the Grid Engine GUI, run:

qmon

Grid Engine Manual

http://docs.sun.com/app/docs/doc/817-6117

General

Support for Laptops

What is supported

We can provide as much support as possible for laptops that have been purchased with funds from your lab. This support includes:

  • Connectivity to your local lab infrastructure and SBGrid's services.
  • Local installations of the SBGrid programs tree.
  • Basic troubleshooting.

 

What is not supported

SBGrid does not provide any support for personally purchased laptops, or machines that you do not use for work.

In addition, SBGrid is NOT responsible for (on any laptop):

  • Repairing laptop hardware.
    • Please contact the institution which you have purchased the laptop from.
  • Obtaining proprietary software licenses.
    • Non-free software is not available from SBGrid. Please contact your institution for purchasing software and obtaining installation media.
  • Software Installation (Other than the SBGrid Tree)
  • Providing backups for your laptop.
Connecting to the CMCD VPN

On Mac OS X:

 Step 1. Click on the Apple Menu and select System Preferences  


 

Step 2. Click on Network 

Step 3. Unlock the panel by clicking on the padlock and enter the username and password of an admin user. 

Step 4. Click on the '+' sign to Create a New Service 

Step 5. The Interface is VPN, the Type is L2TP over IPSec. Enter a Service Name, ie: CMCD VPN. Click on Create. 


 

Step 6. Enter the server address vpn.sbgrid.org. Enter your username. This is the username you use to connect to crystal, and other CMCD services. Click on Authentication Settings .

Step 7. Enter your password under User Authentication. Email help@sbgrid.org for the Shared Secret. Click OK.

Step 8. Check the box Show VPN status in menu bar. Click Apply. 

Step 9. To connect to the VPN, open the VPN menu in the menu bar, and select Connect.

 

Step 10. You'll see the word Connecting scroll across, and once the connection is complete a timer will start.



 

Step 11. When you want to disconnect from the VPN, simply open the VPN menu, and select Disconnect. 

 

On Windows XP:

Step 1. Due to the behavior of WinXP SP2, a registry patch must be applied before connecting to the CMCD VPN. Download this registry patch, extract the contents of the zip file, and then double click it to run the patch. 
 


Step 2. Click Yes when prompted. If the patch was successful, you'll get a message similar to the one below. 


 

Step 3. Reboot your computer after applying the registry patch. 



 

Step 4. Go to the Start Menu, and open the Control Panel. 



 

Step 5. Open Network and Internet Connections. 


 

 

Step 6. Select Create a connection to the network at your workplace.




 

Step 7. Select Virtual Private Network connection. Click Next.




 

Step 8. Enter a name, ie: CMCD VPN. Click Next.


 

Step 9. Enter the server name vpn.sbgrid.org 


 

Step 10. Check the box Add a shortcut to this connection to my desktop. Click Finish. 


 

Step 11. Open the shortcut that was just created on your desktop. Click on Properties. 


 

Step 12. Select the Security tab, and click on IPSec Settings… 


 

Step 13. Check the check-box, and enter the pre-shared key (please contact help@sbgrid.org to obtain the key). Click OK. Click OK again to close the Properties page.


 

Step 14. Enter your username and password and click Connect. This is the username and password you use to connect to crystal, and other CMCD services.




 

Step 15. If everything worked correctly, you'll see the following messages. 


Step 16. To disconnect from the VPN, right-click on the icon in the system tray, and select Disconnect. 

Submitting A Help Ticket

When you submit a help ticket, please try to include as much information as possible. The following is a list of information that should be useful in any ticket that you submit:
 

  • State the problem as best you can. Explain what is happening, and explain what you expect to happen when the issue is not present.
  • Include the name of the computer you are using. If you are not sure, run the `hostname` command from a Terminal.
  • Indicate whether the problem is isolated to a single machine, or if it affects all machines in your lab.
  • Include the operating system where applicable (Mac OS X/Linux).
  • If the issue is related to a piece of software, include the name of the software and version number.
  • If an error message is present, copy and paste the exact error message.

 

To submit a help request, you have two options:
 
 

It is important that you submit a help request for any SBGrid or IT related issue that you may be having, even if you have easy access to speak with SBGrid personnel.

Using the Linux Workstations

Installing Programs Without Root Access

~/bin

If you need to install a program that is not already available on your linux workstation, you can compile the program from source and install it into your home directory. This does not require root access, and if you need to work on another workstation of the same architecture, programs in your home directory will also be available to you there.

First, create a folder called bin in your home directory if it does not already exist. In this folder, you can place pre-compiled binaries of many Linux applications, such as Firefox. If the program you need is not available as a precompiled binary, you may be able to compile it yourself. Just download the source archive of your desired program, extract the source of the application, and cd into the resulting directory. Then, run the following:

./configure --prefix=$HOME
make
make install

If you would like to run the resulting program from a terminal, you can run it directly from it's containing directory, or add the path of your ~/bin folder to your unix $PATH variable.

Please note that SBGrid is not responsible for resolving dependencies or supporting applications that you build yourself. If you need a program for work that you cannot compile and you feel would be a valuable addition to the SBGrid software tree, please use the New Software Request form.

Firefox Plugins

If you need to install a plugin for Firefox, such as Adobe Flash, you can put the plugin .so file into your Mozilla plugins folder, located at ~/.mozilla/plugins

After you install your desired plugin, restart Firefox, and then enter about:plugins into the address bar, and look for the plugin in the list to verify its installation.

Using the Mac Workstations

Portable Home Directories

Portable Home Directories


Apple's network account solution is called Portable Home Directories (PHD). It creates a local account that synchronizes with your network account at login, logout, and every 60 minutes in the background while you are logged in. If you don't want to wait for the login/logout syncs, you can safely cancel them; however, your accounts preferences will never be synchronized if you cancel both login and logout syncs. Your documents and folders will always be synced in the background every 60 minutes.

This solution provides several advantages:

  • Your documents have an automatic backup, as they are stored on both your hard drive and the server.
  • Accounts are local to the machine, and provide full integration with Mac OS X and its services.
  • Applications such as the Adobe and Microsoft Suites function properly.
  • Files are on the network, meaning that you can log into other computers and still have access to your documents.


When using a PHD, please note that your account is subject to the quotas of your network account. While your local machine can store much more data than your quota allows, if you exceed your quota in your home directory, your files will not be backed up, and will not be available on other computers.

While logged into your PHD account, you can easily see the status of your quota usage on the right side of the OS X menubar:

Quota status menu item

Since background syncs only happen every 60 minutes, you may notice some latency between when files are created on your local machine, and when they appear on the server (or vice versa). If you need a file to appear immediately, you can manually initiate a sync via the PHD menu (The double-house icon in the screenshot above).

 

Sync Exclusions

The following items are *not* synchronized to the server. Please keep your own backups.

  • iTunes libraries and all *.mp3 files
  • iPhoto libraries
  • Downloads folder
  • Virtual Machines
  • Dropbox folder
  • ~/Library/Mail folder (POP accounts are not backed up!)


If your home directory syncs are failing, please make sure to check your quota usage, and contact help@sbgrid.org if the problem persists.

Installing non-free software

It is common for lab workstations to contain proprietary, or "non-free", software such as the Adobe Creative Suite, Microsoft Office, or MATLAB.  If you need a program installed that isn't free, you need to make sure that a copy of that software is purchased or accounted for before requesting installation.

In many cases, your organization may contain a site license for some of these software titles that legally covers your work computer to run the software.  You are responsible for determining the presence of a site-license. You must consult with your PI or institutional IT department to check if a license needs to be purchased before contacting SBGrid to install the software. 

SBGrid does not participate in software piracy, and following these steps will ensure that we are appropriately abiding by the commercial software license agreements.
Installing Applications Without Admin Priviledges

If you would like to install an application on your mac that is not included by default, you may do so by placing the application file in your User account Applications folder, located at /Users/<username>/Applications (you may create this folder if it does not already exist). Note that this location is unique from the system Applications folder, which is located at /Applications.

At /Users/<username>/Applications, you are the owner of the folder and have write permissions and can install simple application files, such as Firefox.app, via drag and drop.

Some applications include installers which ask for the Administrator password. If this is the case, the installer is either modifying the system Applications folder, or needs to modify important system files to fully install itself.  If an application you need to install requires an Administrator password, please send an email to help@sbgrid.org.

Using Email

Lab Mailing Lists

Editing Preferences

 
You can view and edit your mailing list subscription options by logging into the Sympa mailing list manager at https://cmcd.hms.harvard.edu/sympa.
 

Unsubscribing

 
There is currently no way to completely unsubscribe from the lab mailing lists, but it is possible to disable receiving mail in your sympa preferences. To do this, log in via the link above, select the mailing list on the left, and then click "Subscriber Options". In the 'Receiving Mode' dropdown list, select 'no mail'.
 

Sending Mail

 
In order to send an email to your lab, just send an email to the mailing list address, which you can find by clicking on the mailing list name in the menu on the left of the page. The address will be listed in the top-center of the page after clicking that link. Sending an email to that address will send an email to everyone in your lab's group (i.e. the sch address will send mail to everyone in the sche group, with a [SCH] prefix).
 
Note: Replies to the 'SCHE' list do not send to the entire list, but only the sender of the originating message.
CMCD Email

You may access your cmcd.hms.harvard.edu mailing address in two ways; Through the webmail interface, and through IMAP.

Webmail

A webmail interface is available for you to use at http://cmcd.hms.harvard.edu/mail. To log in, simply enter your network username and password.

Configuring a Mail Client for IMAP

To configure your address in a client such as Mail.app or Thunderbird, use the following options:

  • Incoming Mail Server: crystal.harvard.edu, SSL enabled using port 465

  • Outgoing Mail Server: crystal.harvard.edu, SSL enabled using port 993

IMPORTANT NOTES:

If you're not planning on using your crystal email account, you must login to the webmail address above and set up forwarding for your account. To set up forwarding, click on "Settings" after logging in, then select "Vacation", and put the email address(es) you'd like your crystal email to forward to in the 'Forward Messages To' box. Optionally un-check the "Keep a copy" box if you don't want copies of your crystal email stored on our servers. Click "Save" to save your forwarding preferences.

All users of the CMCD network are expected to either check their crystal email periodically or have their email forwarded. Failure to be reachable by your crystal email account is grounds for account termination.

You can set a vacation responder in the webmail client by clicking 'Personal Settings' and then clicking the 'Vacation' tab.

Using SBGrid Programs

Running GUI Programs Remotely

It is currently possible to run programs remotely over SSH through a process called X-forwarding. To accomplish this, you may simply use the -Y flag in your SSH command, like so:

[user@host ~]$ ssh -Y user@remote-host

If you have to access your machine through a gateway (i.e., ssh to crystal.harvard.edu, then to a workstation), you can use a command such as:

[user@host ~]$ ssh -t -Y user@remote-host ssh -Y internal-host

After logging in to the remote host, you may launch programs by running the command name in a Terminal. With X-forwarding enabled, programs will launch their user interfaces on your computer, much like they would if you ran the program locally.

NOTE: On Mac OS X, you may need to change your display output in order for certain programs to run correctly. You can do this by opening the X11 preferences, clicking the output tab, and selecting a different option from the 'Colors:' dropdown (the default is "From Display"). 256 Colors will emulate an 8-bit color scheme for running applications such as WEB.

If you need to download a newer version of X11 for OS X, you can find the binaries here: http://xquartz.macosforge.org/trac/wiki

Running SBGrid Programs

Running programs from the SBGrid software tree should be as simple as opening a Terminal and typing the name of the application you would like to run. You may see a list of available software packages at: http://sbgrid.org/software.php
If you are unable to run programs in this manner, you'll need to add the appropriate lines to your shell configuration. Run echo $SHELL in a terminal to determine which file you should edit.

/bin/tcsh

Edit your ~/.cshrc file to contain:

if ( -e /programs/sbgrid.cshrc ) then
    source /programs/sbgrid.cshrc
endif

/bin/bash

Edit your ~/.bashrc file to contain:

if [ -f /programs/sbgrid.shrc ]; then
    . /programs/sbgrid.shrc
fi

You may also need to create a .bash_profile with the following contents:

if [ -f $HOME/.bashrc ]; then
  . $HOME/.bashrc
fi

 

Software Overrides

Most of the software titles installed in the SBGrid Suite have several versions available. Typically, we try to provide at least three versions of each software package, although very old or very new applications may have fewer versions. To override the default version, follow the instructions posted to the SBGrid Consortium website.