Middlebury

Library & ITS Wiki

Library & ITS Wiki

Drupal Development

Contents

Development with Git

The Git Documentation site is a great resource for tutorials and details on all commands. If you are used to Subversion, the Git-SVN Crash Course is a good resource.

Initial Setup

Setting up your public and private HTML directories

User directories are turned on in apache on anvil and set to public_html, so anything you put in that directory will be accessible to the web server. For a bit of added security, we place the code files in a private_html directory, both of which should be in your home directory. This only needs to be done once.

  1. mkdir ~/public_html
  2. chown USERNAME:webstaff ~/public_html
  3. mkdir ~/private_html

Getting the update scripts project

We have a project containing all of the script files needed to build the various Drupal sites. You will want a copy of this in your home directory.

  1. git clone git@git.middlebury.edu:web/drupal7-drush_scripts.git
  2. In the project you will find an 'aliases' directory. This contains the various drush alias files. Copy all of these into your local .drush directory.
    cp ~/drupal7-drush_scripts/aliases/* ~/.drush/

The drush alias files allow you to use a short reference in drush commands, like "drush @midd" to run a drush command on your copy of the Middlebury site from any path on the filesystem. To do this, each alias file contains a path to that copy of the Drupal project. After you set up your copies of Drupal below, you will want to edit these alias files and change the path to the correct one for your copy of Drupal, e.g. change /var/www/midd to /home/USER/private_html/midd.

Cloning the central repository

Your ssh keys must be authorized to access the central repository. All chisel accounts' ssh keys have been authorized to do so at the time of this writing. Contact Adam to add ssh keys for other machines (such as your desktop) if desired.

  1. cd to your private_html directory on anvil:
    cd ~/private_html/
  2. clone the repository and create a working directory with git-clone (replace "SITE" with a short name for the site, i.e. 'midd'):
    • D6:
      git clone git@chisel.middlebury.edu:web/drupal.git SITE
    • D7:
      git clone git@git.middlebury.edu:web/drupal7.git SITE
  3. You should now have a working directory at ~/private_html/SITE/
  4. Create a symbolic link to the code in your public_html directory:
    • ln -s ~/private_html/SITE ~/public_html/SITE
  5. chown -h USERNAME:webstaff ~/public_html/SITE

Configuration

  1. cd to your drupal directory:
    cd ~/private_html/SITE/
  2. Copy the sample .htaccess file from the drupal7-drush_scripts project into your site root (there is also a .htaccess.D6.example file in the project):
    cp ~/drupal7-drush_scripts/.htaccess.D7.example .htaccess
  3. Edit the .htaccess file
  4. Modify the .htaccess file. In particular you want to change the RewriteBase to point at the path of your Drupal installation. Change the line
      # RewriteBase /SITE/
    to
    RewriteBase /~yourname/SITE/
  5. Set up a site directory for your Drupal project. This should match the domain of the site in production. For example, if you're working on the main Middlebury site:
    mkdir sites/www.middlebury.edu
  6. Copy the sample settings.php file from the drupal7-drush_scripts project into your site root. There is a settings.php file in each of the sub-directories of this project. For example:
    cp ~/drupal7-drush_scripts/midd/settings.php sites/www.middlebury.edu/
  7. Edit the settings.php file and make these changes:
    • In the database connection string, change the name of the database to USER_SITE
    • In the database connection string, add the password for the test account
    • If you are using this site to test an upgrade from Drupal 6 to 7, set the blocked_ips configuration variable to an empty array. See here for details.
      $conf['blocked_ips'] = array();
    • Ensure that the mimedetect_magic configuration variable is set to
      $conf['mimedetect_magic'] = '/usr/share/file/magic';
  8. Create a symbolic link to the local copy of the files directory on anvil. For example, for the MIIS website:
    ln -s /var/www/images/www.miis.edu/files sites/www.miis.edu/files
    • Note: for the sites that we have not yet upgraded to Drupal 7, if you're working on a D7 version of the project, you will need to append "-drupal7" to the path to the local files directory on anvil. This applies to www.middlebury.edu, courses.middlebury.edu, and courses.miis.edu. For example, for the Middlebury site:
      ln -s /var/www/images/www.middlebury.edu-drupal7/files sites/www.middlebury.edu/files
  9. Create a symbolic link for the settings file from the default directory to the site domain directory:
    ln -s sites/www.middlebury.edu/settings.php sites/default/settings.php
  10. Change ownership of the various files we just created:
    • chown -Rh USER:webstaff sites/www.middlebury.edu
    • chown -Rh USER:webstaff sites/default

Copy the database to development

There is a database refresh script, refresh.sh in each site's directory in the drupal7-drush_scripts project. For each of these, copy the file refresh.config.sh.example to refresh.config.sh and edit it to match your local configuration. Here is an example of a config file for the refresh script:

#!/bin/bash

DRUSH_ALIAS="@SITE"

SOURCE_DB="anvil_SITE"
SOURCE_DB_HOST="localhost"
SOURCE_DB_USER="USER"
SOURCE_DB_PASS="PASSWORD"

DEST_DB="USER_SITE"
DEST_DB_HOST="localhost"
DEST_DB_USER="USER"
DEST_DB_PASS="PASSWORD"

LOCAL_USER="USER"
LOCAL_PATH="/home/USER/private_html/SITE"
LOCAL_FILES_PRIVATE="sites/SITE.middlebury.edu/files/private"
LOCAL_FILES_PUBLIC="sites/SITE.middlebury.edu/files"
LOCAL_FILES_TEMP="/tmp"
LOCAL_BASE_PATH="anvil.middlebury.edu/~USER/"
LOCAL_CDN_BASE="//anvil.middlebury.edu/~USER/cdn"

Portions of the values of the above variables in caps will need to be edited. Once you have set up your config file for the refresh script you can refresh your test site by running the refresh script for the site (e.g. midd/refresh.sh) or you can run a refresh script for all of your test sites by running the refresh.sh script in the root of the drupal7-drush_scripts project. Comment out the sites you do not wish to update locally.

Browsing History

You can use the command-line program git-log and git-show-branchto browse the history, but a much better experience is provided by the gitk graphical browser.

gitk can be started from the git-gui's "Repository" menu, or from the command line:

  1. SSH to chisel using an XTerm:
    ssh -X chisel.middlebury.edu
  2. cd to the working directory:
    cd ~/public_html/drupal/
  3. open gitk showing all branches:
    gitk --all &

Committing Customizations

No customizations to Drupal should be made on the core or modules branches. These two branches are reserved for code coming directly from third-parties. Keeping our customizations off of these branches allows us to be able to easily see where our modifications to core code and modules reside and to maintain those modifications more easily through upgrades.

  1. cd to the working directory:
    cd ~/public_html/drupal/
  2. Ensure that you are on the master branch (BC:5/27/2011-  git-status "command not found" - ask Adam):
    git-status
    If not, check out your master branch:
    git-checkout master
  3. Get any changes from the central (origin) repository if desired:
    git-pull origin master
  4. Make some changes to files....
  5. Stage changes to one or more files:
    git-add path/to/desired/file
    or stage all changes:
    git-add .
  6. Commit the staged changes to your repository:
    git-commit -m "My commit message."
  7. Repeat steps 4-6 as many times as desired.
  8. Push your changes to the central repository:
    git-push origin master

Push conflicts

Sometimes you get an error like the following when you go to push your changes to the central repository:

[afranco@chisel drupal-tmp (master)]$ git-push origin master
To git@chisel.middlebury.edu:drupal.git
! [rejected]        master -> master (non-fast forward)
error: failed to push some refs to 'git@chisel.middlebury.edu:drupal.git'

This error indicates that there are new change-sets in the central repository that come after commit that your new changes are based off of. To resolve this issue, the easiest way is to "rebase" your changes on top of those in the central repository. This is done with:

git-pull --rebase origin master

This is equivalent to:

git-fetch origin master
git-rebase origin/master

With this command, your change-sets since you diverged from the central repository's master branch will be set aside, your master branch updated from the central repository's, then your change-sets will be replayed. Once your repository has been rebased, you can push successfully:

git-push origin master

Adding or Upgrading a Module

Most of our modules are updated by downloading the release tarball from drupal.org. These release tarballs include a version tag in their module.info files (which isn't in the git repository) that indicates to the update system which version of the module you are running. For some modules which aren't available on drupal.org (e.g. Lawrence's mm_tweaks), update them with the Managing modules with git subtree process.

  1. cd to the working directory:
    cd ~/public_html/drupal/
  2. This process will wipe out your .htaccess file since it is untracked, so copy it to a location outside of your working directory:
    mkdir ~/drupal-settings/; cp .htaccess ~/drupal-settings/
  3. Check out your modules branch.
    • You will initially need to create a modules branch that will track the central repository's modules branch. Do this with
      git-checkout --track -b modules origin/modules
    • If you have already created your modules branch, use
      git-checkout modules
      to check out your modules branch, then
      git-pull origin modules
      to bring your modules branch up to date with the central repository's modules branch.
  4. cd to the modules directory:
    cd sites/all/modules
  5. Upgrade the module. If you are upgrading multiple modules, repeat these steps for each module.
    1. Delete the original modules directory if upgrading so that we get rid of files that are no longer in the new version
      rm -R modulename
    2. fetch the new module tarball:
      wget http://ftp.drupal.org/files/projects/modulename-6.x-2.6.tar.gz
    3. Decompress the new module tarball:
      tar xzf modulename-6.x-2.6.tar.gz
    4. Delete the new module tarball:
      rm modulename-6.x-2.6.tar.gz
    5. Stage the changes (and additions):
      git-add modulename
    6. Stage the removals:
      git-add -u modulename
    7. Commit the changes (and additions/removals):
      git-commit -m "Upgraded modulename to version 6.x-2.6 from tarball."
  6. Check out your master branch:
    git-checkout master
  7. Merge the changes from the modules branch into the master branch:
    git-merge modules
    Note that any customizations to the module should be retained in the merge.
  8. Copy your .htaccess file back:
    cp ~/drupal-settings/.htaccess .
  9. Verify that everything is hunky-dory and the new versions of modules work as expected.
  10. Push all changes in your master and modules branches to the matching branches in the central (origin) repository:
    git push origin modules
    then
    git push origin master
    These can be both pushed at the same time with
    git push origin master modules

Managing modules with git subtree

Generally we want to use tarball releases from drupal.org since they include version tags in the module.info files that isn't in the repository. However, there are a few modules (such as Lawrence University's MM Tweaks) that don't go through the Drupal.org release process and are only available as a git repository. One option is to download a .zip of their module and add it like others. Alternatively, we can use git subtree to merge their repository (and all of its commits) into our repository under a sub-directory.

Adding

  1. Check out the modules branch for the 3rd party module:
    git checkout modules
  2. You need to get the remote repository's commits into your repository so that they can be referenced:
    git remote add mm_tweaks https://github.com/jaydansand/mm_tweaks.git 
    git fetch mm_tweaks
  3. Add the external project into a sub-directory. This can be done in two ways:
    • Full history:
      git subtree add -P sites/all/modules/mm_tweaks mm_tweaks/master
    • Squashed history:
      git subtree add --squash -P sites/all/modules/mm_tweaks mm_tweaks/master
      Squashing the history will create a new commit that encompasses the latest state of the module's repository so that our project doesn't include the full history. This is useful for projects with long histories, but prevents us from seeing itemized commits as we update the module.
  4. Merge onto master:
    git checkout master 
    git merge modules

Updating

When managing a module with git subtree you can easily update to the latest version:

  1. Check out the modules branch for the 3rd party module:
    git checkout modules
  2. Get the latest commits for the module
    git fetch mm_tweaks
  3. Merge the external project's new commits into a sub-directory. This can be done in two ways:
    • Full history:
      git subtree merge -P sites/all/modules/mm_tweaks mm_tweaks/master
    • Squashed history:
      git subtree merge --squash -P sites/all/modules/mm_tweaks mm_tweaks/master
  4. Merge onto master:
    git checkout master 
    git merge modules

Moving to releases

If the module that you are managing with git subtree moves to the normal release process, you can simply start updating it as you would other modules. No reworking is necessary.

Modules using git subtree

Drupal7
Name Path Repository URL Squash?
phpCAS sites/all/libraries/CAS git://github.com/Jasig/phpCAS.git yes
MM Tweaks sites/all/modules/mm_tweaks https://github.com/jaydansand/mm_tweaks.git no
MM Porter sites/all/modules/mmporter http://git.drupal.org/sandbox/adamfranco/1475768.git no
NodePorter sites/all/modules/nodeporter http://git.drupal.org/sandbox/adamfranco/1475756.git no

Removing a Module

Removing a module is like upgrading one, but we use the git-rm command to delete files and stage the deletions.

  1. cd to the working directory:
    cd ~/public_html/drupal/
  2. This process will wipe out your .htaccess file since it is untracked, so copy it to a location outside of your working directory:
    mkdir ~/drupal-settings/; cp .htaccess ~/drupal-settings/
  3. Check out your modules branch.
    • You will initially need to create a modules branch that will track the central repository's modules branch. Do this with
      git-checkout --track -b modules origin/modules
    • If you have already created your modules branch, use
      git-checkout modules
      to check out your modules branch, then
      git-pull origin modules
      to bring your modules branch up to date with the central repository's modules branch.
  4. cd to the modules directory:
    cd sites/all/modules
  5. Delete the original modules directory:
    git-rm -rf modulename
  6. Commit the deletion:
    git-commit -m "Deleted modulename."
  7. Check out your master branch:
    git-checkout master
  8. Merge the changes from the modules branch into the master branch:
    git-merge modules
    Note that any customizations to the module may result in merge conflicts that will need to be resolved (by using git-rm again on the master branch, then committing.
  9. Copy your .htaccess file back:
    cp ~/drupal-settings/.htaccess .
  10. Verify that everything is hunky-dory and no dependencies are broken.
  11. Push all changes in your master and modules branches to the matching branches in the central (origin) repository:
    git push origin modules
    then
    git push origin master
    These can be both pushed at the same time with
    git push origin master modules

Upgrading the Drupal Core

Upgrading core code is similar to upgrading a module. The only difference to watch out for is that we want to delete everything in our working directory except the .git/ directory before we add in the new code.

  1. cd to the working directory:
    cd ~/public_html/drupal/
  2. This process will wipe out your settings in .htaccess and sites/default/settings.php, so copy them to a location outside of your working directory:
    mkdir ~/drupal-settings/; cp .htaccess sites/default/settings.php ~/drupal-settings/
  3. Check out your core branch.
    • You will initially need to create a core branch that will track the central repository's core branch. Do this with
      git-checkout --track -b core origin/core
    • If you have already created your core branch, use
      git-checkout core
      to check out your core branch, then
      git-pull origin core
      to bring your core branch up to date with the central repository's core branch.
  4. Delete all files and directories except the .git directory
    find . -maxdepth 1 ! -name '.git*' ! -name '.' ! -name '..' -exec rm -Rf {} \;
  5. Find the latest version
  6. Fetch the new Drupal tarball:
    • D6:
      wget -O ../drupal.tar.gz http://launchpad.net/pressflow/6.x/6.16.77/+download/pressflow-6.16.77.tar.gz
    • D7:
      wget -O ../drupal.tar.gz http://ftp.drupal.org/files/projects/drupal-7.25.tar.gz
  7. Decompress the new Drupal tarball:
    tar --strip-components 1 -xzf ../drupal.tar.gz
  8. Delete the new Drupal tarball:
    rm ../pressflow.tar.gz
  9. Stage the changes (and additions):
    git-add .
  10. Stage the removals:
    git-add -u .
  11. Commit the changes (and additions/removals):
    git-commit -m "Upgraded Drupal to version 6.16 from tarball."
  12. Check out your modules branch:
    git-checkout modules
  13. Merge the changes from the core branch into the modules branch:
    git-merge core
    Note that any module additions will be retained in the merge.
  14. While you are on the modules branch, upgrade any modules that need upgrading (as detailed above).
  15. Check out your master branch:
    git-checkout master
  16. Merge the changes from the modules branch into the master branch:
    git-merge modules
    Note that any customizations should be retained in the merge.
  17. Copy your .htaccess and sites/default/settings.php back:
    cp ~/drupal-settings/.htaccess .; cp ~/drupal-settings/settings.php sites/default/
  18. Verify that everything is hunky-dory and the new versions of modules work as expected.
  19. Push all changes in your master, core and modules branches to the matching branches in the central (origin) repository:
    git push origin master core modules

Git on Windows

Installing Git Extensions

Git Extensions is a toolkit aimed to make working with Git under Windows more intuitive. The shell extension will integrate in Windows Explorer and presents a context menu on files and directories. There is also a Visual Studio plug-in to use Git from Visual Studio.

Download Git Extension here:

http://sourceforge.net/projects/gitextensions/files/Git%20Extensions/Version%201.91/GitExtensions191SetupComplete.msi/download

Launch the Git Extensions installer and follow the installation screens.

The only changes from the defaults are:

  1. On the Select SSH Client, select OpenSSH
  2. When asked to specify line end support choose the option to leave line endings unchanged

Configuration Settings

General client configuration:

  1. Launch Git Extensions
  2. Click on the "settings" menu
  3. Click on the "Global settings" tab
  4. Input the proper values for the "User name" and "User email" fields
  5. Click the "Ok" button

Creating SSH keys:

Refer to the Git Extension's User Manual that is installed with Git Extensions.  This document can be accessed from the "Help" menu.

Once the SSH keys have been created, send the public SSH key to Adam Franco

Using Git on Windows

There is two ways of using Git On windows.

  1. Git Exntesion Gui -  use the User Manual for help if needed
  2. Git Bash - the commands for using Git Bash on windows are the same as they are for Git for Linux.

Note**: If you are planning to use Git on windows with a samba share, you should clone the project using the windows tool.  If you do not use the windows tool in this case there will be issues with file execute permissions, that will cause files to show up as changed.

Running git-gui/gitk remotely via X

This page describes how to set up an X-server on Windows and use it to display the GUI of graphical applications running on Linux servers.

While running the graphical Git tools directly on a Windows machine may be sufficient or preferred for many tasks, it can sometimes be advantageous to run GUI programs remotely via X. The primary case for this is to open the graphical tools (gui-gui and gitk) on production machines to allow browsing the history of the the production repository (including configuration changes).

Ignoring file-mode changes

There are some cases where the file-permissions don't translate properly between Linux hosts and Windows hosts over Samba shares. This can result in Git on one of the platforms showing every file as modified, drastically slowing down the operation of Git.

To fix this, run the following command while in your working directory:

git config core.filemode false

Production Deployment

The production copies of Drupal code are created just like the development copies: cloning from the central repository. Where they differ slightly is that their master branches include production configuration commits and therefor diverge from the central-repository's master branch. Changes made in production will generally not be pushed back to the central repository.

Version-control of production configuration

Drupal 6

Unlike the development environments, the master branch on each webserver has its production configuration committed to it. These branches track the central repository's master branch, but diverge from it and include change-sets from the central master branch as well configuration commits.

DrupalProdConfigBranch.png

After making changes to production config files please commit your changes with a message describing the change.

git status
git-add .
git-commit -m "Updated configuration with foo and bar."

These configuration change-sets should not be pushed back to the central repository, but will just live in the local repository on each machine.

Drupal 7

In Drupal 7 we are not version-controlling site configuration files. Make changes on one server, then rsync them to the other two machines:

Test to see if there is any outstanding config changes with the "dry-run" -n flag:

rsync -avn /var/www/drupal7/sites/ firehose:/var/www/drupal7/sites/
rsync -avn /var/www/drupal7/sites/ watercannon:/var/www/drupal7/sites/

If a file is changed, you can see diffs with vimdiff using its scp file paths:

vimdiff /path/to/file scp://remotehost//path/to/file


Make configurations changes on Firehose, then rsync them:

rsync -av /var/www/drupal7/sites/ firehose:/var/www/drupal7/sites/
rsync -av /var/www/drupal7/sites/ watercannon:/var/www/drupal7/sites/

Scripted deployment to Production

A Bash script lives on supersoaker that makes use of single-purpose ssh keys to run git-pull on all four webservers. To deploy to all 4 webservers, ssh to supersoaker as root, then run

deploy_drupal

Press enter at each prompt to deploy to the next webserver in the list.

Please test the operation of the new code on http://drupalmiddadmin.middlebury.edu/ and http://drupalmiisadmin.middlebury.edu/ before continuing to deploy to the other three webservers.

Manually deploying change-sets to Production

Manual updating of production is done by sshing to each webserver then running the following:

cd /var/www/drupal
git-pull

Note that the working directory must not have any uncommitted changes on it. If configuration changes have been made, please commit them (see below) before running git-pull.

A script will soon be written that will allow these commands to be run remotely with a single invocation on supersoaker.

Reverting Commits

Let's say that you run a deployment to production but find that it caused more problems than it fixed. To resolve this issue we want to revert the problematic commits. Reverting means to make a new commit that is the inverse of a previous commit.

  1. Find the problematic commit SHA1 id[s].
  2. Run
    git revert XXXXXXX
    where XXXXXX is the SHA1 id of the most recent commit you wish to revert. This will create a new commit that undoes XXXXXXX's changes.
  3. If you need to revert multiple commits, rerun git revert on the next older commit[s].
  4. Push your new reverting commits and deploy again.

If you want to replace the code that you reverted you can simply revert the revert-commits.

Resetting Deployments

Caution: Use Drupal_Development#Reverting Commits if at all possible. Resetting will cause way more confusion and be harder to sort out.

Let's say that you run a deployment to production but find that it caused more problems than it fixed. To resolve this issue we want to reset the production drupal directory back to its last operational point. You can do this all from the command line, but using gitk to identify your rollback point makes it much easier to see where you want to go.

Each deployment results in a merge-commit where the change-sets from the central-repository's master branch are merged with the production configuration change-sets on the webservers' master branch. To reset a single deployment, we want to rest to the second-to-last merge-commit.

Resetting with the gitk GUI

  1. SSH to the first webserver with an XTerm (you could also open an XTerm from the Desktop of the webserver accessed through the VMWare client) and open the history browser, gitk:
    ssh -X root@firehose
    cd /var/www/drupal
    gitk --all &
  2. You should see a view that looks something like this DrupalProdGitk.png
    In this example, the change-set labeled "Fixed a javascript error..." (second from the top) is the one that introduced the issue, so we want to reset to the merge-commit before it was introduced. Locate the merge commit before the problems were introduced.
  3. Right-click on the merge-commit and choose "Reset master branch to here": DrupalProdGitkReset.png
  4. Choose "Hard" as the reset type
    DrupalProdGitkReset2.png
  5. Hit "Ok". This will result in a history that looks like the following: DrupalProdGitkReset3.png
    What this shows is that the master branch is now at commit 23d8724d017199f58136761eb2ecdfd7442a4db5 the later merge-commit still in the repository, but will be removed during the next garbage-collection cycle.

Resetting from the command line

Rather than resetting through the GUI, you can also do so through the command line:

1. Use gitk or git-log to browse the history. Ex:

git-log
commit 904c5ea8409e1cbacfffc4b0d6207b40ba04474e

Merge: 23d8724... 9d0af3b...
Author: root <root@firehose.middlebury.edu>
Date:   Thu Apr 15 11:09:35 2010 -0400

Merge branch 'master' of git@chisel.middlebury.edu:drupal

commit 9d0af3b3621935527139ad25165411be85700021
Author: Adam Franco <afranco@middlebury.edu>
Date:   Thu Apr 15 11:08:58 2010 -0400

Fixed a javascript error in the library search portal caused by upgrading Views.

The library Quick-Search portal has a dependency on the jQuery UI Tabs javascript
provided by the Views module.

In Views 2.10 (commit c41b3281adde3b36307bbd1314a609700dfb4f1c)
the tabs instantiation was moved under the Drupal.Views.Tabs namespace to avoid
conflicts with alternatve versions of jquery, jquery UI.

commit 23d8724d017199f58136761eb2ecdfd7442a4db5
Merge: ca48b0c... ed71d6e...
Author: root <root@firehose.middlebury.edu>
Date:   Thu Apr 15 09:55:53 2010 -0400

Merge branch 'master' of git@chisel.middlebury.edu:drupal

commit ed71d6ef087ead838b177e0c77bd704d64a1a6ef
Author: Adam Franco <afranco@middlebury.edu>
Date:   Thu Apr 15 09:54:10 2010 -0400

Added whitespace to test deployment.

This commit should have no effect on operation and is just used to test deployment.

...

2. Copy the SHA1 ID (in this case, 23d8724d017199f58136761eb2ecdfd7442a4db5) of the merge-commit you want to reset to.

3. Run with that SHA1 ID

git-reset --hard 23d8724d017199f58136761eb2ecdfd7442a4db5

Setting up new sites in production (Drupal 7)

Required Configuration

Directories

Create the sites directory:

mkdir /var/www/drupal7/sites/www.hostname.edu
cd /var/www/drupal7/sites/www.hostname.edu

Create the files directory and subdirectories on the network share:

mkdir /mnt/images/www.hostname.edu
cd /mnt/images/www.hostname.edu
mkdir files
chown apache:apache files/

# Make a symlink to the high-availability files link in <code>/var/www/images</code>
cd /var/www/drupal7/sites/www.hostname.edu/
ln -s /var/www/images/www.hostname.edu/files

Site configuration

Copy the default configuration:

cp ../default/default.settings.php ./settings.php

Edit the settings.php file and be sure to add at least the following:

# Database
$databases['default']['default'] = array(
  'driver' => 'mysql',
  'database' => 'ddddddd',
  'username' => 'uuuuuuu',
  'password' => 'pppppppp',
  'host' => 'hhhhhhh.middlebury.edu',
  'prefix' => '',
);

# Who are our proxy servers? So that logs are properly attributed...
$conf['reverse_proxy'] = TRUE;
$conf['reverse_proxy_addresses'] = array('140.233.2.166', '140.233.2.167', '140.233.2.169', '140.233.2.170');

Apache configuration

Front-end Apache configuration:

# Forms Midd site configuration
<VirtualHost *:8080>
    ServerName www.hostname.edu
#    ServerAlias www
    ServerAlias hostname.edu

    ServerAdmin webmaster@middlebury.edu
    DocumentRoot /var/www/drupal7
    ErrorLog "|/bin/logger -t www.hostname.edu -p local3.info"
    # Use the client's IP rather than the Varnish Proxy's.
    # Append the response time (microseconds) and session cookie value to help with finding slow pages.
    CustomLog  "|/bin/logger -t www.hostname.edu -p local1.info" "%h \"%{X-Forwarded-For}i\" %l %u \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %D %{SSESSxxxxxxxxxxxxxxxxxx}C"

    <Directory />
        AddOutputFilterByType DEFLATE text/html text/plain text/css application/x-javascript

        Options FollowSymLinks
        AllowOverride All
    </Directory>

    RewriteEngine on
    RewriteRule /\.git/ - [R=404,L]
    RewriteRule /(install\.php|CHANGELOG\.txt|INSTALL\.txt|INSTALL\.mysql\.txt|INSTALL\.pgsql\.txt|LICENSE\.txt|MAINTAINERS\.txt|UPGRADE\.txt)(.*)? - [R=403|L]

    # Expand to full host-name
#    RewriteCond %{HTTP_HOST}    ^www$ [NC]
#    RewriteRule ^/(.*)          http://www.hostname.edu/$1 [L,R=301]
    RewriteCond %{HTTP_HOST}    ^hostname.edu [NC]
    RewriteRule ^/(.*)          http://www.hostname.edu/$1 [L,R=301]

</VirtualHost>

Maintenance Apache configuration (for cron jobs). Note that we don't actually need DNS entries for these hostnames as we will target localhost but pass this hostname as the Host: header in the request.


# Forms midd configuration
<VirtualHost *:8080>
    ServerName hostnameadmin.middlebury.edu
    ServerAlias hostnameadmin

    ServerAdmin webmaster@middlebury.edu
    DocumentRoot /var/www/drupal7
    ErrorLog "|/bin/logger -t www.hostname.edu -p local3.info"
    # Use the client's IP rather than the Varnish Proxy's.
    # Append the response time (seconds) and session cookie value to help with finding slow pages.
    CustomLog  "|/bin/logger -t www.hostname.edu -p local1.info" "%h \"%{X-Forwarded-For}i\" %l %u \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %D %{SSESSxxxxxxxxxxxxxxx}C"

    <Directory />
        AddOutputFilterByType DEFLATE text/html text/plain text/css application/x-javascript
        Options FollowSymLinks
        AllowOverride All
    </Directory>

    <IfModule mod_php5.c>
        php_value memory_limit 0000M
        php_value max_input_time 000
        php_value max_execution_time 000
    </IfModule>

    RewriteEngine on
    RewriteRule /\.git/ - [R=404,L]

</VirtualHost>

Cron jobs

Add one cron job that runs once an hour for basic cleanup tasks:

23 * * * *      if [[ `hostname -s` = "supersoaker" ]]; then    curl --silent --header 'Host: hostnameadmin.middlebury.edu' http://localhost:8080/cron.php?cron_key=xxxxxxxxxxxxxxxxxx > /dev/null       ; fi

If needed, add a less frequent cron job to sync users:

3 4 * * *               if [[ `hostname -s` = "supersoaker" ]]; then    curl --silent --header 'Host:  hostnameadmin.middlebury.edu' 'http://localhost:8080/cron.php?cron_key=xxxxxxxxxxxxxxx&casmmsync_sync_users=TRUE' > /dev/null     ; fi

Deploy scripts

The deploy_drupal7 script doesn't need updates, but the cache-clearing script does.

Edit /usr/local/bin/clear_all_drupal7_caches and add a new line for the new service:

drush -r /var/www/drupal7/ --uri=http://www.hostname.edu cache-clear all


Optional configuration -- SSL

Set up certs in Apache on the proxy hosts if they are not already covered.

Add the following to the site's settings.php:

/**
 * Convince Drupal to make all URLs in content HTTPS even though our
 * application server his working without SSL (behind the front-end SSL-terminating Apache and Varnish)
 */
$_SERVER['HTTPS'] = 'on';

Maybe more is needed?

Optional configuation -- Memcache

Set up a new memcache daemon for this site if not upgrading. This is usually hosted on the database server cluster. Each site needs it's own daemon since they will conflict otherwise.

Add the memcache configuration to the site's settings.php:

# Memcache
$conf['cache_inc'] = './sites/all/modules/memcache/memcache.inc';
$conf['memcache_servers'] = array(
  'mmmmmm.middlebury.edu:000000' => 'default',
);

Production Notes

The production environment differs slightly from our development server in a few important ways:

  1. Drupal caching is done via Memcache rather than in the database.
  2. Page caching is handled Varnish reverse-proxy servers that sit in front of each webserver (on the same host machine) and cache anonymous requests (those without session cookies).
  3. The production environment has two database servers and uses MySQL Proxy to fail over to the secondary one if the primary database goes down.

For the most part these differences will not result in development problems, but if necessary you can test by deploying to the test5 host which is configured with Memcache, Varnish, MySQL Proxy, and two test database servers.

Installing Varnish

  1. Download the Varnish and Varnish-libs RPMs from Source Forge
  2. Install expect (to get /usr/bin/mkpasswd which is required by Varnish):
    yum install expect
  3. Install the Varnish libs:
    rpm -i varnish-libs-2.x.x-x.el5.x86_64.rpm
  4. Install Varnish:
    rpm -i varnish-2.x.x-x.el5.x86_64.rpm
  5. Edit /etc/varnish/default.vcl and
    1. Set the backend default block to
      backend default {
      .host = "127.0.0.1";
      .port = "8080";
      }
    2. Add the sub vcl_recv block from https://wiki.fourkitchens.com/display/PF/Configure+Varnish+for+Pressflow.
  6. Edit /etc/sysconfig/varnish and cange the DAEMON_OPTS line to use malloc rather than file-based caching. Also at the -a :80 so that Varnish will listen on port 80. Also set the thread_pools number to match the number of cores on the system as recommended by this tuning article. Finally, set the connection timeout a bit higher than default as described by http://blogs.osuosl.org/gchaix/2009/10/12/pressflow-varnish-and-caching/ for good measure.
  7. Set Varnish to start on boot:
    chkconfig varnish on
  8. Edit /etc/httpd/conf/httpd.conf and any vhost config files to set Apache to listen on port 8080.
  9. Restart Apache.
  10. Start Varnish.

Varnish Tips

More information on how Varnish relates to Apache and the rest of the system can be found in this post on the LIS Blog.

Cache Clearing

A script has been placed on each production webserver that will clear the Varnish cache on that machine:

purge_varnish_cache
This script simply runs the following command:
varnishadm -S /etc/varnish/secret -T localhost:6082 purge.url '.*'
To clear the Drupal cache tables as well as the Varnish caches on all systems, log into supersoaker and run
clear_all_drupal_caches

Statistics and Info

Varnish has a few handy tools that help to visualize its performance.

varnishstat

Running varnishstat on a host will show statistics about how Varnish is operating, including cache hits, misses, pass-throughs, memory usage, etc. Read this tutorial for more info on how to interpret the output.

varnishhist

This program creates a histogram of requests where the x-axis is response-time, the y-axis is number of requests. The | character is used for cache hits and the # character signifies requests that are misses or pass-throughs.

varnishncsa

Outputs an Apache-style webserver log.

Tips and Tricks

View customizations we've made to a module

git-checkout master
git-diff origin/modules sites/all/modules/webform/

This can also be done on individual files.

Undo customizations to a module

git-checkout master
git-diff -p origin/modules sites/all/modules/webform/ | git-apply -R
git-add sites/all/modules/webform/
git-commit -m "Undid all of our customizations to webform."

Clear uncommitted changes

This command will get rid of any uncommitted changes in your working directory. Very handy if you have debugging statements hanging around that you don't want to commit.

git-reset --hard

Show the current branch in your Bash prompt

Add the following to your ~/.bashrc:

# Git completion script provides the $__git_ps1 variable
if [ -f /usr/local/share/git-completion.bash ]; then
source /usr/local/share/git-completion.bash
fi

# Prompt format
export PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ '
Powered by MediaWiki