Development with Git

The Git Documentation site is a great resource for tutorials and details on all commands. If you are used to Subversion, the Git-SVN Crash Course is a good resource.

Initial Setup

Cloning the central repository

Your ssh keys must be authorized to access the central repository. All chisel accounts' ssh keys have been authorized to do so at the time of this writing. Contact Adam to add ssh keys for other machines (such as your desktop) if desired.

cd to your public_html directory on chisel:
```
cd ~/public_html/
```
clone the repository and create a working directory with git-clone:
```
git clone git@chisel.middlebury.edu:drupal.git
```
You should now have a working directory at ~/public_html/drupal/

Configuration

cd to your drupal directory:
```
cd ~/public_html/drupal/
```
Export the default versions of the two ignored files, .htaccess and sites/default/settings.php to your working directory:
```
git diff -p origin/core -- .htaccess | git apply -R
```
```
cp sites/default/default.settings.php sites/default/settings.php
```
Modify the .htaccess file. In particular you want to change the RewriteBase to point at the path of your Drupal installation. Change the line
```
  # RewriteBase /
```
to
```
RewriteBase /~yourname/drupal/
```
Modify your sites/default/settings.php to point at your database on chisel. Change $db_url to
```
$db_url = 'mysql://testuser:testpassword@localhost/yourname_drupal';
```
Make your sites/default/files/ directory writable by the webserver:
```
chmod -R 777 sites/default/files
```

Copy the database to development

Create a new database on chisel using http://chisel/phpmyadmin. If you preface the database name with "yourname_", the testuser account will have automatic access to the database. Otherwise, you will need to set up user privileges for the database. In order for drupal to work with the modules we have installed, you will need to grant the database user the following privileges:

Data
- SELECT
- INSERT
- UPDATE
- DELETE
- FILE
Structure
- CREATE
- ALTER
- INDEX
- DROP
- CREATE TEMPORARY TABLES
- SHOW VIEW
- CREATE ROUTINE
- ALTER ROUTINE
- EXECUTE
- CREATE VIEW
Administration
- LOCK TABLES
- REFERENCES

Dump the production data and import it into your database with the following command:

mysqldump -h production_db_host -u production_db_user -p production_db_database > ~/drupal.sql
mysql -utestuser -ptestpassword -D yourname_drupal < ~/drupal.sql

Log in as the admin user

Go to http://chisel.middlebury.edu/~yourname/drupal/?q=user and log in as 'admin'.

If you are not privy to the admin password, you can reset it in your database by running the following command:

echo "UPDATE users SET pass = MD5( 'MyPassword' ) WHERE uid = 1;" | mysql -utestuser -ptestpassword -D yourname_drupal

Browsing History

You can use the command-line program git-log and git-show-branchto browse the history, but a much better experience is provided by the gitk graphical browser.

gitk can be started from the git-gui's "Repository" menu, or from the command line:

SSH to chisel using an XTerm:
```
ssh -X chisel.middlebury.edu
```
cd to the working directory:
```
cd ~/public_html/drupal/
```
open gitk showing all branches:
```
gitk --all &
```

Committing Customizations

No customizations to Drupal should be made on the core or modules branches. These two branches are reserved for code coming directly from third-parties. Keeping our customizations off of these branches allows us to be able to easily see where our modifications to core code and modules reside and to maintain those modifications more easily through upgrades.

cd to the working directory:
```
cd ~/public_html/drupal/
```
Ensure that you are on the master branch (BC:5/27/2011- git-status "command not found" - ask Adam):
```
git-status
```
If not, check out your master branch:
```
git-checkout master
```
Get any changes from the central (origin) repository if desired:
```
git-pull origin master
```
Make some changes to files....
Stage changes to one or more files:
```
git-add path/to/desired/file
```
or stage all changes:
```
git-add .
```
Commit the staged changes to your repository:
```
git-commit -m "My commit message."
```
Repeat steps 4-6 as many times as desired.
Push your changes to the central repository:
```
git-push origin master
```

Push conflicts

Sometimes you get an error like the following when you go to push your changes to the central repository:

[afranco@chisel drupal-tmp (master)]$ git-push origin master
To git@chisel.middlebury.edu:drupal.git
! [rejected]        master -> master (non-fast forward)
error: failed to push some refs to 'git@chisel.middlebury.edu:drupal.git'

This error indicates that there are new change-sets in the central repository that come after commit that your new changes are based off of. To resolve this issue, the easiest way is to "rebase" your changes on top of those in the central repository. This is done with:

git-pull --rebase origin master

This is equivalent to:

git-fetch origin master
git-rebase origin/master

With this command, your change-sets since you diverged from the central repository's master branch will be set aside, your master branch updated from the central repository's, then your change-sets will be replayed. Once your repository has been rebased, you can push successfully:

git-push origin master

Adding or Upgrading a Module

cd to the working directory:
```
cd ~/public_html/drupal/
```
This process will wipe out your .htaccess file since it is untracked, so copy it to a location outside of your working directory:
```
mkdir ~/drupal-settings/; cp .htaccess ~/drupal-settings/
```
Check out your modules branch.
- You will initially need to create a modules branch that will track the central repository's modules branch. Do this with
```
git-checkout --track -b modules origin/modules
```
- If you have already created your modules branch, use
```
git-checkout modules
```
  to check out your modules branch, then
```
git-pull origin modules
```
  to bring your modules branch up to date with the central repository's modules branch.
cd to the modules directory:
```
cd sites/all/modules
```
Delete the original modules directory if upgrading so that we get rid of files that are no longer in the new version
```
rm -R modulename
```

fetch the new module tarball:

wget http://ftp.drupal.org/files/projects/modulename-6.x-2.6.tar.gz

Decompress the new module tarball:
```
tar xzf modulename-6.x-2.6.tar.gz
```
Delete the new module tarball:
```
rm modulename-6.x-2.6.tar.gz
```
Stage the changes (and additions):
```
git-add modulename
```
Stage the removals:
```
git-add -u modulename
```

Commit the changes (and additions/removals):

git-commit -m "Upgraded modulename to version 6.x-2.6 from tarball."

If you are upgrading multiple modules, repeat steps 5-11 for each module.
Check out your master branch:
```
git-checkout master
```
Merge the changes from the modules branch into the master branch:
```
git-merge modules
```
Note that any customizations to the module should be retained in the merge.
Copy your .htaccess file back:
```
cp ~/drupal-settings/.htaccess .
```
Verify that everything is hunky-dory and the new versions of modules work as expected.
Push all changes in your master and modules branches to the matching branches in the central (origin) repository:
```
git push origin modules
```
then
```
git push origin master
```
These can be both pushed at the same time with
```
git push origin master modules
```

Removing a Module

Removing a module is like upgrading one, but we use the git-rm command to delete files and stage the deletions.

cd to the working directory:
```
cd ~/public_html/drupal/
```
This process will wipe out your .htaccess file since it is untracked, so copy it to a location outside of your working directory:
```
mkdir ~/drupal-settings/; cp .htaccess ~/drupal-settings/
```
Check out your modules branch.
- You will initially need to create a modules branch that will track the central repository's modules branch. Do this with
```
git-checkout --track -b modules origin/modules
```
- If you have already created your modules branch, use
```
git-checkout modules
```
  to check out your modules branch, then
```
git-pull origin modules
```
  to bring your modules branch up to date with the central repository's modules branch.
cd to the modules directory:
```
cd sites/all/modules
```
Delete the original modules directory:
```
git-rm -rf modulename
```
Commit the deletion:
```
git-commit -m "Deleted modulename."
```
Check out your master branch:
```
git-checkout master
```
Merge the changes from the modules branch into the master branch:
```
git-merge modules
```
Note that any customizations to the module may result in merge conflicts that will need to be resolved (by using git-rm again on the master branch, then committing.
Copy your .htaccess file back:
```
cp ~/drupal-settings/.htaccess .
```
Verify that everything is hunky-dory and no dependencies are broken.
Push all changes in your master and modules branches to the matching branches in the central (origin) repository:
```
git push origin modules
```
then
```
git push origin master
```
These can be both pushed at the same time with
```
git push origin master modules
```

Upgrading the Drupal Core

Upgrading core code is similar to upgrading a module. The only difference to watch out for is that we want to delete everything in our working directory except the .git/ directory before we add in the new code.

cd to the working directory:
```
cd ~/public_html/drupal/
```
This process will wipe out your settings in .htaccess and sites/default/settings.php, so copy them to a location outside of your working directory:
```
mkdir ~/drupal-settings/; cp .htaccess sites/default/settings.php ~/drupal-settings/
```
Check out your core branch.
- You will initially need to create a core branch that will track the central repository's core branch. Do this with
```
git-checkout --track -b core origin/core
```
- If you have already created your core branch, use
```
git-checkout core
```
  to check out your core branch, then
```
git-pull origin core
```
  to bring your core branch up to date with the central repository's core branch.

Delete all files and directories except the .git directory

find . -maxdepth 1 ! -name '.git*' ! -name '.' ! -name '..' -exec rm -Rf {} \;

Find the latest version
- D6: Pressflow at https://launchpad.net/pressflow/+download
- D7: Drupal 7 at https://drupal.org/node/3060/release?api_version=103

Fetch the new Drupal tarball:

D6:

wget -O ../drupal.tar.gz http://launchpad.net/pressflow/6.x/6.16.77/+download/pressflow-6.16.77.tar.gz

D7:

wget -O ../drupal.tar.gz http://ftp.drupal.org/files/projects/drupal-7.25.tar.gz

Decompress the new Drupal tarball:

tar --strip-components 1 -xzf ../drupal.tar.gz

Delete the new Drupal tarball:
```
rm ../pressflow.tar.gz
```
Stage the changes (and additions):
```
git-add .
```
Stage the removals:
```
git-add -u .
```

Commit the changes (and additions/removals):

git-commit -m "Upgraded Drupal to version 6.16 from tarball."

Check out your modules branch:
```
git-checkout modules
```
Merge the changes from the core branch into the modules branch:
```
git-merge core
```
Note that any module additions will be retained in the merge.
While you are on the modules branch, upgrade any modules that need upgrading (as detailed above).
Check out your master branch:
```
git-checkout master
```
Merge the changes from the modules branch into the master branch:
```
git-merge modules
```
Note that any customizations should be retained in the merge.

Copy your .htaccess and sites/default/settings.php back:

cp ~/drupal-settings/.htaccess .; cp ~/drupal-settings/settings.php sites/default/

Verify that everything is hunky-dory and the new versions of modules work as expected.
Push all changes in your master, core and modules branches to the matching branches in the central (origin) repository:
```
git push origin master core modules
```

Git on Windows

Installing Git Extensions

Git Extensions is a toolkit aimed to make working with Git under Windows more intuitive. The shell extension will integrate in Windows Explorer and presents a context menu on files and directories. There is also a Visual Studio plug-in to use Git from Visual Studio.

Download Git Extension here:

http://sourceforge.net/projects/gitextensions/files/Git%20Extensions/Version%201.91/GitExtensions191SetupComplete.msi/download

Launch the Git Extensions installer and follow the installation screens.

The only changes from the defaults are:

On the Select SSH Client, select OpenSSH
When asked to specify line end support choose the option to leave line endings unchanged

Configuration Settings

General client configuration:

Launch Git Extensions
Click on the "settings" menu
Click on the "Global settings" tab
Input the proper values for the "User name" and "User email" fields
Click the "Ok" button

Creating SSH keys:

Refer to the Git Extension's User Manual that is installed with Git Extensions. This document can be accessed from the "Help" menu.

Once the SSH keys have been created, send the public SSH key to Adam Franco

Using Git on Windows

There is two ways of using Git On windows.

Git Exntesion Gui - use the User Manual for help if needed
Git Bash - the commands for using Git Bash on windows are the same as they are for Git for Linux.

Note**: If you are planning to use Git on windows with a samba share, you should clone the project using the windows tool. If you do not use the windows tool in this case there will be issues with file execute permissions, that will cause files to show up as changed.

Running git-gui/gitk remotely via X

This page describes how to set up an X-server on Windows and use it to display the GUI of graphical applications running on Linux servers.

While running the graphical Git tools directly on a Windows machine may be sufficient or preferred for many tasks, it can sometimes be advantageous to run GUI programs remotely via X. The primary case for this is to open the graphical tools (gui-gui and gitk) on production machines to allow browsing the history of the the production repository (including configuration changes).

Ignoring file-mode changes

There are some cases where the file-permissions don't translate properly between Linux hosts and Windows hosts over Samba shares. This can result in Git on one of the platforms showing every file as modified, drastically slowing down the operation of Git.

To fix this, run the following command while in your working directory:

git config core.filemode false

Production Deployment

The production copies of Drupal code are created just like the development copies: cloning from the central repository. Where they differ slightly is that their master branches include production configuration commits and therefor diverge from the central-repository's master branch. Changes made in production will generally not be pushed back to the central repository.

Version-control of production configuration

Unlike the development environments, the master branch on each webserver has its production configuration committed to it. These branches track the central repository's master branch, but diverge from it and include change-sets from the central master branch as well configuration commits.

After making changes to production config files please commit your changes with a message describing the change.

git status
git-add .
git-commit -m "Updated configuration with foo and bar."

These configuration change-sets should not be pushed back to the central repository, but will just live in the local repository on each machine.

Scripted deployment to Production

A Bash script lives on supersoaker that makes use of single-purpose ssh keys to run git-pull on all four webservers. To deploy to all 4 webservers, ssh to supersoaker as root, then run

deploy_drupal

Press enter at each prompt to deploy to the next webserver in the list.

Please test the operation of the new code on http://drupalmiddadmin.middlebury.edu/ and http://drupalmiisadmin.middlebury.edu/ before continuing to deploy to the other three webservers.

Manually deploying change-sets to Production

Manual updating of production is done by sshing to each webserver then running the following:

cd /var/www/drupal
git-pull

Note that the working directory must not have any uncommitted changes on it. If configuration changes have been made, please commit them (see below) before running git-pull.

A script will soon be written that will allow these commands to be run remotely with a single invocation on supersoaker.

Reverting Deployments

Let's say that you run a deployment to production but find that it caused more problems than it fixed. To resolve this issue we want to reset the production drupal directory back to its last operational point. You can do this all from the command line, but using gitk to identify your rollback point makes it much easier to see where you want to go.

Each deployment results in a merge-commit where the change-sets from the central-repository's master branch are merged with the production configuration change-sets on the webservers' master branch. To revert a single deployment, we want to revert to the second-to-last merge-commit.

Reverting with the gitk GUI

SSH to the first webserver with an XTerm (you could also open an XTerm from the Desktop of the webserver accessed through the VMWare client) and open the history browser, gitk:
```
ssh -X root@firehose
```
```
cd /var/www/drupal
```
```
gitk --all &
```
You should see a view that looks something like this
In this example, the change-set labeled "Fixed a javascript error..." (second from the top) is the one that introduced the issue, so we want to revert to the merge-commit before it was introduced. Locate the merge commit before the problems were introduced.
Right-click on the merge-commit and choose "Reset master branch to here":
Choose "Hard" as the reset type
Hit "Ok". This will result in a history that looks like the following:
What this shows is that the master branch is now at commit 23d8724d017199f58136761eb2ecdfd7442a4db5 the later merge-commit still in the repository, but will be removed during the next garbage-collection cycle.

Reverting from the command line

Rather than resetting through the GUI, you can also do so through the command line:

1. Use gitk or git-log to browse the history. Ex:

git-log

commit 904c5ea8409e1cbacfffc4b0d6207b40ba04474e

Merge: 23d8724... 9d0af3b...
Author: root <root@firehose.middlebury.edu>
Date:   Thu Apr 15 11:09:35 2010 -0400

Merge branch 'master' of git@chisel.middlebury.edu:drupal

commit 9d0af3b3621935527139ad25165411be85700021
Author: Adam Franco <afranco@middlebury.edu>
Date:   Thu Apr 15 11:08:58 2010 -0400

Fixed a javascript error in the library search portal caused by upgrading Views.

The library Quick-Search portal has a dependency on the jQuery UI Tabs javascript
provided by the Views module.

In Views 2.10 (commit c41b3281adde3b36307bbd1314a609700dfb4f1c)
the tabs instantiation was moved under the Drupal.Views.Tabs namespace to avoid
conflicts with alternatve versions of jquery, jquery UI.

commit 23d8724d017199f58136761eb2ecdfd7442a4db5
Merge: ca48b0c... ed71d6e...
Author: root <root@firehose.middlebury.edu>
Date:   Thu Apr 15 09:55:53 2010 -0400

Merge branch 'master' of git@chisel.middlebury.edu:drupal

commit ed71d6ef087ead838b177e0c77bd704d64a1a6ef
Author: Adam Franco <afranco@middlebury.edu>
Date:   Thu Apr 15 09:54:10 2010 -0400

Added whitespace to test deployment.

This commit should have no effect on operation and is just used to test deployment.

...

2. Copy the SHA1 ID (in this case, 23d8724d017199f58136761eb2ecdfd7442a4db5) of the merge-commit you want to revert to.

3. Run with that SHA1 ID

git-reset --hard 23d8724d017199f58136761eb2ecdfd7442a4db5

Production Notes

The production environment differs slightly from our development server in a few important ways:

Drupal caching is done via Memcache rather than in the database.
Page caching is handled Varnish reverse-proxy servers that sit in front of each webserver (on the same host machine) and cache anonymous requests (those without session cookies).
The production environment has two database servers and uses MySQL Proxy to fail over to the secondary one if the primary database goes down.

For the most part these differences will not result in development problems, but if necessary you can test by deploying to the test5 host which is configured with Memcache, Varnish, MySQL Proxy, and two test database servers.

Installing Varnish

Download the Varnish and Varnish-libs RPMs from Source Forge
Install expect (to get /usr/bin/mkpasswd which is required by Varnish):
```
yum install expect
```

Install the Varnish libs:

rpm -i varnish-libs-2.x.x-x.el5.x86_64.rpm

Install Varnish:
```
rpm -i varnish-2.x.x-x.el5.x86_64.rpm
```
Edit /etc/varnish/default.vcl and
1. Set the backend default block to
  backend default { .host = "127.0.0.1"; .port = "8080"; }
2. Add the sub vcl_recv block from https://wiki.fourkitchens.com/display/PF/Configure+Varnish+for+Pressflow.
Edit /etc/sysconfig/varnish and cange the DAEMON_OPTS line to use malloc rather than file-based caching. Also at the -a :80 so that Varnish will listen on port 80. Also set the thread_pools number to match the number of cores on the system as recommended by this tuning article. Finally, set the connection timeout a bit higher than default as described by http://blogs.osuosl.org/gchaix/2009/10/12/pressflow-varnish-and-caching/ for good measure.
Set Varnish to start on boot:
```
chkconfig varnish on
```
Edit /etc/httpd/conf/httpd.conf and any vhost config files to set Apache to listen on port 8080.
Restart Apache.
Start Varnish.

Varnish Tips

More information on how Varnish relates to Apache and the rest of the system can be found in this post on the LIS Blog.

Cache Clearing

A script has been placed on each production webserver that will clear the Varnish cache on that machine:

purge_varnish_cache

This script simply runs the following command:

varnishadm -S /etc/varnish/secret -T localhost:6082 purge.url '.*'

To clear the Drupal cache tables as well as the Varnish caches on all systems, log into supersoaker and run

clear_all_drupal_caches

Statistics and Info

Varnish has a few handy tools that help to visualize its performance.

varnishstat

Running varnishstat on a host will show statistics about how Varnish is operating, including cache hits, misses, pass-throughs, memory usage, etc. Read this tutorial for more info on how to interpret the output.

varnishhist

This program creates a histogram of requests where the x-axis is response-time, the y-axis is number of requests. The | character is used for cache hits and the # character signifies requests that are misses or pass-throughs.

varnishncsa

Outputs an Apache-style webserver log.

Tips and Tricks

View customizations we've made to a module

git-checkout master
git-diff origin/modules sites/all/modules/webform/

This can also be done on individual files.

Undo customizations to a module

git-checkout master
git-diff -p origin/modules sites/all/modules/webform/ | git-apply -R
git-add sites/all/modules/webform/
git-commit -m "Undid all of our customizations to webform."

Clear uncommitted changes

This command will get rid of any uncommitted changes in your working directory. Very handy if you have debugging statements hanging around that you don't want to commit.

git-reset --hard

Show the current branch in your Bash prompt

Add the following to your ~/.bashrc:

# Git completion script provides the $__git_ps1 variable
if [ -f /usr/local/share/git-completion.bash ]; then
source /usr/local/share/git-completion.bash
fi

# Prompt format
export PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ '

Library & ITS Wiki

Drupal Development

Contents