Useful Tools
The following is a list of useful tools that you may find helpful while navigating your CS major/minor
If you have any tools you believe belong on this list, please email an ASI: Robert Lichenstein or Catherine Miller
Notes on Style
In order to easily read this guide, you should be aware of the notation being used. A statement may read:
$ [user@my-machine ~]: command <variable> some/kind/of/path
The $ is simply denoting that this line is from a console window. In other words, the text following a $ should be read as though it were in a Terminal window.
The [ ] enclose a typical terminal prompt, which usually contains the name of the user @ the machine they are currently logged into.
The ~ is shorthand for home folder (and is generally the default location a console opens to). If you changed to a different directory, that ~ would become the bottom-level folder of your current path (e.g. if you are in /home/<username>/Documents/cs101, you would just see cs101 -- NOTE: You may change these settings by editing your ~/.bashrc file)
The : will be followed by the command being referenced.
Any variable value that changes based on the user will be enclosed in < >. (e.g. when a command requires a <username>, we don't actually enter "<username>", but instead your username).
A command that requires some/kind/of/path will need you to specify a path for the command to run to/from. Generally speaking, the path can be relative or absolute
Application Managers
Miniconda
Miniconda is a stand-alone release of the popular application package manager conda. Its older brother is Anaconda -- the conda package manager bundled with 150+ packages.
You can download Miniconda for your operating system here. The official documentation is here. The following is designed to be a sort of quick-start guide, NOT a set of full documentation.
Why?
Miniconda is useful for two major reasons:
- Installing packages can be a pain. Depending on how your Python environment is structured, and which packages depend on other packages, installing Python modules manually can often cause conflicts or unforeseen problems. A package manager keeps track of dependencies and installs everything for you.
- Environments. Environments allow you segment different sets of packages from one another to reduce conflicts.
- Example: Say you often program with packages X and Y, both of which require package Z to function properly. But, what happens if package X requires version 1.0 of package Z, while package Y requires version 2.0 of package Z. An application package manager allows you to create one environment with the correct versions of X and Z, and a completely separate environment with the correct versions of Y and Z. Depending on what you are working on, you can switch between the environments, as needed. Further, say you use package Q in every project. You can install package Q a level above your environments so it is always available.
Use for CS at Midd
Without a package manager like Miniconda, any time you want to do any kind of work with Python on the lab machines and a package you need is not installed, you have to ask the admins for it. In turn, the admins have to prepare an image with the specific Python package you need, plan a scheduled downtime for the lab, and reimage the lab to include the new package. This could potentially take a few days. If you then determine you need another package, the cycle repeats itself.
Miniconda allows you to create your own, self-contained Python environments owned by you, which are usable on all lab machines at any time. This is especially useful if you plan to use the HTCondor system, as you can guarantee that all the packages you need will always be available for condor to use.
Installation
You can get the installer you need from this page. The Windows version comes in an executable that you just need to run. The Mac version comes as a bash script, which will be similar to the Linux instructions below. If you need assistance getting Miniconda on your personal machine, talk with an ASI.
For the Linux lab machines, do the following:
- Download the "64-bit (bash installer)". From a lab machine, this can be achieved by opening a terminal and running:
$ [user@lab-machine ~] wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
- Run the installer. Assuming you are in the same directory as where you just downloaded the bash script above, all you need to do is run the command:
$ [user@lab-machine ~] bash Miniconda3-latest-Linux-x86_64.sh
- -This should result in the following output:
Welcome to Miniconda3 <version> (by Continuum Analytics, Inc.) In order to continue the installation process, please review the license agreement. Please, press ENTER to continue
- -Press ENTER
- -The Terms and Conditions will pop up. Press SPACE until you are met with the following prompt:
Do you approve the license terms? [yes|no]
- -Type "yes"
Miniconda3 will now be installed into this location: /home/<username>/miniconda3 - Press ENTER to confirm the location - Press CTRL-C to abort the installation - Or specify a different location below [/home/<username>/miniconda3] >>>
- -Press ENTER
- The Miniconda installer will then install a bunch of stuff. When you are met with the following prompt:
Do you wish the installer to prepend the Miniconda3 install location to PATH in your /home/<username>/.bashrc ? [yes|no] [no] >>>
- -Type "yes"
You're all set! Now you should be able to create environments and install packages and have them available on whichever lab machine (including basin) that you are using.
Managing Environments
Making an Environment
To view your environments, use the command:
$ [user@machine ~] conda info --envs
If you have just installed Miniconda, you will receive output something like this:
# conda environments: # root * /home/<username>/miniconda3
This is telling you that you are currently using the highest level (root) Python environment. If you followed the installation instructions above, that environment should be /home/<username>/miniconda3.
To make a new environment, we invoke the create command:
$ [user@machine ~] conda create -n <env_name> -y python=<python_version> <list_of_packages_to_install>
Example:
$ [user@machine ~] conda create -n myEnv -y python=3.6 scipy numpy pytorch pillow pip matplotlib
If you receive this message then your environment has been created:
# # To activate this environment, use: # > source activate myEnv # # To deactivate this environment, use: # > source deactivate myEnv #
Deleting an Environment
To delete an environment, make sure it is first deactivated:
$ [user@machine ~] source deactivate <name_of_environment>
Then, run the remove command:
$ [user@machine ~] conda remove -n <name_of_environment> --all
Conda will list to you all of the packages it will remove, accept this at the prompt:
$ Proceed ([y]/n)? y
Activating/Changing Environments
When you first login to a console, you will start off in your root environment (indicated by the asterisk).
$ [user@machine ~] conda info --envs # conda environments: # myEnv /home/<username>/miniconda3/envs/myEnv root * /home/<username>/miniconda3
To change this, we use the activate command:
$ [user@machine ~] source activate myEnv $ (myEnv) [user@machine ~]
The active environment will be listed in parentheses before your usual prompt. To verify this:
$ [user@machine ~] conda info --envs # conda environments: # myEnv * /home/<username>/miniconda3/envs/myEnv root /home/<username>/miniconda
This means that invoking python (or python3) will use the packages from the myEnv environment.
To deactivate the current environment:
$ (myEnv) [user@machine ~] source deactivate $ [user@machine ~]
Pip Within Conda
pip is the official tool for installing Python packages. It has access to an extremely large number of package databases from which you can install a variety of Python software. Conda allows you to embed separate versions of pip inside your environments, making it even easier to get access to software that conda may not host in its databases.
With the correct environment activated, you can:
$ (myEnv) [user@machine ~] conda install -y pip
to get pip. From now on, when using the myEnv environment, you will have access to packages available through pip install. If you want all of your environments to have access to pip, install it in the root environment (i.e. deactivate all environments and then use conda install)
File System Resources
By enrolling in a Middlebury CS course, a CS user account on Basin is created for you. This account inherits your Middlebury username and password, but has resources that are distinct from your college account. To utilize this resource, follow the instructions for accessing your files on Basin.
Home Folders
The account that is generated for you is a typical Unix user account. This means it has the standard folder structure you would expect on a Linux or Mac machine (i.e. Desktop, Documents, Pictures, etc).
You are the owner of your account and all files contained within your subdirectory. You are free to delete folders/files inside your user directory as you wish (now, remember, "Just because you can, doesn't mean you should" -- but if you really have an itch to delete everything, you can). You can create subfolders for classes or projects whenever you want. You can upload files to or download files from your user directory freely. It is YOUR account.
By default, your folder is viewable by anyone on the network (permission level rwxr-xr-x, or 755). This means people who are not you can view and execute (but NOT write to) files inside your user directory. If you would like to change this, talk to an ASI or research the chmod command.
public_html Folders
One of the many folders created inside your home folder is your public_html folder. The purpose of the public_html folder is to house any and all documents that you want to be public-facing. This includes a webpage!
By default, this directory contains a single file, index.html.
<html><head> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <title>Empty page</title> </head><body> I have not yet set up my home page. </body></html>
The CS web server is pointed at each user's ~/public_html/index.html file. Consider this your "homepage" (i.e. if someone types in the url www.cs.middlebury.edu/~<your_username>, they will land on your index.html page). Any pages you want to link to from your homepage must be within your public_html folder.
Shell Commands/Tools
PuTTY
NOTE: As of Windows 10, ssh is supported on Windows through the PowerShell application -- PuTTY still works, but you may not need it
PuTTY is an open source SSH client developed for the Windows platform. You can download it here (just the putty.exe binary form will do, but if you are feeling ambitious you can download the .msi installer for all of the tools).
The PuTTY client has many options for customization (similar to optional arguments with the ssh command). But, to get the basic usage out of it, all you need to do is supply the full hostname of the machine you want to connect to.
Example
If the connection can be made, you will be prompted with a window asking you the username that you would like to login with. After entering a username, you will be prompted for the password associated with the username. Upon successful authentication, you should see a terminal-esque window like what you would see in a Unix environment.
SCP
Secure CoPy is a command that combines the ssh command with the cp (copy) command. SCP can be used to push local files to a remote server, or it can be used to get a file from a remote server and save it locally.
You can find the manual page here or by typing "man scp" in a terminal window.
NOTE: This command only works for UNIX environments. For Windows, see the pscp utility provided by PuTTY
Typical Usage
$ [user@my-machine ~]: scp path/to/file/to/send <username>@remote-location:path/to/location/to/put/file
OR
$ [user@my-machine ~]: scp <username>@remote-location:path/to/file/to/get path/to/location/to/put/file/locally
Examples
$ [user@my-machine ~]: scp ./Documents/my_homepage.html user@basin.cs.middlebury.edu:~/public_html/homepage/
$ [user@my-machine ~]: scp user@basin.cs.middlebury.edu:~/cs101/homework1.py ~/Documents/this_semester/cs101/
Tip: You can select multiple source files in one scp command. SCP will evaluate as many files as you select until it reads a destination location (i.e. if you provide many local files, it knows to continue reading local files as sources, and when it reads a remote location, that is the destination, and vice versa). You can also supply the -r argument to recursively push entire directories.
SSH
Secure SHell is a network protocol that allows remote console (i.e. terminal) login from one machine to another. The Middlebury CS department machines all support SSH from the on-campus network. In addition, there is one machine (basin) that is specifically given a hole in the Middlebury firewall to allow off-campus connections.
You can find the manual page for the SSH command here or by typing "man ssh" on a Mac or Linux terminal. From the manual, the SSH command looks complicated; it's not! There are many optional arguments supported, but to get basic functionality all you need to supply to the command is the username you want to connect with and which machine you want to connect to. If the connection can be established, you will be prompted for the password of the account you are trying to connect with. If the connection cannot be established, you will be given some form of a "cannot resolve hostname" or "connection timed out" error (usually this means the machine is either disconnected from the network, or powered off).
NOTE: The examples below are all in UNIX format. To use SSH from a Windows machine, see the #PuTTY section.
Verifying Authenticity
The first time you remotely connect to a machine, you will be given a warning that the authenticity of the machine you are trying to connect to cannot be verified. Assuming you have correctly entered the name of a Middlebury managed machine or a Middlebury IP address, you can safely ignore this and enter "yes". In the grand scheme of things, if it's your first time connecting to a machine and you are expecting to get this response, you can usually ignore it and enter "yes". But, for the sake of completeness, you should be aware that spoofing is a thing.
Typical Usage
$ [user@my-machine ~]: ssh <username>@machine-name
OR
$ [user@my-machine ~]: ssh <username>@ip-address
Examples
$ [user@my-machine ~]: ssh <username>@killington.cs.middlebury.edu
$ [user@my-machine ~]: ssh <username>@140.233.20.155
Tip: If you are off-campus and need to connect to a specific machine, you can tunnel through basin to the machine you need with two ssh commands.
$ [user@my-machine ~]: ssh <username>@basin.cs.middlebury.edu
$ [username@basin ~]: ssh <username>@killington.cs.middlebury.edu
OR
$ [user@my-machine ~]: ssh -t <username>@basin.cs.middlebury.edu ssh <username>@killington.cs.middlebury.edu
Tmux
Tmux is a Terminal MUltipleXer. It allows for multiple consoles within a single window, as well as the ability to detach and reattach processes from a single session. You can find the manual page here or by typing "man tmux" in a console.
NOTE: Tmux is a UNIX-only command. While untested by this author, the popular Windows-equivalent is ConEmu (short for Console Emulator). It allows for multiple Command Prompt or PuTTY sessions to be emulated alongside one another.
NOTE 2: If you are a Mac user and would like this utility installed on your machine, go here. It is installed on all CS Linux machines by default.
Keyboard Shortcuts
Keyboard shortcuts can be edited in the file ~/.tmux.conf. Here is a popular community cheat sheet.
Prefix
Tmux commands come after what is referred to as a "command prefix". By default, "ctrl" + "b" is the command prefix. You can edit your prefix in #Keyboard_Shortcuts. In this tutorial, "ctrl" + "b" will be shortened to CB, and commands will be written in the format CB --> <key_to_press_after_prefix>.
Examples
You can achieve the most basic functionality of Tmux simply by calling it:
$ [user@my-machine ~]: tmux
Nothing will appear to happen, except the bottom of your console should change color. This means you are in a tmux session with one pane.
Let's split our single tmux pane horizontally. CB --> "
Now we have 2 consoles, one on top of the other. These consoles are independent of one another. You can use one to ssh to a remote server, and the other to search for local files on your machine, for example.
You can swap panes with CB --> o or, you can allow tmux to read mouse input by inserting the line set -g mouse on into your .tmux.conf file. You can split panes as many times as you want (or until they become illegible!) with CB --> " for horizontal and CB --> % for vertical.
You can detach a tmux session from the current console with CB --> d (or by typing tmux detach). This means the process remains running, but as a distinct program separate from the current console window (useful, for example, if you are ssh'd into a server and want a process to continue running after you logout). You can reattach a tmux session with:
$ [user@my-machine ~]: tmux attach
If you intend to have multiple sessions of tmux running inside a single console session, you can name, attach, detach, and switch between them:
$ [user@my-machine ~]: tmux new -s session1
$ [user@my-machine ~]: tmux detach
$ [user@my-machine ~]: tmux new -s session2
$ [user@my-machine ~]: tmux detach
$ [user@my-machine ~]: tmux attach -t session1
$ [user@my-machine ~]: tmux switch -t session2
If you don't remember all of the sessions you have running, you can use the call:
$ [user@my-machine ~]: tmux list-sessions
to remind you what they are all called.
To kill the current pane use CB --> x. This will prompt at the bottom of your screen for a y/n to confirm.
There is so much more you can do with tmux, but this is just some basic functionality to get you up and running.
