XClose

MPHY0021: Research Software Engineering With Python

Home
Menu

Working with multiple remotes

Distributed versus centralised

Older version control systems (cvs, svn) were "centralised"; the history was kept only on a server, and all commits required an internet.

Centralised Distributed
Server has history Every user has full history
Your computer has one snapshot Many local branches
To access history, need internet History always available
You commit to remote server Users synchronise histories
cvs, subversion(svn) git, mercurial (hg), bazaar (bzr)

With modern distributed systems, we can add a second remote. This might be a personal fork on github:

In [1]:
import os
top_dir = os.getcwd()
git_dir = os.path.join(top_dir, 'learning_git')
working_dir = os.path.join(git_dir, 'git_example')
os.chdir(working_dir)
In [2]:
%%bash
git checkout master
git remote add rits git@github.com:ucl-rits/github-example.git
git remote -v
origin	git@github.com:UCL/github-example.git (fetch)
origin	git@github.com:UCL/github-example.git (push)
rits	git@github.com:ucl-rits/github-example.git (fetch)
rits	git@github.com:ucl-rits/github-example.git (push)
Switched to branch 'master'

We can push to a named remote:

In [3]:
%%writefile Pennines.md

Mountains In the Pennines
========================

* Cross Fell
* Whernside
Overwriting Pennines.md
In [4]:
%%bash
git commit -am "Add Whernside"
[master 5ce2cc9] Add Whernside
 1 file changed, 1 insertion(+)
In [5]:
%%bash
git push -uf rits master
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
/tmp/ipykernel_10269/3737658953.py in <module>
----> 1 get_ipython().run_cell_magic('bash', '', 'git push -uf rits master\n')

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2417             with self.builtin_trap:
   2418                 args = (magic_arg_s, cell)
-> 2419                 result = fn(*args, **kwargs)
   2420             return result
   2421 

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/IPython/core/magics/script.py in named_script_magic(line, cell)
    140             else:
    141                 line = script
--> 142             return self.shebang(line, cell)
    143 
    144         # write a basic docstring:

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/decorator.py in fun(*args, **kw)
    230             if not kwsyntax:
    231                 args, kw = fix(args, kw, sig)
--> 232             return caller(func, *(extras + args), **kw)
    233     fun.__name__ = func.__name__
    234     fun.__doc__ = func.__doc__

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188 
    189         if callable(arg):

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/IPython/core/magics/script.py in shebang(self, line, cell)
    243             sys.stderr.flush()
    244         if args.raise_error and p.returncode!=0:
--> 245             raise CalledProcessError(p.returncode, cell, output=out, stderr=err)
    246 
    247     def _run_script(self, p, cell, to_close):

CalledProcessError: Command 'b'git push -uf rits master\n'' returned non-zero exit status 128.

Referencing remotes

You can always refer to commits on a remote like this:

In [6]:
%%bash
git fetch
git log --oneline --left-right rits/master...origin/master
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: ambiguous argument 'rits/master...origin/master': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
/tmp/ipykernel_10269/748777457.py in <module>
----> 1 get_ipython().run_cell_magic('bash', '', 'git fetch\ngit log --oneline --left-right rits/master...origin/master\n')

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2417             with self.builtin_trap:
   2418                 args = (magic_arg_s, cell)
-> 2419                 result = fn(*args, **kwargs)
   2420             return result
   2421 

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/IPython/core/magics/script.py in named_script_magic(line, cell)
    140             else:
    141                 line = script
--> 142             return self.shebang(line, cell)
    143 
    144         # write a basic docstring:

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/decorator.py in fun(*args, **kw)
    230             if not kwsyntax:
    231                 args, kw = fix(args, kw, sig)
--> 232             return caller(func, *(extras + args), **kw)
    233     fun.__name__ = func.__name__
    234     fun.__doc__ = func.__doc__

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188 
    189         if callable(arg):

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/IPython/core/magics/script.py in shebang(self, line, cell)
    243             sys.stderr.flush()
    244         if args.raise_error and p.returncode!=0:
--> 245             raise CalledProcessError(p.returncode, cell, output=out, stderr=err)
    246 
    247     def _run_script(self, p, cell, to_close):

CalledProcessError: Command 'b'git fetch\ngit log --oneline --left-right rits/master...origin/master\n'' returned non-zero exit status 128.

To see the differences between remotes, for example.

To see what files you have changed that aren't updated on a particular remote, for example:

In [7]:
%%bash
git diff --name-only origin/master
fatal: ambiguous argument 'origin/master': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
/tmp/ipykernel_10269/3488311237.py in <module>
----> 1 get_ipython().run_cell_magic('bash', '', 'git diff --name-only origin/master\n')

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2417             with self.builtin_trap:
   2418                 args = (magic_arg_s, cell)
-> 2419                 result = fn(*args, **kwargs)
   2420             return result
   2421 

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/IPython/core/magics/script.py in named_script_magic(line, cell)
    140             else:
    141                 line = script
--> 142             return self.shebang(line, cell)
    143 
    144         # write a basic docstring:

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/decorator.py in fun(*args, **kw)
    230             if not kwsyntax:
    231                 args, kw = fix(args, kw, sig)
--> 232             return caller(func, *(extras + args), **kw)
    233     fun.__name__ = func.__name__
    234     fun.__doc__ = func.__doc__

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188 
    189         if callable(arg):

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/IPython/core/magics/script.py in shebang(self, line, cell)
    243             sys.stderr.flush()
    244         if args.raise_error and p.returncode!=0:
--> 245             raise CalledProcessError(p.returncode, cell, output=out, stderr=err)
    246 
    247     def _run_script(self, p, cell, to_close):

CalledProcessError: Command 'b'git diff --name-only origin/master\n'' returned non-zero exit status 128.

When you reference remotes like this, you're working with a cached copy of the last time you interacted with the remote. You can do git fetch to update local data with the remotes without actually pulling. You can also get useful information about whether tracking branches are ahead or behind the remote branches they track:

In [8]:
%%bash
git branch -vv
  gh-pages 13aee04 Add github pages YAML frontmatter
* master   5ce2cc9 Add Whernside

Hosting Servers

Hosting a local server

  • Any repository can be a remote for pulls
  • Can pull/push over shared folders or ssh
  • Pushing to someone's working copy is dangerous
  • Use git init --bare to make a copy for pushing
  • You don't need to create a "server" as such, any 'bare' git repo will do.
In [9]:
bare_dir = os.path.join(git_dir, 'bare_repo')
os.chdir(git_dir)
In [10]:
%%bash
mkdir -p bare_repo
cd bare_repo
git init --bare
Initialized empty Git repository in /home/runner/work/rsd-engineeringcourse/rsd-engineeringcourse/ch02git/learning_git/bare_repo/
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint: 
hint: 	git config --global init.defaultBranch <name>
hint: 
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint: 
hint: 	git branch -m <name>
In [11]:
os.chdir(working_dir)
In [12]:
%%bash
git remote add local_bare ../bare_repo
git push -u local_bare master
Branch 'master' set up to track remote branch 'master' from 'local_bare'.
To ../bare_repo
 * [new branch]      master -> master
In [13]:
%%bash
git remote -v
local_bare	../bare_repo (fetch)
local_bare	../bare_repo (push)
origin	git@github.com:UCL/github-example.git (fetch)
origin	git@github.com:UCL/github-example.git (push)
rits	git@github.com:ucl-rits/github-example.git (fetch)
rits	git@github.com:ucl-rits/github-example.git (push)

You can now work with this local repository, just as with any other git server. If you have a colleague on a shared file system, you can use this approach to collaborate through that file system.

Home-made SSH servers

Classroom exercise: Try creating a server for yourself using a machine you can SSH to:

ssh <mymachine>
mkdir mygitserver
cd mygitserver
git init --bare
exit
git remote add <somename> ssh://user@host/mygitserver
git push -u <somename> master

SSH keys and GitHub

Classroom exercise: If you haven't already, you should set things up so that you don't have to keep typing in your password whenever you interact with GitHub via the command line.

You can do this with an "ssh keypair". You may have created a keypair in the Software Carpentry shell training. Go to the ssh settings page on GitHub and upload your public key by copying the content from your computer. (Probably at .ssh/id_rsa.pub)

If you have difficulties, the instructions for this are on the GitHub website.