OpenSolaris

  subsites:   Code Reviews   Gates   Issues   Defects   Polls   Test   PKG   Planet   Mail
You are not signed in. Sign in or register.

OpenSolaris

Source Code Management Implementation Specification

(DRAFT)

Copyright 2006 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms.

$Id: d-scm-hosting.txt 270 2006-10-13 12:12:26Z $ SMI"

1. Summary

The SCM deployment method for opensolaris.org is outlined, with specific details presented for the initial Subversion support. Although the SCM facility is not intended to support the download of large static files, an HTTP URL scheme is defined such that it may be used for simple document sharing by the community.

2. Introduction

Source code and its manipulation is the key next step for fostering collaboration on opensolaris.org. Although numerous consolidations require distributed SCM support, a variety of efforts' needs are well met by a centralized SCM solution, such as Subversion. Accordingly, we present a means of implementing a hosting facility for a large number of repositories.

Assumptions guiding the implementation are as follows:

  • Contributors will particpate in one or more projects;
  • Projects will have one or more repositories;
  • There are no secret repositories, so all repositories will have anonymous read only access;
  • There will be two supported SCM solutions: one centralized, one distributed.

The implementation is restricted by the network security guidelines from our hosting organization, which merely means that the system will be running a minimized version of Solaris 10.

3. Repository access, write side

We'll use the SCMs tunnelled through SSH, implying the requirement that the SCMs can be tunnelled in such a manner. This moves the authentication and authorization to a model we understand well, and removes the need to understand the individual SCMs' distinct direct connect approaches.

The account on the server side will be placed in a chroot(2) environment and assigned a umask(2) of 0000, by means of a group-specific PAM session module. This environment will consist of a read-only home directory, populated with read-write loopback mounts of the repositories the user is a committer to and of enough of / and /usr to make the SCMs function.

3.1 Example paths.

As an example, Alice is a committer to the main and docs repositories of the foo project and to the main repository of the bar project. Her home directory would look, from a global perspective, like

/home/alice/ [read only]
/home/alice/bin [read only]
/home/alice/lib [read only]
/home/alice/usr/bin [read only]
/home/alice/usr/lib [read only]
... [as needed for svn function]
/home/alice/svn/bar/main [read write]
/home/alice/svn/foo/docs [read write]
/home/alice/svn/foo/main [read write]

Once chroot("/home/alice") has been issued, this tree becomes

/ [read only]
/bin [read only]
/lib [read only]
/usr/bin [read only]
/usr/lib [read only]
... [as needed for svn function]
/svn/bar/main [read write]
/svn/foo/docs [read write]
/svn/foo/main [read write]

With this approach, all committers have identical paths to all repositories. That is, the repositories have Subversion URLs

svn+ssh://[user]@svn.opensolaris.org/svn/bar/main
svn+ssh://[user]@svn.opensolaris.org/svn/foo/docs
svn+ssh://[user]@svn.opensolaris.org/svn/foo/main

for all users, although non-committers use the "anon" account. (These URLs may be compared to the CVS repository specifications

ext:developername at cvs dot sourceforge dot net:/cvsroot/[project]
server:anonymous at cvs dot sourceforge dot net:/cvsroot/[project]

and the SVN repository URL

https://svn.sourceforge.net/svnroot/[project]/[repository]/trunk

for SourceForge's implementations of repository hosting.)

Additional SCM schemes will have similar URLs, using the pattern

[scm].opensolaris.org/[scm]/[project]/[repository]

The automount table(s) will be populated from the application database, specifically from the user-project membership and project-repository association tables (or query results, depending on schema implementation). The /etc/auto_home file for our example involving Alice is

/home/alice -ro /export/home/pad /home/alice/bin -ro /bin ... \
/home/alice/svn/bar/main -rw /svn/bar/main ...

This output can be generated by explicit map construction, or by means of an executable automount map. See automount(1M) for further details.

The "usr" and "user" projects should be reserved from the project/community namespace for future use (if we elect to offer per-user repositories).

3.2. User authentication.

The machine hosting the SCM has a passwd(4) map generated automatically from the application database. (Eventually, a network name service should be used for the passwd(4) map.) At present, the passwd(4) entries are in "NP" mode, where no password is used, but the accounts are not locked. Authentication is instead done using SSH keys exclusively.

In part this is due to password hashing incompatibilities between the web application and Solaris, but the overall experience of using Subversion via SSH is unpleasant if manual password entry is required. Instead, efforts to document use of ssh-agent(1) and ssh-add(1) will be made.

4. Repository access, read side.

4.1 As repository.

Read only access to the repositories will be provided as anonymous access tunnelled through SSH. (For the purposes of initial deployment, we ignore for the moment the various browsable tree CGI scripts the various SCMs offer.) That is, an account entitled "anonymous" will be created with a scheme similiar to the example in Section 2, but with all repositories loopback mounted read-only.

Thus, URIs appear as

svn+ssh://anon at svn dot opensolaris dot org/svn/project/repo

4.2 As static content.

An HTTP server will be established with the various repositories accessible as loopback read-only mounts. Thus,

http://svn.opensolaris.org/svn/project/repo/foo.c

will retrieve the HEAD version of foo.c from the project/repo repository.

5. Future concerns.

5.1 Scaling.

In principle, the number of repositories or the number of operations on the repositories may exhaust the capabilities of a single system. The application may be partitioned initially by separating the SSH-based interactions from the HTTP-based interactions, but the ultimate solution will require splitting the repositories across multiple systems. Achieving this split without fragmenting the external namespace may be problematic, although having SCMs that function correctly across NFS could provide an easy means of alleviating loading issues.

5.2 Browsable trees.

Because of the technical attractiveness and modular structure of OpenGrok [1], we defer consideration of the use of per-SCM specific code search/browsing support to a separate, future discussion.

5.3 Additional repositories.

It is expected that users will also be permitted to have individual repositories. The project and community namespace has not been managed for conflict with the user namespace, so distinguished paths will be required.

6. References.

[1] Chandan Bellur Nandakumaraiah, OpenGrok code search and cross reference engine,http://www.opensolaris.org/os/project/opengrok/, 2005.

Stephen Hahn, PhD Solaris Kernel Development, Sun Microsystems
stephen dot hahn at sun dot com http://blogs.sun.com/sch/