This chapter documents various tasks for preparing a major, minor or an update GAP release. To clarify the terminology, releasing 4.X.Y+1 after 4.X.Y is called a "minor release" and releasing 4.X+1.Y after 4.X.Z is called a "major release". An "update release" means releasing the latest officially distributed version of the core GAP system with new and updated packages (thus, the version number of the core GAP system stays the same).
The wrapping and publication processes are essentially the same for all three kinds of releases with some minor differences. For example, there is some more work to set up new paths on the ftp server for the major release, and there is no need to update main GAP manuals for the update release.
Changes that should appear in the next release should first appear in the the stable branch for the next minor release or in the master branch for the next major release. Normally they may first be developed and documented in feature branches, then tested and only after that merged into the appropriate release branch.
The dev/Updates directory is used to describe changes intended for the next release. It contains the README file explaining in more details how to document changes, and two templates: template-long with explanatory comments and template-short without them. To document a change, copy one of these templates to an appropriately named new file and fill in the details. This should be done in the same branch where the change itself has been developed in order to keep changes and their documentation together and facilitate tracking those changes which are ready for the release.
During the release preparation, files with these descriptions will be used to compose the overview of changes introduced in the release. When they include test code, it will be used to extend the test file tst/bugfix.tst.
Documenting changes, do not describe the wrong behaviour in present tense like in:
"Function XYZ returns the wrong result when computing ABC."
Rather, write it like in these examples:
"Function XYZ returned the wrong result when computing ABC."
"Fixed an error in function XYZ which returned the wrong result when computing ABC."
"Function XYZ now returns the correct result when computing ABC."
Otherwise, it will sound as if we knowingly introduced a new bug.
Please assume that the text from your description will be pasted verbatim into the overview of changes for the release announcement (only cross-references and other needed GAPDoc markup will be added).
After the release, descriptions of those changes from dev/Updates which made their way into the release should be moved to a new directory with a name containing the release version (e.g. dev/Updates4.5.7) using git mv command to keep them under version control. The dev/Updates directory in the branch from which the release has been made must be reset to contain only the README file and templates (and possibly descriptions of changes that has been added there after the release but before such ressetting will take place).
WARNING: The content of the dev/Updates directory may look different dependently on the branch! For example, in the integration branch it may list changes ready to both next minor and major releases, as well as changes merged into the integration branch for testing. It is important to update your clone of the repository to the proper branch (normally, stable) before resetting dev/Updates.
Here is an overview about the hard- and software needed for the wrapping process (at least with the currently available scripts and tools).
a UNIX/Linux machine
a disc partition with at least several GB of space
the standard sh shell and basic UNIX utilities like cp, mv or cmp
git
a TeX installation (which can process big files)
a C-compiler and make
tar, gzip, bzip2, zip, and python (for repacking into the various formats)
an installed GAP
persons who can provide Windows binaries of the GAP kernel
The following descriptions will explain what has to be done in the various steps, and will mention some tools which are available in the dev/DistributionUpdate/dist45 directory of the GAP git repository (the name dist45 reflects that these tools were revised for the GAP 4.5 release; we are continuing to use them to produce all subseqent releases).
Here we describe preparational steps for wrapping a minor or major release. Since an update release has no changes in the core GAP system, to wrap an update release you may proceed with steps described in Section 2.2-3.
Before thinking about wrapping apply all the standard tests described in Chapter 6 to a checked out git release branch and make sure there are no problems. (But also do the tests again later, starting with the archives which will be actually delivered to the users!)
setvar* files.For each of the three release types, some basic variables for the scripts used below are set in the files setvarmajor, setvarminor and setvarupdate in the dev/DistributionUpdate/dist45 directory. Follow instructions from that files to adjust the variables as required. Normally you will need to change only one or two lines with version numbers and do not need to edit anything else, except when setting up release wrapping on a new machine. In that case, choose a directory for the wrapping which can hold at least several GB of data and specify it as DISTROOT. Find the path to the directory with the merged package archive (we assume that it is available locally) and specify it as MERGEDPKGLOC.
Update file.Changes from previous version(s) are accumulated in individual files in the dev/Updates directory, as explained in 2.1-2.
Test code contained there may be tested regularly in the period between releases using the testupdates target from the GAP test suite (see Chapter 6) to ensure that it works in various settings.
The test code may be copied into tst/bugfix.tst at any stage, as soon as it is clear that the relevant change will be a part of the next release. At latest, it should appear in the version of tst/bugfix.tst which will be actually tested in the release candidate and delivered to the users (exceptions may be made for the test code which may be checked only interactively or requies a very specific testing environment).
Usually, copying of the test code to tst/bugfix.tst may happen gradually in several iterations if there will be several wrapping attempts; it can be left out while the wrapped archives are not yet expected to become release candidates and were just produced as a part of nightly/weekly tests.
Remark: formerly it was possible to use the file Update.g from dev/DistributionUpdate/maindist to extract test code, e.g. by calling
d := UpdateData("../../Update");;
l := FixesAndOthersData(d);;
PrintTestLines("guck.tst", l);
to extract it to the file guck.tst. Now Update.g needs to be adjusted since test codes are now distributed across multiple files. The testupdates target from the GAP test suite extracts tests from these files, but it does not save them to a file, testing them "on the fly".
CHANGES.md fileIf regular nightly/weekly tests go well and you hope to wrap a suitable release candidate, you may start to prepare an overview of changes that will appear in the next release in CHANGES.md, combining it from descriptions provided in files from the dev/Updates directory.
You may postpone this step until the very release, or perform it in several iterations, keeping track on new entries that will appear in the dev/Updates directory. You may group various entries together and re-order them so that entries which are likely to be interesting for more GAP users are closer to the top of each category. While converting plain text to markdown, add cross-references and other markup as appropriate. Ideally, you need only to add the markup, and the text of descriptions need not further editing, but please evaluate it critically. At least you may need to change tenses or reorder the text to better fit surrounding items. Also, do not forget to describe new packages added for the redistribution or packages with essential changes in their new versions.
Remark: formerly it was possible to use the file Util.g to extract descriptions from the dev/Update file and roughly sort them into several categories, e.g. by calling
d := UpdateData("../../Update");;
l := FixesAndOthersData(d);;
PrintDescriptionLines("guck.mixer", l);
to extract it to the file guck.mixer. Now Util.g needs to be adjusted since test codes are now distributed across multiple files, and we need to produce GAPDoc markup and not Mixer.
Now we explain how to wrap the archives of the GAP source distribution (note that the GAP binaries for Windows are not a part of the GAP source distribution, and they should added to the -win.zip archive at a later step).
The GAP source distribution will be produced in the form of .tar.gz, .tar.bz2 and .zip archives, and also in the -nobin-win.zip archive whose text files have Windows-stile line breaks.
To wrap the archives, change to the dev/DistributionUpdate/dist45/ directory of the stable branch of the GAP git repository, check that you have properly set up version numbers in setvar* file(s) as explained in 2.2-2 and then call one of the following:
make minor make major make update
dependently on the type of the release that you are going to wrap.
We will now explain in more details which steps will be performed. If not specified, all files are assumed to be from the same dev/DistributionUpdate/dist45/ directory.
First, after calling make with the appropriate target, the corresponding setvar* file will be copied into the setvar file (which may be removed with make clean if needed). Then the script ./doit will be called. This script contains only calls to other scripts, arranged in three stages with clean interfaces between them (so, theoretically, each step may be performed on a different machine):
checkout and archive the release branch;
make necessary preparations in the GAP core;
merge the GAP core with packages.
We will now describe in some more details what each of the called scripts is doing (you don't normally need to call them directly except while debugging; calling them out of order is not guaranteed to work!).
./setvarsets various environment variables (paths, version numbers, etc.).
./checkoutgitcreates a new clone of the GAP git repository. Dependently on the release (minor, major or update) it will set the working directory to the stable branch, master branch or to the revision taggest as the last minor release respectively. It will also record the revision on which the working directory is based and the time of cloning which will be subsequently used in the timestamp.
./classifyfilesclassifies files into text and binary files to be included in the main archive or in the tools archive, or not shipped at all, using the Python script classifyfiles.py and files patternscolor.txt and patternstextbinary.txt which specify classification rules.
./zipreleasebranchwraps files from the core GAP system that are marked for the redistribution into the release branch archive and the tools archive
./zipmetainfowraps the meta-information archive, containing the .servar file, details about the checkout time and various lists of files to be included. This script completes stage 1.
./unpackreleasebranchunpacks all three archives produced at two preceding steps: the release branch archive, the tools archive and the meta-information archive (at this step, all files and subdirectories that are not redistributed are already excluded).
./updateversioninfoinserts version number, release date and other depending information (so these adjustments will never appear in the repository; instead, after the release we will find the parent revision used to wrap the release and will tag it with an appropriate tag).
Version numbers must be adjusted in the following files:
configure.ac, look at the top of the file
./fixpermissionsgives directories 755-permissions and files 644-permissions, and also gives 755-permissions to a few explicitly listed there executable files (so it must be adjusted if new executable files are added to the GAP distribution).
./zipgapcoreproduces the etc/tools.tar.gz archive, also adding to it the gapmacrodoc.pdf document, and then prepares the archive of the GAP core system.
./updatemetainfoupdates the meta-information archive, containing the .servar file, details about the checkout time and various lists of files to be included. This script completes stage 2.
./unpackgapcoreunpacks the GAP core system archive and the meta-information archive produced by previous two steps (at this step, the core GAP system is exactly in the same state as will be included in the distribution, except hat manuals are yet to be built).
./unpackpackagesfinds in the location specified in MERGEDPKGLOC and unpacks the latest merged packages archive and its corresponding metainformation archive (with, in particular, list of text/binary files).
./checkpermissionschecks for potentially problematic file permissions, i.e.:
directories not having 755 permissions;
files not having 644 or 755 permissions;
files with 755 permissions, not recognized as executables by file.
These situations may occur in some packages, since we adjust all permissions in the core GAP system, but we do not modify permissions chosen by package authors in case they deliberately use non-standard settings.
./makedocbuilds the main GAP manuals (the packages archives were unpacked on the previous step to ensure that cross-references to package manuals will be resolved).
./addmanualfilesadds lists of text and binary GAP documentation files to the lists of files to be included in the distribution (thus, despite the previous step builds GAP binaries needed to run GAP to build its manuals, our relying on classifyfiles.py and explicit control what has to be included ensures that nothing produced by that build will appear in the release, except of the manuals themselves).
./zipgapsourcedistroproduces final GAP source distribution archives in all four formats: .tar.gz, .tar.bz2, .zip archives and -nobin-win.zip with Windows-style line breaks.
./finalisemetainfoproduces the final version of the meta-information archive.
Now archives with GAP source are ready for testing on as many different systems as possible. Recall the test suites described in Chapter 6.
To produce the -win.zip archive with binaries for GAP and some packages, you need the -nobin-win.zip archive with Windows-style line breaks from the GAP source distribution produced as described above, and a Windows computer with Cygwin (https://www.cygwin.com/) installed.
Unpack the archive, start Cygwin shell, change to the GAP root directory created after unpacking and enter
./configure make cygwin
This will build GAP executable for Windows. After that, you may also build packages with kernel modules which seem to work under Windows. As in GAP 4.6, these are the following: Browse, cvec, EDIM, IO and orb. Each of them supports the standard ./configure ; make building procedure.
Note that there is a nasty problem with respect to dynamically linked libraries used by cygwin under Windows: There seems to be no working version bookkeeping with DLLs. That is, GAP might not be working on Windows systems where another version than the version against which the GAP executable was linked is installed (at least as long as a process is running that uses the other version of cygwin1.dll). However, we do not see a clean solution to this problem.
After all archives are wrapped and checked and no more errors are found, one has to do the following things to properly publish the release:
Install the brand new version yourself using one of the produced GAP source distribution archives (one needs to have this during the next few steps). The archive will already have documentation created and will contain latest released versions of all currently redistributed packages. DO NOT USE any versions of packages from other sources!
Copy .tar.bz2 .tar.gz, .zip archives, -win.zip and -noreadline-win.zip archives (with Windows binaries), and also .exe Windows installer to the ftp server, distributed into directories by the archive type. Also, place the archive of the core GAP system without packages into ftp://ftp.gap-system.org/pub/gap/gap4core/ for developers.
From this point, the release is considered to be published, since the archives are now publicly available.
Prepare individual archives for the same versions of GAP packages that were included in the release and copy them to the ftp server. The procedure should be explained in more details in Chapter 8 (while this chapter is empty, see the DistributionUpdate/PackageUpdate directory from the https://github.com/gap-system/gap-distribution repository for more details). In brief, you should change to the DistributionUpdate/PackageUpdate directory of the mentioned repository and do the following:
mark those versions of packages which are included in the release as "stable" using ./markStableRevisions or ./markAllLatestStable scripts
mark all stable versions of packages with the bookmark denoting the release, e.g.
./markAllStableWithTimestamp gap4r5p6_2012_11_04-18_46
create individual archives for stable versions of GAP packages with ./mergePackages all stable
update package manuals for the website with ./updatePackageDocs
generate Mixer code for packages section of the website with ./writePackageWebInfos
copy package archives to the ftp server with ./CopyToFtpServer (this script will perform a dry run first, so you will be able to check if the list is correct before copying)
update the local clone of the GAP website repository to the latest revision, then copy there package manuals and autogenerated Mixer code using ./CopyToWWW2 (this script will also perform a dry run first, so you will be able to check if everything is right before copying)
change to the local clone of the GAP website repository. Copy manuals of updated packages to the public website. Revise, commit and push changes in the files kept under the version control. Some files for packages which are just added for the redistribution may have to be placed under version control.
Since now the autogenerated data are pushed into the central repository, mosts of the website editing steps listed below may be done on another machine different from the one used for checking for package updates and wrapping package archives.
Using some clone of the GAP website repository, do further editing as necessary:
In case of a minor or major release, read the file dev/LinksOfAllHelpSections.g from the stable branch of the GAP git repository using your brand new installation of the GAP version to be released (remember to start it with -r option to avoid loading your private packages). This will create the file AllLinksOfAllHelpSections.data which is needed to create links from the webpages into the manual. Now update lib/AllLinksOfAllHelpSections.data with this new version.
Change lib/config to update version numbers, release date and time, etc.
For a minor or major release of GAP 4.X.Y, add the file Releases/4.X.Y.mixer under the version control. You may use a similar file for the previous release as a template. Links and names of the archives (version number) and file sizes have to be adjusted manually. For the specification of all packages included in the distribition, see the file Packages/pkgstaticlist. Update the symlink Releases/index.html to point to the new Releases/4.X.Y.html file (you may need to run Mixer to create it first). Add a new entry to the Releases/tree file.
For an update release, you need only to continue the existing page for the latest release. You may also use previous section on that page as a template, and find detauls about packages in the file Packages/pkgstaticlist. You do not need to list all changed packages in this case: listing only new and updated packages will suffice.
If there is a new package added for the redistribution, you should edit Packages/packages.mixer and Packages/tree to add links to its overview page from the list of packages and from the left navigation panel.
Change Download/index.mixer and Download/upgrade.mixer to update version numbers, the GAP banner etc.
For a major release only, change WWW2/Download/Updates/index.mixer: add new entry for the new release with a link to the Changes manual.
Update package installation scripts Download/InstPackages.sh and Download/InstPackages32.sh, if there were any changes.
Build a local copy of the website using Mixer, check that it works, then commit and push all changes to the central repository. If need be (e.g. for major releases with more structural changes), you may use the testing website to check it there first. See 9 for further instructions.
Copy the complete doc directory (of your brand new installation) to the public website Manuals/doc so that manual links will work.
In the GAP repository, tag what has been released with v4.X.Y where X is the major release number and Y is the minor release number (e.g. v4.6.3). The tag command has the format
git tag TAGNAME REVISION
and the revision that should be tagged should be determined from the release wrapping log. Normally this should be the tip of the stable branch at the moment of wrapping the release (not at the moment of tagging!).
Update dev/Updates directory:
Create dev/Update4.X.Y directory and add it to the version control.
Move all files with documented changes from dev/Updates directory to dev/Update4.X.Y directory using git mv command, so that the only files left in dev/Updates directory should be README and two templates (important to do this in the stable branch, since other branches may have more documented changes).
Commit and push changes.
Edit variables in dev/DistributionUpdate/dist45/setvar* files, setting the numbers as appropriate for wrapping the next release candidate.
Test download links and proofread web pages.
Announce the new version in the forum (for the major release, it is preferable to wait with the announcement until it will be picked up by all alternative distributions as well).
generated by GAPDoc2HTML