metomi/fcm

:hammer: FCM: a modern Fortran build system + wrappers to Subversion for scientific software development

fcm's Issues

Use pertidy to beautify code

We should use pertidy to beautify the style of our Perl source code.

fcm make: build: source to target diagnostics

It would be desirable to print out the available targets for each source.

fcm make: build: separate handling of C++ files from C files

Although they are pretty much the same, C and C++ source files often need to be compiled with different options. They should be treated differently from the user's perspective (but implementation can otherwise share the same logic).

fcm make: build: support wildcards in property namespaces

Wildcard syntax such as:

build.prop{fc.flags}[*/foo/bar] = --some-option
build.prop{fc.flags}[foo/bar/a*] = --some-other-options

can be useful for declaring multiple compiler options for a selection of name spaces.

fcm make: global property for a step class

It is worth supporting a syntax to define global properties for step class. E.g. if we are to test some compiler options in 2 builds of the same source tree, it would be desirable for the builds to share most options, and only overrides the global setting if required.

Perhaps a syntax like this:

build.prop{class,fc.flags} = -O3

fcm make test suite improvement

It was agreed that we should review the approach of the test suite for fcm build, fcm extract and fcm make. It would be nice if we could move the test suite into a form closer to what we have done for rose test-battery, and make it generic for different sites.

fcm make: build: build.prop{name}[target]

It should be possible to use the name of the target (as well as its namespace) for a build.prop that applies to a target. E.g. It may be easier to do build.prop{fc.flags}[hello.o]=-O2 instead of build.prop{fc.flags}[hello/world/hello.f90]=-O2.

fcm make: extract: from remote host

Extract from source trees on a remote host.

fcm merge: consider unwrapping svn merge

The wrapping of svn merge in fcm merge was introduced because of the lack of proper merge tracking in Subversion at the time. Subversion's merge handling has since become much better (mergeinfo, reintegrate handling), and we should consider whether we can reduce the fcm merge wrapping or remove it entirely.

fcm make: extract: from a local git clone

Extract from a local git clone. Something like this:

extract.location{primary}[foo]=$HOME/foo.git
extract.location[foo]=v10.0
extract.location{diff}[foo]=v10.0-149-g0a0f863 v10.0-172-ga81bf81

(The revisions should be anything acceptable by the git archive command. A . would denote the working tree. The extract.location should default to .?)

See also #13.

New command to print default+site+user configuration

We need a new command to print the default+site+user configuration.

fcm make: report FCM version in log

fcm make should report FCM version in log.

fcm make: combine SHELL and OUT events

When a build task triggers a shell command, the shell utility generates a SHELL event when the command returns, and the task generates a separate OUT event to print the standard output and error. In a multi-process environment, these event may not be handled back-to-back and may get intermingled with other events. It would be desirable if the 2 events can be optionally combined.

fcm make: build: prop{dep.o} limitation

Using it with the root namespace results in a cyclic dependency. Presumably this shouldn't happen since it is only meant to define a link-time object dependency (using ns-dep.o in a similar way doesn't suffer this problem).

fcm make: .prop{dep.o} target keys as name-spaces

#65 introduced target keys as name-spaces for properties such as compiler options - e.g.

build.prop{fc}[world.o] = my-fc

This does not work for dependency properties such as dep.o - if it did, it would be a nice feature to have. For example, it would be good to be able to write:

build.prop{dep.o}[hello.exe]=world.o greet.o

instead of the current:

build.prop{dep.o}[src/main/hello.f90]=world.o greet.o

Currently, writing this in a configuration file will be accepted (it is a valid name-space), but it will not do anything.

Support more diff/merge tools

FCM uses xxdiff for interactive diff and merge. Perhaps we should move to something like vimdiff, meld or diffuse?

Subversion 1.8

Test and modify FCM to support Subversion 1.8.

fcm make: build: compile order key

Introduce a compile order key setting in the configuration to allow users to specify which source file will take a long time to compile. Ready tasks with order keys will be ranked by sorting their order keys in alphabetical order. Tasks without an order key will come after those with order keys. The dependency manager will then always attempt to emit the highest ranking task to the job runner.

fcm make: build: props for library names and library / include / module paths

There are options to define the flags to use for an include search path (fc.flag-include), a library (fc.flag-lib), a library search path (fc.flag-lib-path) and a module search path (fc.flag-module). However, there is no way to define, for instance, a list of paths to be added to the include search path. At the moment you have to add the options to the fc.flags or fc.flags-ld yourself. Allowing them to be defined separately would make those settings more compiler independent.

Subtree mergeinfo generation

The current Subversion version (1.6) can generate the svn:mergeinfo property on a file or directory under certain benign conditions, such as an svn copy of a file from another branch. This example is desirable behaviour for some tree conflict situations.

This property is useless from an internal FCM standpoint and confusing for the user, as it is updated for each file or directory at every merge and appears in the status output, hiding genuine changes.

Should we

• automatically remove subtree mergeinfo?
• alter the tree conflict handler to automatically remove if it generates it?
• wrap svn copy?

fcm make: friendlier log file location?

Move .fcm-make/log to fcm-make.log?

fcm make: build: option to keep intermediate build library

An option to keep the intermediate build library would be useful.

fcm make: trap signal to ensure lock directory is removed

The lock directory is left behind if fcm make is killed by a signal. Signal trapping in OO Perl was not straightforward in the past, so it was not done. We should look again to see if there are good ways to do this.

fcm make: log file is large

The .fcm-make/log file generated by fcm make can be quite large. It may be friendlier to users if it is separated into a main file and an individual file for each step.

hook: log for each run

In the current system, we only keep the log for the latest run of a repository hook script. This can make it quite difficult to diagnose problems when the corresponding log has been overwritten.

The log file for each run of each repository hook script should be kept individually for a given period of time.

fcm make: FCM::Util::TaskRunner: warn if task runs for a long time

FCM::Util::TaskRunner: should probably warn if a task is running for a long time.

Clash between Fcm::* and FCM::*

We have a namespace issue between Fcm::* and FCM::* modules While this is OK in POSIX compliant systems, the clash can cause problems in case insensitive file systems. To avoid this, we should rename the Fcm::* namespace.

See also #24.

fcm make: build: separate compiler and linker

There is a now use case for separating the compiler and linker commands for a given language.

fcm make: if one step fails, continue to do next step

Allow fcm make to continue with the next step if a step fails. (Obviously, the next step will only work if it is independent of the failed step.)

fcm tag-create

Some requirements:

• It should fail if you try to use an existing tag name
• You should be able to create a tag from:
  • A particular version of the trunk
  • A particular version of a branch?
  • The current contents of your working copy (which may contain mixed revisions)
• We need to define some sort of naming convention

fcm swicth/update: dry-run or confirmation

A careless switch or update can break a working copy. This is known to have caused a lot of headaches amongst inexperienced users, who might become reluctant to use these very useful commands.

One solution is to introduce an extra confirmation stage, like in fcm merge, where we run svn merge --dry-run to list all the changes before asking the user for a confirmation. Although svn switch and svn update do not accept a --dry-run option, we should be able to mimic the behaviour using svn diff --summarize.

In addition, the result of svn diff --summarize can be compared with svn status, which can be used to warn the user for potential merge/conflict.

documentation: auto generation of HTML from POD

The user guide command reference and the CLI PODs are repeating one another and may become out of sync. Ideally, we should generate HTML from POD.

We should also generate HTML from POD in source files in lib/.

svn patch: investigate

We should look into using svn patch for fcm mkpatch and similar offline usage.

fcm merge: improve support for merging branches of branches

Consider if you have a branch of a branch. If you try to merge this branch into an unrelated branch then the merge command fails due to no common ancestor. To get the merge to work you have to specify

--custom --revision X:Y

We can't get away from this being a custom merge since we can't support 2-way merging in this scenario. However, this is a common use case so perhaps the default behaviour of --custom should be to assume you want to use the base of the branch and give you a choice of revisions (as you would get with an automatic merge).

There is also nothing in the Code Management System section of the User Guide about branches of branches so perhaps we should add something at the same time.

fcm branch create: alternate interfaces

The command line interface of branch create has confused some users as it uses the project URL as an argument - as well as taking many different options. Alternate interfaces are recommended:

# Current interface
fcm branch-create [OPTIONS] NAME [PROJECT-URL]
# New interface 1
fcm branch-create [OPTIONS] BRANCH-URL [...] [PARENT-URL]
# New interface 2
fcm branch-create [OPTIONS] TICKET-NUM [...] [PARENT-URL]

In the 1st new interface, the argument list will be a list of URL of the branches to create. Each of these must conform to the FCM branch naming convention. The final argument will be the parent URL. (This argument is optional if the current working directory is a working copy to the parent URL.)

In the 2nd new interface, the argument list will be a list of ticket numbers. This is similar to the current usage, except that the final argument will be the parent URL, and the options --rev-flag=NONE will be automatic, i.e. the resulting branch path will look like branches/dev/fred/1234.

fcm make: build: deep dependency tree performance

It may be worth seeing if there is any way to cache the build state tree for incremental builds.

Tutorial improvement

We should make the tutorial separate from the User Guide. This will make the printed User Guide much shorter. It will also allow us to consider alternative ways of presenting the tutorial. A single HTML page is far from ideal.

• Perhaps consider a similar approach as Rose?
• Add topic on how to do interactive conflict resolution?

FCM: to require Perl 5.10 or above

We should make FCM requires Perl 5.10 or above, and withdraw support for running FCM under Perl 5.8.

Perl 5.10 has many new features that can be useful to us. Some of them are listed here:

• New defined-or operator //.
• Regular expression enhancement:
  • Named capture.
  • Smart operator ~~.
• New module:
  • Compress::Zlib, IO::Zlib, Archive::Tar, Archive::Extract:
    we can use one of these instead of gzip.
  • Module::Load, Module::Pluggable, Module::Load::Conditional:
    we can use one of these to load modules at run time.
  • Term::UI: use this to prompt for user inputs.
  • IPC::Cmd: use this to run shell commands.

See: http://perldoc.perl.org/perl5100delta.html.

fcm make: build: inheritance limitation: handling of include files

If a compiler or pre-processor searches for include files from the container directory of a source file before it does the paths specified by the -I option, it may wrongly use the include file in an inherited build instead of the current one.

The detail of the problem is documented in the User Guide and referenced in the release notes. Simple test code is provided in the release to help check whether the problem exists with a particular compiler.

@dpmatthews has done a bit of work to investigate this further a few years back.

For Fortran includes:

• ifort does not work correctly. However ifort -assume nosource_include works fine.
• gfortran works fine.
• xlf95 on the IBM HPC works fine.

For CPP includes:

• Using the Fortran compiler to do the pre-processing does not work correctly with any of the compilers.
• cpp does not work correctly.
• Using cpp taking its input from stdin works fine. This is potentially an easy solution but can we assume that the CPP tool will work in this way?

Remove deprecated features

• The fcm branch command.
• The --branch option to fcm diff.
• Support for $HOME/.fcm and $HOME/.met-um/ for user configuration.

fcm make: append to a configuration declaration

It would be nice if we can develop a syntax to allow a configuration declaration to be appended.

fcm branch naming should use subversion account name

fcm bc uses your local userid for the branch name. This does not make sense if you are using a repository where you have a different account name.

fcm make build: incremental target removal

In an incremental extract, the system compares the list of targets with that of the previous extract and removes any items that no longer exist in the current list. An fcm make incremental build should be able to make a similar analysis. (An fcm build incremental build does not do this because it is not able to retrieve the list of targets in the previous build.)

fcm CM: get working copy root should use new svn info property

When using Subversion 1.7.0+, we should use the svn info working copy root information introduced in Subversion 1.7.0 to find the top of the working copy.

fcm make: build: handle dependency changes caused by OMP sentinels

If OMP becomes more widespread in our Fortran code, the build system will need to support dependency changes caused by OMP sentinels. E.g.:

!$ use a_module_useful_only_in_omp

In order to support this there would need to be a flag to tell the build system that OMP is in use.

Do we really need this? If the extra dependencies are to code provided by external libraries then we don't need to know this. Could references to local code be protected by CPP directives instead?

fcm make: build: shared objects

Enhance the build system to support the making of shared objects (i.e. *.so).

fcm make: keep old log in incremental mode

In incremental mode, the .fcm-make/log file of the previous run should be rotated.

Fcm::Cm: migrate to FCM::System::CM

The majority of the Fcm::Cm code was left untouched when we introduced a new framework for FCM system logic. It would be desirable to migrate the code to the new framework under FCM::System::CM.

fcm mkpatch: Incorrect #! line in 'apply_patch'

The 'apply_patch' script that FCM generates asks for the Bourne shell in its shebang line, however the script itself uses features not available in Bourne, e.g. '[[' for tests.

This results in errors along the lines of

UMUI_vn8.4-vn8.5_patch/57689/apply-patch: 27: UMUI_vn8.4-vn8.5_patch/57689/apply-patch: [[: not found

Can this be fixed, either by changing the shebang line to '/bin/bash' or using sh syntax for the tests?

fcm make: build: Fortran 2K support

The following was received via email following an analysis by a colleague.

The following F2K features need support by the build system of fcm make:

1. Dummy procedure arguments:

This sort of dummy argument seems to be fine:

external :: foo
integer, external :: bar

In Fortran 2003 you can also use a procedure declaration:

subroutine sub(a)
procedure(foo), pointer :: a

where foo is the interface, or

procedure(), pointer :: a

where there is no interface. If the foo is coming from a module the above can be made to work by simply adding procedure to the list of declarations in FCM::System::Make::Build::Task::ExtractInterface line 36. However, it is also possible (and may be preferable in some cases) to declare the interface explicitly in the routine, i.e.:

subroutine sub(a)
interface
  subroutine foo
  end subroutine foo
end interface
procedure(foo), pointer :: a
! ...

The ability to give an interface to a procedure argument is not new to F2003, in F90 you can do

subroutine sub(a)
interface
  subroutine a
  end subroutine a
end interface
! ...

FCM interface generation cannot handle this sort of interface at the moment and I don't know how difficult it would be to support it. One could of course simply use module subroutines instead in which case all is well but it would be good if it was documented in the FCM user guide that this sort of interface declarations can only be used in module procedures.

To further confuse issues you can also do

abstract interface
! ...
end interface

instead of

interface
! ...
end interface

so ideally FCM would handle interface and abstract interface the same. The difference between abstract and non-abstract interfaces are not relevant to procedure dummy arguments so either could be used.

2. Class declarations:

These are very similar to TYPE declarations, i.e. instead of

subroutine foo(a)
type(bar) :: a

you have

subroutine foo(a)
class(bar) :: a

Again this can be supported by appending class to line 36 of FCM::System::Make::Build::Task::ExtractInterface.

3. Use intrinsic:

In addition to USE foo[, only: ...] one can do:

use, intrinsic :: foo [, only: ...]

The rules as I understand them are:

1. With intrinsic not specified a user defined module is used if found, otherwise the intrinsic module is used.
2. With intrinsic specified only intrinsic modules are used, compilation fails if there is no intrinsic module

This is similar to external and intrinsic for procedures although the search order is reversed in the default case which is a bit confusing. FCM should support intrinsic and the coding standards should recommend that for intrinsic modules use, intrinsic is used in preference to just use. When use is used on its own FCM should base its dependencies on the hard-coded list as it does now but it should never generate a dependency for "use, intrinsic" as in principle compilers could add their own intrinsic modules (in the same way that they can add their own intrinsinc procedures) and we may want to use those modules for whatever reason. Note that e.g.

subroutine foo(a)
use, intrinsic :: iso_c_binding
integer(c_int) :: a

still needs to include the reference to iso_c_binding in the interface block even though there is no dependency.

4. iso_fortran_env

The intrinsic module iso_fortran_env should be added to line 39 of FCM::System::Make::Build::FileType::Fortran.

Briefly, for Fortran 2008 there will be the following issues for FCM

5. Double Complex

I noticed that this appears to be missing from line 36 of FCM::System::Make::Build::Task::ExtractInterface. It is actually optional in F90, so technically it counts as an F2K change. I don't know of any code that uses it but it should probably be included for completeness.

1. Submodules

module foo
! ...
end module foo

submodule (foo) foo_bar
! ...

The idea is that foo contains interface blocks and foo_bar contains implementations although no doubt people will find other uses in future. The dependency generator would need to ensure that foo_bar depends on foo.

2. impure elemental

In F2008 there is an impure elemental procedure, sort of like elemental but less so (sorry, I haven't checked the exact details), i.e.

impure elemental subroutine foo(a)
! ...

I presume this would mean updating line 53 of FCM::System::Make::Build::FileType::Fortran.

I'm not sure what else is relevant to interface generation. In FCM::System::Make::Build::Task::ExtractInterface there is

my %TYPE_DECL_KEYWORD_SET = map { ($_, 1) } qw{
    allocatable
    dimension
    in
    inout
    intent
    kind
    len
    optional
    out
    parameter
    pointer
    save
    target
};

which appears to be a list of attributes that dummy arguments may have, but I tried declaring dummy arguments with things not on the list:

volatile
asynchronous
external
value
contiguous

and the interface block seemed to include them. It appears that FCM just copies the entire line (which is the correct thing to do). Similarly for function or subroutine names the bind(c) is included correctly even though bind isn't mentioned anywhere in the Perl code.