Filter Tests with xUnit and TeamCity

As part of our migration to TeamCity and our Continuous Integration efforts we've integrated our xUnit.net tests into the build and test process. One of the primary limitations I've found with xUnit is the inability to batch tests outside of the IDE. During development it's easy to run only a specific set of tests ala R#. However as part of the build process don't have that luxury.

There's a couple reasons that I'd like to run tests in batches.

• So I can make sure the longest running tests are performed last after all the other tests have completed successfully.
• So I can perform different tests at different stages of the build process. For example running basic functionality tests immediately after compile, and then more complex tests after the entire product has been built and packaged.

Brad Wilson and the other guys in the xUnit project have really done a fantastic job laying a foundation for extensibility. I was able to easily create my own custom attribute to establish filters for different tests. The Group attribute derives from RunWithAttribute so that we can tell xUnit the TestClassCommand to use when testing the class. This lets me take over how the test is performed -- including skipping it.

public class GroupAttribute : 
// Have to use a switch cause Resharper does not detect the tests when an 
// attribute derived from RunWithAttribute is applied to the class. 
#if WITHGROUPS
RunWithAttribute    
#else
Attribute
#endif
{
    public GroupAttribute() : this( "*" ){}
    public GroupAttribute( string group ) 
#if WITHGROUPS
    : base( typeof(TestGroupClassCommand) )
#endif
    {
        if (group == null || group.Length == 0) throw new ArgumentNullException("group");
        Group = group;
    }
    public string Group{ get; private set; }
}

The TestGroupClassCommand looks for a setting in the environment or <AppSettings> section of the .config file to see if the tests should be filtered by a group. The EnumerateTestMethods is where the real magic happens.

public IEnumerable  EnumerateTestMethods()   
{   
    string testGroup = Environment.GetEnvironmentVariable( "TestGroup" );   
    if( testGroup == null || testGroup.Length == 0 )   
        testGroup = ConfigurationManager.AppSettings[ "TestGroup" ];   
    if( testGroup == null || testGroup.Length == 0 )    
        return _cmd.EnumerateTestMethods();  
        
    string[] testGroups = testGroup.Split( _delims, StringSplitOptions.RemoveEmptyEntries );  
    GroupAttribute attr = Attribute.GetCustomAttribute( TypeUnderTest.Type,   typeof( GroupAttribute ), true ) as GroupAttribute;  
    if( attr != null )  
    {  
        if( attr.Group == "*" ) 
            return _cmd.EnumerateTestMethods();  
        
        string[] memberGroups = attr.Group.Split( _delims,   StringSplitOptions.RemoveEmptyEntries );  
        
        foreach( string g in testGroups )  
        {  
            foreach( string mg in memberGroups )  
            {  
                if( String.Compare( g, mg, true ) == 0 )  
                    return _cmd.EnumerateTestMethods();  
                }  
            }  
        }  
        
        return new List();  
    }
}

The convenience of this method is that I can use the .config functionality already built into xUnit to specify the test group or use the Environment Variables of the TeamCity build configuration settings to run only specific tests.