If you're using SpecFlow and WatiN to automate your web testing, things work great on localhost, but tend to fall apart on the continuous integration server. It's a "chicken or the egg" paradox. If the new code hasn't been deployed yet, it can't be externally tested with WatiN, but it shouldn't be deployed until all the tests have passed. What are you supposed to do?
TeamCity
If your SpecFlow and WatiN tests are in their own assembly (something like MyProject.AcceptanceTests.dll), and you're using TeamCity, it's easy to run your unit tests and integration tests, then deploy the code to a web server, then run your acceptance tests in a separate build step. It's just a matter of breaking up the steps and designating which test assemblies will be run at which time.
This step in the screen shot below is running the unit tests and integration tests. A separate, later step will run the tests in the **\*AcceptanceTests.dll assembly after the code has been deployed to the web server.
AppHarbor
If you're using AppHarbor to run your tests on check in with git push appharbor master, (and you should - it's super easy), you don't want to run your web tests until the code is deployed, but AppHarbor doesn't deploy anything until all the tests pass.
What we want is for AppHarbor to ignore the web automation tests until the code is deployed. AppHarbor flips a *.config appSetting key named "Environment" to "Test" when it's testing and to "Release" when the code is deployed. So while your tests are running, your App.config in your test project should look something like this:
<appSettings>
<add key="Environment" value="Test"/>
</appSettings>
We can take advantage of this appSetting test runner change to tell SpecFlow to ignore this set of tests, which allows AppHarbor to continue running other tests and deploying the code.
This is not be the most elegant way to do this, but I got this working today:
[Binding]
public class MyStepDefinition
{
[BeforeFeature]
public void BeforeFeature()
{
if (ConfigurationManager.AppSettings["Environment"] == "Test")
Assert.Ignore();
}
// Rest of step definition code...
}
The BeforeFeature attribute tells the SpecFlow test runner to execute that method before each feature. With a quick check on the Environment appSetting value, we can tell AppHarbor to ignore this test and continue processing. This will ignore all scenarios in all feature files. Since all my scenarios are web automation tests, that works for me.
But what if you have some scenarios that rely on web automation testing and others that don't? A more targeted approach would be to set a SpecFlow tag, like @web over the SpecFlow scenarios that require web automation testing to pass. Then in the [BeforeScenario("web")] handlers for that tag, you could set Assert.Ignore to skip those scenarios while executing all other scenarios.
[BeforeScenario("web")]
public void BeforeScenario()
{
if (ConfigurationManager.AppSettings["Environment"] == "Test")
Assert.Ignore();
}
Once your non-web automation tests have passed and the code is deployed, you can point your local test runner at the AppHarbor URL and test against the newly deployed web app.
