Getting Robot Framework Results in the Email from Jenkins

The report and log pages generated by Robot Framework are great, but it is also nice to have an executive summary of the test results. If you use the Robot Framework Plugin and Email-ext plugin for Jenkins, you can get results in the email body that look like this:

Robot Framework Results Detailed Report Pass Percentage: 80.0% Test Name Status Execution Datetime Customer Customer.Ordering Place Credit Card Order PASS Mon Jul 30 16:17:25 CDT 2012 Save to Wishlist PASS Mon Jul 30 16:19:42 CDT 2012 Place Paypal Order PASS Mon Jul 30 16:29:01 CDT 2012 Customer.History View Last Order PASS Mon Jul 30 16:32:00 CDT 2012 Shipping Status FAIL Mon Jul 30 16:38:02 CDT 2012

Here is a snippet from the Groovy template that generated the above:

 <%  
 import java.text.DateFormat  
 import java.text.SimpleDateFormat  
 %>  
 <!-- Robot Framework Results -->  
 <%  
 def robotResults = false  
 def actions = build.actions // List<hudson.model.Action>  
 actions.each() { action ->  
  if( action.class.simpleName.equals("RobotBuildAction") ) { // hudson.plugins.robot.RobotBuildAction  
   robotResults = true %>  
 <p><h4>Robot Framework Results</h4></p>  
 <p><a href="${rooturl}${build.url}robot/report/report.html">Detailed Report</a></p>  
 <p>Pass Percentage: <%= action.overallPassPercentage %>%</p>  
 <table cellspacing="0" cellpadding="4" border="1" align="center">  
 <thead>  
 <tr bgcolor="#F3F3F3">  
  <td><b>Test Name</b></td>  
  <td><b>Status</b></td>  
  <td><b>Execution Datetime</b></td>  
 </tr>  
 </thead>  
 <tbody>  
 <% def suites = action.result.allSuites  
   suites.each() { suite ->   
    def currSuite = suite  
    def suiteName = currSuite.displayName  
    // ignore top 2 elements in the structure as they are placeholders  
    while (currSuite.parent != null && currSuite.parent.parent != null) {  
     currSuite = currSuite.parent  
     suiteName = currSuite.displayName + "." + suiteName  
    } %>  
 <tr><td colspan="3"><b><%= suiteName %></b></td></tr>  
 <%  DateFormat format = new SimpleDateFormat("yyyyMMdd HH:mm:ss.SS")
    def execDateTcPairs = []
    suite.caseResults.each() { tc ->  
      Date execDate = format.parse(tc.starttime)
      execDateTcPairs << [execDate, tc]
    }
    // primary sort execDate, secondary displayName
    execDateTcPairs = execDateTcPairs.sort{ a,b -> a[1].displayName <=> b[1].displayName }
    execDateTcPairs = execDateTcPairs.sort{ a,b -> a[0] <=> b[0] }
    execDateTcPairs.each() {
      def execDate = it[0]
      def tc = it[1]  %>
 <tr>  
  <td><%= tc.displayName %></td>  
  <td style="color: <%= tc.isPassed() ? "#66CC00" : "#FF3333" %>"><%= tc.isPassed() ? "PASS" : "FAIL" %></td>  
  <td><%= execDate %></td>  
 </tr>  
 <%  } // tests  
   } // suites %>  
 </tbody></table><%  
  } // robot results  
 }  
 if (!robotResults) { %>  
 <p>No Robot Framework test results found.</p>  
 <%  
 } %>  

The date conversion is there to convert the date from the format Robot uses to the Java default format, which is more familiar to us.

This is the first thing I have ever done in Groovy, and I must say it is a pleasant language to write in. I especially like how getters and setters are mapped to the Groovy concept of properties.

Happy roboting.

Multiple Instances of a Remote Test Library in Robot Framework

One big shortcoming with the remote library API is that is has no mechanism for distinguishing between different instances of Robot Framework running either locally or remotely (i.e. it is a session-less protocol). If the library has any kind of state information, then two callers cannot reliably share the same instance of the library. This is clearly a problem when two tests are executing concurrently that both use the same library.

I have devised a simple solution that applies to execution by Jenkins if there is only one instance of Robot Framework running per executor. Jenkins defines many environmental variables that can be useful, including EXECUTOR_NUMBER, which is number of the executor running the current job. We can use this as an offset to a base port to be assured that each job will have its own instance of the remote library. For example, using a base port of 4000, the Robot Framework instance in executor #3 would use port 4003. The library must be parameterized so that the port can be chosen on startup. Here is some code showing how to start a library on the correct port prior to running Robot Framework in a Windows environment.

 set MY_REMOTE_LIBRARY_PORT=4000  
 if defined EXECUTOR_NUMBER set /a MY_REMOTE_LIBRARY_PORT=%MY_REMOTE_LIBRARY_PORT% + %EXECUTOR_NUMBER%  
 start java -jar my-remote-library.jar %MY_REMOTE_LIBRARY_PORT%  

When the library is imported, the appropriate port number must be used. You could set the variable as a command line argument to or use a variables file as shown below.

 import os  
   
 def get_variables():  
   port_offset = 0 if not 'EXECUTOR_NUMBER' in os.environ else int(os.environ['EXECUTOR_NUMBER'])  
   return {'MY REMOTE LIBRARY PORT': str(4000 + port_offset)}  

Note that if running outside of Jenkins, the base port is used.
The resource file would look something like this:

*** Settings ***
Variables         remote_library_port.py
Library           Remote    http://127.0.0.1:${MY REMOTE LIBRARY PORT}/

If you run Robot Framework instances in parallel in the same executor, then some other mechanism for assigning port numbers would be required. I cannot think of a simple, robust way to do this and would love to hear if you do.

Let Jenkins visitors know their access is restricted

We use Jenkins in the QA department and expose it to other teams and departments. For new users it is easy to be confused as to why they cannot do certain things, such as start a build, because Jenkins simply hides icons that a visitor does not have access to. In this article I explain how to get a header on every page Jenkins serves that lets them know their access is restricted:



Follow these simple steps:

1. Install the Page Markup Plugin
2. Go to Manage Jenkins > Configure System
3. For Header HTML in the Additional Page HTML Markup section, put:
   

    <div id="my-banner" align="center" style="display:none;">  
    <p /><img src="/userContent/error.png" /><b>You are not logged in. To start a job or make changes you must log in. If you do not have an account, you can <a href="/signup">sign up</a> for one.</b>  
    </div>  
   

   Note that the image referenced here does not come with Jenkins and needs to be put in the userContent folder.
4. For Footer HTML in the Additional Page HTML Markup section, put:
   

    <script type="text/javascript">  
    function loggedIn() {  
        var x;  
        x = document.getElementById('login-field')  
        if(x.innerText) {  
            x = x.innerText;  
        }else{  
            x = x.textContent;  
        }  
        return (x.indexOf('log out') > 0);  
    }  
    if (!loggedIn()) {  
        document.getElementById('my-banner').style.display = 'block';   
    }  
    </script>  
   

5. Hit Save

The header will only display if a visitor is not signed in and Javascript is enabled.