Statistical testing for bias
Statistical testing is a recommended part of ensuring data-driven tools are non-discriminatory. This testing is designed to highlight disparity between groups and therefore concerns about the tool, it is not designed to guide decision making about individual applicants.
The testing works by setting a baseline for your organisation and then testing the tool against this baseline to assure you that the tool is operating as it should, or to alert you of any issues. This testing should be conducted at the pilot stage and monitored throughout the lifecycle of the tool (note that this will require sustainable internal resource).
1. Articulate your organisation’s baseline: before introducing a data-driven tool, you should produce a baseline of progression for applicants from groups with different characteristics. [If there is enough data to draw meaningful conclusions, use intersectional categories, such as “Asian women” or “women above age 50.”] To start this process, you will need to identify the range of characteristics that are relevant in your context (you will likely be drawing from the nine protected characteristics). The information you could consider in developing your baseline includes:
a. The progression of applicants in previous recruitment rounds, where you are not using the tool.
b. If data on previous recruitment rounds is unavailable, consider ways to identify a baseline such as running data collection on a new recruitment round.
c. In addition to your baseline from previous recruitment rounds, you may wish to consider a number of factors which set a higher bar for combating bias in recruitment; this is particularly important if your organisation is not demographically diverse. Factors include:
● The demography of the population you are recruiting from (e.g. the London workforce).
● The industry baseline.
Note: you will need to consider which of these data sets are accessible to you and adjust your baseline accordingly.
2. Measure the impact of the tool in new recruitment rounds: conduct the same analysis (sift, interview, offer) to understand the trends for the same protected characteristics whilst using the tool.
3. Evaluate: compare how different demographic groups progress when the recruitment tool is used in comparison to the position previously. If the analysis shows worrying trends, for example fewer women are now progressing, then this should alert you to a problem with the new tool which requires assessment to ensure that no discrimination is taking place. Alternatively, if the analysis shows parity of outcomes for different demographic groups, this is evidence that the tool is functioning appropriately.