Feds underestimated complexity, ignored concerns about Phoenix pay system, review finds

An independent consultant's report laid out 'lessons that are yet to be learned' more than a year after the problem-plagued pay system was first launched.

Treasury Board President Scott Brison, left, and Public Services and Procurement Minister Carla Qualtrough take questions from reporters on Oct. 5 after they released an independent review of what went wrong with the Phoenix pay system rollout. The Hill Times photograph by Andrew Meade

PUBLISHED :Thursday, Oct. 5, 2017 8:40 PM

An independent report on the problem-plagued Phoenix pay system rollout released Thursday says the government underestimated its complexity.

Public Services and Procurement Minister Carla Qualtrough (Delta, B.C.) apologized to affected public servants in an Oct. 5 press conference, and said the review confirmed many of the thoughts she already had. She noted fixing the pay system is the first priority in her mandate letter from the prime minister, given to her after he moved her into the job in August. 

“The implementation of such a complex business transformation across the entire government of Canada was a massive undertaking that I believe history will record was set up to fail,” she said, noting there was poor planning on almost every key aspect. “There’s no easy or quick fix for the problems in the pay system.”

The report’s “lessons learned” included: assigning accountability and authority to a single office, properly defining the scope of the project, fully testing the software before launch, and not expecting savings until well after implementation.


Overall, the independent consultants hired to do the review found the officials working on the project were most concerned with being on time and on budget. Briefings were usually positive and the culture of the department responsible prevented people from speaking negatively about the project. The report said bad news was usually buried, with concerns mostly ignored. Even at the pay system’s launch in February 2016, testing was incomplete, with no planned fix date and no contingency plan.

Other lessons included making sure to have sufficient employee knowledge capacity, and communicating effectively. The report noted workplace culture was also important, and that the 17 lessons it outlined “are yet to be learned” by the government.

Though work on the new pay system began under the previous Conservative government, the Liberals launched it in February 2016. It was supposed to consolidate the payroll of over 300,000 public service employees, but instead it has left many of them overpaid, underpaid, or not paid at all. Radio-Canada recently reported that as of Aug. 8 nearly half of the 313,734 federal public servants paid through Phoenix had been waiting at least a month to have their complaints dealt with. The government contracted IBM to configure the software for the government payroll system.

The program was supposed to save the government $70-million per year, but so far the Liberals have sunk in about $400-million to fix it.


The review the government commissioned from Ottawa-based Goss Gilroy Inc., cost $165,000. Treasury Board President Scott Brison (Kings-Hants, N.S.) said it was commissioned to identify mistakes so they aren’t repeated.

The 60-page report covers the Transformation of Pay Administration Initiative from 2008 until April 2016. The initiative was defined as two projects: Pay Modernization and Pay Consolidation. Pay Modernization was the replacement of the 40-year-old existing payroll system with a commercial off-the-shelf option—the implementation of Phoenix. Pay Consolidation was the transferring of departmental pay services to one centralized pay centre, located in Miramichi, N.B. This included laying off experienced compensation advisers in Ottawa and hiring a new smaller number of compensation advisers in Miramichi.

The report said it was not meant to lay blame, but provide guidance for current and future projects. The consultants conducted almost 40 individual interviews and eight half-day workshops that involved over 100 people. They interviewed employees, compensation advisers, managers, and executives, as well as union representatives and private sector entities.

Mr. Brison and Ms. Qualtrough said they have already started implementing the lessons, such as by designating a working group of cabinet ministers ultimately responsible for fixing Phoenix. The report found part of the problem was that no one group had the accountability, authority, and capability to carry out the project.


Public Services and Procurement Canada was responsible for the implementation of Phoenix, but the report said it did not appear to have the authority to ensure other government departments did what needed to happen to make the system work.

The Public Service Management Advisory Committee, a group of more than 40 deputy ministers, was the senior coordinating body for the project, but it wasn’t accountable for it.


Unions react swiftly

Donna Lackie, former national president of the Public Service Alliance of Canada’s Government Services Union, said she has not yet had a chance to read the report, but agreed with the lessons The Hill Times read out to her on Oct. 5. The union represents the compensation advisers on the front lines of Phoenix.

She agreed with the report’s recommendation to make sure there are enough workers, expertise, and corporate memory to implement the project. The consultants also noted the importance of an easily implemented contingency plan, which they said it found no evidence of.

“We knew that divesting yourself of your corporate knowledge and your experienced compensation advisers was certainly not the way to go,” Ms. Lackie said.

Debi Daviau, president of the Professional Institute of Public Service of Canada, said the report missed one key concern of her members: she said the government is over-relying on outsourced information technology services.

“The surprise is that it was allowed to go ahead at all and against the warnings of so many, including, prior to its rollout, unions like mine,” she said in a statement.


Government adding more offices to process backlog

The government announced Thursday it’s planning to set up two more temporary pay offices in Kirkland Lake, Ont., and Charlottetown P.E.I., in order to process backlogged Phoenix cases, with Lackie saying another 200 compensation advisers would be hired to help clear the backlog.

The government has already hired over 350 temporary compensation advisers, as well as 30 federal information technology professionals to work out what one union representative said was the system’s 1,000-plus bugs, on top of other IT professionals already working it.

The latest source of backlogged cases has been the implementation of new collective agreements.

Most public service unions have recently hammered out new contracts with the government, but acting on those new contracts often involves retroactive pay and signing bonuses.

In some cases, compensation advisers must retrieve data from an old pay system. This time-consuming process has meant more compensation advisers were moved to dealing with collective agreements instead of solving backlogged Phoenix cases, leading the government to hire more compensation advisers.

Mr. Brison noted the collective agreements at the Oct. 5 press conference, saying they were working on getting the problems solved. Mr. Brison and Ms. Qualtrough said the case backlog will probably get worse before it gets better.

The number of pay changes waiting to be processed grew from 237,000 on Aug. 23 to 257,000 as of Sept. 20, according to the government’s latest monthly update on how it’s dealing with the backlog.

“There is no question that we have made significant progress in terms of [the collective bargaining] negotiations, but it does impose challenges, which we’re seeing right now in terms of the Phoenix system,” said Mr. Brison.

The Hill Times


The Goss Gilroy report’s ‘lessons learned’ from Phoenix:

  1. Properly define what is changing
    • Define the scope of areas undergoing change and define the changes as discrete projects with associated interrelationships and interdependencies within the overall initiative.
  2. Articulate discrete projects and how they interrelate
    • Develop a roadmap to articulate the interrelationships and interdependencies between the discrete projects within the defined scope of the transformation initiative (as well as with other related projects).
  3. Assign accountability and authority to a single office
    • Assign accountability and authority for a multi-department/agency or government-wide transformation initiative to a single minister and deputy head, with the accountabilities, authorities, roles and responsibilities of other implicated organizations being designed, documented, and implemented as part of an overall accountability framework.
  4. Establish broad and inclusive governance
    • Establish governance that fully reflects the broad range of stakeholders affected by the entire initiative. For example a committee with membership from end-to-end process owner, the organization(s) leading projects, and different types of stakeholders should oversee the initiative
  5. Ensure oversight provides sufficient challenge
    • Establish a challenge function for effective independent oversight that encompasses the complete scope of the transformation, drawing upon organizations with international experience and individuals in that transformation area, and reporting directly to the minister or deputy head accountable for the overall initiative.
  6. Assess and then adapt or manage the culture and leadership environment
    • Assess the culture of the organizations in which the transformation is occurring. Ideally, organizational culture should be agile, open to change, engaging, responsive to stakeholder feedback and willing to receive unwelcome and unexpected information. Where culture is not ideal, take steps to manage the transformation within that culture (for example, heightened challenge function, effective communications).
  7. Treat change management as a priority
    • Plan and implement change management as a priority throughout the life of any high risk, complex transformation initiative. Change management cannot be considered optional, an add-on, nor expendable when looking for ways to save time or money.
  8. Focus on communications that are effective
    • Communicate in a relevant, timely, comprehensive manner as a key priority of the change process, integrating into and across every project, phase, and activity, with the effectiveness of communication being the measure of success.
  9. Engage and communicate with stakeholders regarding anticipated IT solution functionality
    • Communicate the anticipated functionality of any required new IT solution (including any limitations) to all stakeholders. Strategies for dealing with any limitations should be discussed and developed in collaboration with affected stakeholders and be ready for implementation to go-live.
  10. Do not conduct extensive or risky customizations if they can be avoided
    • Align business processes and organizational models as much as possible to the underlying structure of any cut off the shelf software IT solution being considered, rather than engaging in extensive or risky customizations.
  11. Implement outcomes management throughout the life of the initiative
    • Develop and fully implement explicit outcomes management throughout the life of the transformation initiative from inception to post go-live. Success must be assessed for progress on implementing various projects that make up the initiative, including business transformation, change management and any new IT solutions.
  12. Do not expect savings until well after implementation
    • Realistically challenge the timing of when benefits (for example, efficiencies, cost savings, service quality) will be actually achieved recognizing experience to date and the experiences of others, then adjusting as necessary over the life of the transformation initiative.
  13. Support affected stakeholders if they are the play a role in the initiative
    • Fund and equip departments and agencies to successfully carry out the necessary activities to support horizontal transformation initiatives, including access to change management expertise, financial and human resources.
  14. Fully test the IT solution before launch
    • Launch any required new IT solution only after it has been fully tested with end-to-end real-life simulations using a broad spectrum of real users and when all doubts regarding success have been addressed and verified independently.
  15. Leverage and engage the private sector to maximize initiative capacity and capabilities
    • Involve government and the private sector fully and collaboratively in the critical initial stages of any transformation initiative, before adopting one particular approach and with full consideration that each can bring to possible approaches.
  16. Reassess, learn, and adjust
    • Update and revisit regularly the business case for the transformation initiative and individual projects as the implementations evolve. In addition to assessing whether key assumptions were accurate pertaining to time and budget, consideration should include any changes in the initiative and projects’ scope, the progress and success in stakeholder engagement, the effectiveness and projected timing of any business or organizational transformation and the effectiveness of governance and leadership.
  17. Identify and establish required capacity prior to go-live.
    • Manage the required workforce transformation as an explicit function or project through the life of the initiative from inception to post go-live ensuring the necessary workforce capacity, expertise and corporate memory are available through the end state. Establish a detailed contingency plan that can be readily implemented, if necessary.