Agile Project Management and Data DevOps

The modern best-practice for delivering data solutions is to use agile project management principles (such as Scrum or Kanban) and agile development operations known as DevOps.

Agile projects broadly apply the tenants of the agile manifesto, that have been stated as favouring:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

DevOps, on the other hand, is a set of practices intended to reduce the time between committing a change to a system and the change being placed into normal production all while ensuring high quality. Both approaches need to be enabled by a delivery team and adequate infrastructure investment in hardware and software. Usually, the delivery team will carry certifications on the use of a given software tool which can accelerate development. Preference should be given to technologies that are both widely available and that the local industry supports.

The typical roles and responsibilities for such a team are outlined below:

Product Owner– Primary stakeholder for the project from the client-side
– Responsible for the vision of the project and communicating that to business and development team stakeholders.
Lead Technical Architect – Effectively communicates with clients and it able to explain project direction to both business and technical stakeholders
– Articulates a high-level technical design, architecture and approaches
– Overall responsibility for solution data model
– Prescribes accelerators and frameworks (such as auditing and package templates)
Project Manager / Scrum Master– Planning and coordinating the team using agile principles
– Managing the execution of user stories and related tasks
– Removing any blockers that may affect the development team
Business Analysts– Responsible for requirements gathering
– Interface between business and development team
– Drafting user stories (As I, I want, So that…)
– 1st line responsibility for user acceptance testing (UAT)
Data Developers– Configure analytical cubes used for business user consumption
– Configure dashboards and produce visualisations
– Produce visual insights from presentation level data
– Align to the technical architecture (such as audit frameworks and implementing technical controls)
Testers– Develop SQL test scripts to “reverse formulate” business rules and logic
– Cross-check the warehouse for consistency
– Responsible for fault finding and troubleshooting of data issues before they become a problem to business
– Unit tests are the control used for ensuring data quality.
Data Scientists– Provides statistical and machine learning expertise to find a pattern in data that can profit the organisation.
– Need to work in the context of operationalising analytical models, where appropriate.
Database Administrators– Responsible for administering and optimising database queries
– Handles deployments to testing and production environments

It is advised that the team uses modern business requirement gathering and source control tools (like Jira and Git) to support their DevOp practices.

Business stakeholders can start seeing a return on investment once everyone gets the ‘game-plan’ and works toward a common goal as a single unit.

Proposal / Call to action

Ilion is looking to help financial services organisations optimise their return of investment in data centred around financial products and services. Our executive team have extensive experience in financial analysis for banks, credit risk and stress testing and data management. We see an overlap in these areas that provide a sweet spot for a return on investment made.

When you are working on your next financial analytical undertaking, please don’t hesitate to call us. We would be happy to help to find a solution that is optimal for your organisation.

This article is available in a downloadable whitepaper: