A Overview to how Drupal Development is managed by miggle.co.uk

 A Overview to how Drupal Development is managed by miggle.co.uk Alick Mighall – February 2014 Table of Contents How we manage projects ................................................................................................... 3 Delivering self-­‐sufficiency ............................................................................................................................... 3 Project management .......................................................................................................................................... 4 Overview ................................................................................................................................................................. 4 Communications ................................................................................................................................................. 5 Risk management ............................................................................................................................................... 5 Business continuity planning (BCP) ........................................................................................................... 6 Change control ..................................................................................................................................................... 6 Controls and escalations .................................................................................................................................. 6 Training, documentation and code commenting .................................................................................. 7 Training .................................................................................................................................................................. 7 Documentation .................................................................................................................................................... 7 Code commenting ............................................................................................................................................... 7 How we use Drupal ............................................................................................................ 8 Core Approach ...................................................................................................................................................... 8 Drupal Theming ................................................................................................................................................... 8 Drupal Site building ........................................................................................................................................... 8 Custom module creation .................................................................................................................................. 8 Drupal community engagement ................................................................................................................... 8 Managing Content in Drupal .......................................................................................................................... 9 Future proofing and reusable technology ............................................................................................. 10 Minor upgrades to Drupal ........................................................................................................................... 10 Major upgrades to Drupal ........................................................................................................................... 10 Hosting and Deployment .................................................................................................. 11 Hosting ................................................................................................................................................................. 11 Continuous Integration ................................................................................................................................. 11 CI Workflow ....................................................................................................................................................... 11 Hosting ................................................................................................................................................................. 13 Live and staging environments ................................................................................................................. 13 Development environments ........................................................................................................................ 13 Back ups .............................................................................................................................................................. 13 Our approach to testing .................................................................................................... 14 Unit testing ......................................................................................................................................................... 14 Test on build – at dev/WIP (Work in Progress) ................................................................................. 14 QA (Quality Assurance) ................................................................................................................................. 14 UAT (User Acceptance Testing) ................................................................................................................. 15 Smoke/Regression testing ........................................................................................................................... 15 Stress and Pen (Penetration) testing ...................................................................................................... 15 How testing fits into workflow .................................................................................................................. 16 “At miggle we believe in your freedom to operate. Everything we do is designed to make you as self-­‐sufficient as you chose to be. Our team believe they are empowered to achieve this. Freedom is baked into our collective source-­‐code. We strive to understand the freedom you describe and work with you to deliver it as a reality” How we manage projects This section outlines how at a high-­‐level we engage with projects, the way we work and the processes we generally use. We can tailor these as best suits you, but what we have in place here should demonstrate that if you chose us to look after your project, we can be a safe pair of hands. Delivering self-­‐sufficiency We believe projects work best when we help provide a roadmap to self-­‐sufficiency, as this is one of the key advantages to be exploited from use of open source technologies. Most of this section, in as far as it relates to project management, is about how we aim to deliver self-­‐sufficiency by how we manage your project. We also believe it’s key to have a good approach to business continuity planning (BCP), the bed rock of which are supplying solutions which are fully installable from source, backed up by clearly defined documentation and processes. At miggle we always feel this is achieved by:-­‐ I.
Documentation II.
Clear processes III.
Open source technology IV.
Use of accepted standards within open source technology a. Within the context of Drupal:-­‐ i. Commitment to coding standards (http://www.drupal.org/coding-­‐standards) ii. No core or contributed hacks 1. If we find bugs in code, patch it and contribute it back to the community iii. Use of contributed modules over custom modules 1. The less custom code we write the more future proof your solution and the easier to find staff or contactors to manage it iv. Where possible, use of base themes for design 1. Again, this makes it easier to find people who will know how to work on your solution if required V.
Dedicated, third party hosting a. We recommend that you own the relationship with the hosting provider so that you are not dependent on hosting through your web site development business. b. All the time you engage us we can manage the technical contact with the hosting provider on your behalf if required. VI.
A flexible support and maintenance package a. Regardless of how self-­‐sufficient you aim to be there are likely to always be areas on which you need support, or new features deployed. We offer flexible support packages which help you manage this Project management Overview From a miggle standpoint, projects are either led by our MD, lead developer or one of our Producers. They can be a single point of contact if you prefer, or put in place a formal process whereby we update on progress via a weekly report or call, and/or enable direct contact with the key people working on your project, if that helps. Generally at miggle we’ll have a number of projects on the go at once. Each developer usually has two projects in which they are primarily involved and then they’ll additionally support team members on their primary projects. This helps us make best use of what are complimentary and similar skills sets across the team. All of the team have a solid Drupal understanding, but, as an example, Louis is more of a SOLR search expert, Ian is more of a front-­‐end expert and Steve is more of a deployment expert. Then, of course, from a servicing the client perspective, we need to be able to take into account that clients’ staff often have other responsibilities, take holidays and are subject to the impact of over-­‐arching priorities within their own businesses. The upside of this is that this is what generally allows us to manage project for multiple clients concurrently. As a result of all of this, because we are never focussed solely on one project, we don’t run a single project management methodology, but you’ll commonly see us do the following:-­‐ Detail requirements Our initial responses to briefs, RFPs, ITTs or Market Requirements Documents (MRDs) are often quite detailed and thus often transition into the Schedule of Works (SOW) or Product Requirements Document (PRD) that projects require, particularly when there are multiple stakeholders. These requirements documents can outline either an agile or a waterfall style approach. The former will see us translate requirements into a task backlog, the latter into a project plan with milestones and dependencies outlined. The extent to which it is one or the other will depend on client requirements, whether there is a fixed deadline and budget, whether there is a phased approach or a requirement to establish a minimum viable product (MVP) or prototype first. How we work Unless a specific requirement is made of us to manage work in a certain way, because of the nature of projects we run concurrently we’ll adopt a range of tactics to manage work. I.
We run a monthly resource planning meeting with the senior team, where we map out resources as far in advance as we can. II.
We run a detailed weekly meeting where we review all project progress. III.
We’ll run a daily ad-­‐hoc ‘round the room’ which is like our version of a ‘stand-­‐up’ where we’ll quickly summarise progress and set short-­‐term goals in terms of progress. This is an essential process in an environment where priorities or availability on projects can change quite quickly. Once a project is live, we’ll tend to work in a more agile way, where future releases are built around a shared understanding of what tasks will be addressed and deployed, from a task backlog, within a fixed amount of time. Communications The success of the project will depend on having clear communication between us. Each week we’d contribute to a progress report or call, if required, the purpose of which will be to:-­‐ I.
Identify and deal with any issues as they arise II.
Update where we are versus your main project plan III.
Review key tickets in our issues software IV.
Review risks and identify new ones V.
This will form the core of our communication and should identify any conversations we need to have. Potentially the weekly report can form the basis of a weekly meeting or call if required. We use online tools (like Jira and TRAC, but we can use other systems if you prefer) for tracking issues and project tasks. This enables us all to communicate with our team directly through raising an issue or request online, which the whole team can then see. This means that any issues or requests will get resolved without delay. Risk management Identification and mitigation of risks will be key to the success of the project and if required we can run a risk management process whereby we state upfront risks and manage and monitor them through out the course of the project. Business continuity planning (BCP) In order to help meet any internal business continuity planning objectives you may have, as well as to enable us to meet our own:-­‐ I.
We will provide documentation on the solution we build. II.
The hosting we recommend will have a full service level agreement (SLA) III.
All of our code and data will be backed up, with the site backed up each day. Our deployment methodologies ensure the site is fully recoverable or transferable IV.
Iterative deployments post-­‐launch are regression tested V.
We run separate development and staging environments VI.
The team members on this project will be supported by others in their respective teams. VII.
Our project management methodology and process is accessible to others in the miggle team also. VIII.
By using open source solutions we provide clients with a solution that is not wholly dependent on miggle in the long term Change control Change requests will occur when work requested on the project is outside the scope of this response (once agreed) or the initial brief. Miggle will make any such work known to you and will issue a change request. Additional work cannot be undertaken unless a change request has been issued by miggle and agreed by you. Change requests can result from unclear briefing or from additions or deletions of work by the client while a project is in progress. Once a proposal is agreed by you, it is agreed that only work contained within this statement of work document (or subsequent agreed documents) is performed by miggle. A Change Request will be evaluated in terms of its impact on the project by evaluating: I.
The cost – Determined by the level of man days effort II.
The time -­‐ to complete the tasks III.
The risks -­‐ to the current plan to complete IV.
The business benefits – will change have a significant and positive impact on the project The need to manage change requests more regularly arises in waterfall style projects where we are working to fixed timeline and budgets. It is less of an issue in agile style projects where change becomes part of the general prioritised trade off that occurs around sprint planning. Controls and escalations No one wants anything to go wrong on a project. But in the event that something does, it is essential that there is a clear procedure for addressing issues, so that they can be dealt with as soon as they arise. We can propose and agree this procedure as best suits you. Training, documentation and code commenting Training We’d work with you to determine training requirements, which are most likely to be needed in terms of managing the content, but may also be required in managing CMS set up (so as to be compliant with what disaster recovery procedures we agree). We’ll usually undertake initial training at the point when you have your ‘first look’ at what we’ve built. This then enables you to start to add content to the site, which provides an opportunity to more closely inspect the functionality, which then aids in initial feedback, as well as informing some of the initial QA. Our training is usually done face to face, or on the phone, over two hours. Documentation Once the project is delivered we will complete documentation (given that the training stage can often determine output of documentation). Until documentation is complete we will fully support you in both the maintaining of the platform and content management support – but of course we may well be doing this anyway as part of a maintenance and support schedule. For a project like this we’d generally provide two documents:-­‐ I.
A guide to managing content. This is generally limited to how to use the tools – it doesn’t, as standard, carry any editorial guidance II.
A guide to managing CMS installs onto hosting. Code commenting One of the advantages of working with a framework like Drupal is that a lot of the code is commented already. In so much as commenting helps developers make amends, the ideal way to work with Drupal is to never make any hacks to the core code or to that of modules. This is important obviously as relates to ensuring future-­‐proofing, in that it makes upgrades as seamless a possible. How we use Drupal Core Approach There’s little point in using an open source solution if we then rip it to bits and embellish it with custom code, because the resulting product will quickly lose the benefits that come with iterative improvements to the platform, increasing risks in terms of BCP (Business Continuity Planning) and reducing its level of future proofing. So at miggle, we always look to stay as close to a core and contributed approach as possible, as per Drupal standards – the details of which are described below. Drupal Theming Our general approach with theming is to take a base theme approach, so that we are working within a framework that is freely accessible. In our opinion, this strengthens use of Drupal from a BCP perspective, which is key to us being able to provide operational freedom to clients. Drupal Site building All of the miggle developers have solid site building skills – even our product managers can get so far! Our approach to site building is as follows:-­‐ I.
All code is versioned II.
All code is deployed across multiple environments, which provide opportunities for unit testing, automated testing, regression testing, QA and UAT III.
No core hacks – unless as per point VI IV.
Contributed modules first approach. This is key from a BCP perspective. V.
No contributed module hacks – unless as per point VI VI.
Core and contributed code will remain untouched unless agreed that a patch is required to achieve a specific task -­‐ if this is the case, the patch is committed to the relevant module and typically a note will be logged to the shared issue tracking system. VII.
Adherence to Drupal coding standards as specified on drupal.org – especially in respect of custom development. Custom module creation We’ll look to create custom modules when the contributed route is exhausted. Key to this project is looking at where custom modules may have been created before the contributed route had been fully explored. Drupal community engagement We’d propose that if we can find opportunities to contribute modules back to the community via drupal.org we should do. We think there are benefits to this approach for clients, which is one we have taken previously. Managing Content in Drupal The diagram below focuses more on how the Drupal platform manages content and outlines the importance of having clearly defined content types, taxonomies and user profile data which can be used to order, cross-­‐promote, cherry pick and personalise content. Future proofing and reusable technology We are considering future proofing in the realms of the following:-­‐ Minor upgrades to Drupal Minor upgrades usually cover issues such as security patches and thus help maintain the integrity of the CMS. Given the amount of developers contributing to Drupal as an open source project, the benefit these level of upgrades provide is almost impossible to compete with from a proprietary standpoint. Major upgrades to Drupal Major upgrades to Drupal happen about every 2-­‐3 years. The upgrades to a new version are only generally essential if users wish to take advantage of something specific in the new functionality and are not critical until such time as it becomes apparent support on related functionality on an old version is being replaced for the new. We currently build all new sites in Drupal 7 – but there are still many Drupal 6 instances being actively maintained and developed. Drupal 8 is likely to launch at some time in 2014, ut it is not yet fully clear what the migration path will look like. Also it’s not clear at what point there will be sufficient compatibility with all the contributed modules any given Drupal 7 project uses, which would vary case by case. Drupal 7’s lifecycle will continue to at least the launch of Drupal 9, which, based on our assessment, is likely to be no earlier than 2017. Hosting and Deployment Hosting If infrastructure is something you already have in place then we’d just need to learn more about it and assess its suitability. We wouldn’t foresee any general issues with deploying to it and would expect it to tie into the continuous integration approach we outline below. If we need to provide hosting we can provide a range of quotes based on traffic estimates, budget, support/management requirements and BCP requirements. We generally propose Linux hosting from one of three suppliers, Memset, Rackspace and Acquia (who are Drupal specialists), depending on requirements Continuous Integration Miggle approach projects using a continuous integration methodology. What this means is that all code is evaluated and deployed in such a way that it can be rigorously tested before new functionality is released. Automated tasks are defined to reverse the release process should the need arise. A well-­‐defined testing process (as outlined later) is essential when sharing a codebase across multiple sites, as the shared nature of the proposed solution presents a single point of failure for all sites on the platform should any critical bugs make it to the production environment. CI Workflow Day to Day 1. Code is committed to a central repository 2. A platform build is triggered 3. Platform is tested against predefined criteria 4. Platform is deployed to the ‘Work in Progress’ environment (WIP) on success -­‐ error report is returned on fail Release 1. Code is tagged for release 2. Target Environment and release are input as build arguments 3. A platform build is triggered 4. Platform release is tested against predefined criteria 5. Platform release is deployed to target environment on success -­‐ error report is returned on fail See the following diagram for a graphical outline of this process. Hosting Live and staging environments We’d deploy two environments to your production hosting – Live and Staging (or RC as we call it):-­‐ I.
‘Release candidate' environment (RC). Initially we’ll run Quality Assurance (QA) tests on that environment and then let you know when we think that is production ready and ready for your User Acceptance Testing (UAT). Once you approve what is on RC via your UAT we make that the live site II.
‘Live' environment -­‐ where the live site lives and is where you manage and preview all of your site content Development environments There is also a development environment we run in the background, which builds regularly from the code base. This is called the ‘Work in Progress’ environment (WIP). If WIP cannot build, we know there is a problem. Regular rebuilds of WIP (usually every 30 minutes) give a quick insight as to how robust our development is. This is important because it creates an environment in which multiple developers can commit site builds to from their own local development environments. This provides you with a level of Business Continuity Planning (BCP) in that you can easily move developer responsibility from person to person. Back ups We would need to ascertain with any existing hosting provider what is in place to ensure the files on your hosting are backed up. We would expect that they’d do full server snapshots, so that any server can be restored from the last full known good back up in the result of a catastrophic hardware failure. With our preferred suppliers, we can detail the back up procedures and options each have in place as part of the decision process that might help you identify a suitable hosting partner The back ups at a software application level would fall more under our responsibility. Ordinarily we capture daily back ups of all files (images, documents you upload) and databases (which relates to the pages you manage, your registered users and site config) which we store for a period of up to 28 days on cloud based storage. The code with which your website is built is stored in a version controlled repository which is used to build the site which sits in the Live, RC and WIP areas. This code is also backed up. Our approach to testing For miggle.co.uk, testing is a combination of the following:-­‐ Unit testing This happens at a local development environment level. While developers will run ad-­‐
hoc tests on code at this stage we’ve not, to date, run any automated unit tests on code, even though our deployment methodologies are set up to be able to handle this. This is because, to date, it’s not been a requirement clients have prioritized paying for. Test on build – at dev/WIP (Work in Progress) As mentioned previously, we can configure the dev environment in a WIP format as part of CI to repeatedly build from the code repo to test the validity of the code at any moment. QA (Quality Assurance) QA is generally a process we outsource to our QA partners, whereby they’ll test the product against an agreed browser matrix, set of use cases and functional tests. They take point in resolving issues raised, so quality is managed/governed/approved outside of the development team. Before we hand over to QA we’ll have done basic browser testing. Often, we’ll have give clients a first look of the site, which usually involves training them in the CMS – thus many obvious bugs are flushed out before this stage making the best use of the QA time. With projects that we build from scratch we offer a 12-­‐month bugs fixed for free period (excluding errors in 3rd party code) for clients who take this approach with qualified/certified testing resources. Once QA is complete we’ll indicated to clients we think it’s ‘production ready’ at which point the site is ready for UAT. Our approach to Quality Assurance (QA) would be as follows:-­‐ I.
Once we reach a stage where the majority of HTML code is being generated by the CMS, our dev team would undertake an initial internal browser compatibility and accessibility test against the criteria that has been defined. a. At this point, there’d be an opportunity to give the client a first look of the new site to review progress in advance of any more detailed QA starting. II.
If commissioned, the external QA team starts its work, usually once enough content has been added to the site to make QA worthwhile III.
The QA team will have already written test scripts which will validate the main use cases agreed as part of the project. a. By this stage the QA team will most likely have a number of known issues/fixed bugs they can test/validate IV.
QA will focus on the following:-­‐ a. Test planning, documentation & scripting. To include familiarisation with the project, creation of browser, platform and accessibility scripts & creation of a high level exploratory test script which validates use cases V.
VI.
VII.
VIII.
b. Cross browser testing – does the site work in all target browsers c. Cross platform testing – does the site work as intended on desktops, mobile and tablets d. Accessibility testing – does the site meet the agreed accessibility criteria Complete use case checklist & exploratory test scripts. Issues to be reported to miggle via agreed issue tracking tool and daily reports made on test progress. Issue verification & re-­‐test. Verification of fixes made to bugs by miggle & appropriate re-­‐testing required. miggle process all output of QA. Once QA partners are happy all issues are addressed, we’ll signal that the site is ready for User Acceptance Testing (UAT) by the client. We’d expect UAT to largely follow the validation of issues as per QA as well as focus on key use cases UAT (User Acceptance Testing) UAT is generally managed by clients – often running a subset of tests run at QA. UAT additionally might include testing against internal business processes, legal review, extensive copy checking/proof reading etc. Smoke/Regression testing When subsequent releases of the product are made, we’ll always advise regression testing is undertaken – which will usually involve running tests against the key use cases – signaling the release as production ready once done – so similar to QA – and in that regard, often a process our QA partners will assist with. The way we handle our code repositories, in respect of our ability to roll-­‐back and branch, is key to being able to respond to the output of regression testing. Stress and Pen (Penetration) testing Penetration testing only really ever needs to be done in respect of custom modules – as contributed modules and core can be safely assumed to have been pen tested as part of Drupal’s release process. Our approach for stress testing would depend on traffic levels and who we host with. How testing fits into workflow The diagram below shows how testing fits into our overall workflows.