Valid arguments for push back on inappropriate and unnecessary analysis requests
Let me start by saying that we are good soldiers and scouts and we should be doing all the tasks necessary for our employers to meet their data creation and dissemination obligations. If we were in an over-resourcing organizational crisis within our organizations or within our functions, requiring us to look and act busy, then it wouldn’t be necessary to write this piece. If a scientist comes to us with a theory that melatonin activates cereblon and makes your cancer drug more effective, then, if we had that over-resourcing crisis, we should absolutely say “we will have all 50 tables rerun for the Blonde and Brunette subsets that you have requested to test out pigmentation related effects”. However, we are incredibly busy and even if we have occasional gaps between projects we are proactively working on methods and projects which will be very useful in the future. Our resources are scarce and need to conserved for essential activities through statistically valid push back on inappropriate and unnecessary analysis requests .
Sometimes the sample size of study subgroups are small. Further, subsets sometimes split up very unevenly. Subset analysis requests usually come as a request for many tables repeated for each subgroup. In such circumstances, we could argue that additional unplanned analyses will detract from the primary conclusions. Errant random data in a small subgroup may not agree with overall conclusions drawn in the complete cohort.
Alternate analyses avoiding such onerous tasks include simple SAS PROC outputs to provide preliminary data for main effects and any interactions (with treatment) for the covariates creating the subsets, and the use of Forrest plots. In other contexts, as well, see if you can provide a quick SAS PROC analysis to provide data to convince the team not to go forward with a new set of tables.
The quantity of tables is inversely proportional to the quality and integrity of the tabled information even after extensive QCing. A new table can take 3 to 5 business days. Repeat tables are onerous and are mind-numbing work for programmers to run, statisticians to QC and the clinical team to review. Avoid these to help enrich working lives, towards analytical and creative functions, of many people in our teams – both upstream and downstream.
Reducing the volume of requests
When in doubt ask if an analysis is necessary. Will the data be reported in an abstract, publication, poster/presentation or CSR or is it simply a nice to have? Prioritize the necessary analysis, and under time constraints, ignore or deliver the ‘nice to haves’ last/later.
Request limited contexts for requested analyses – “Would just the cytogenetics tables for this context be adequate?”. Clients sometimes do not want to take the effort it takes to pick and choose what they want and will request everything. Challenge them to help us to prioritize and limit the scope of requests.
Avoid catering to too many analyses addressing ‘what if journal reviewer requests” or “what if we are asked when presenting the poster” or ‘what if there were a regulator question’. There are usually 10 what ifs for every request that is eventually made and that request is typically not one of the anticipated what ifs. You can say “We will provide if a journal reviewer requests” and you can encourage those manning our posters to say “that is a good question – we will investigate that and get back to you in a few days”. This way you address the one query that does come up instead of 10, and you can address the question post-conference or after submission for publication or to an agency when you have more time and resources.
Push disruptive add on requests to the next stage. You could say that you will certainly do it — not now for the abstract, but at the time of the conference/poster or if it is at the time of the poster, you could offer to do it for the manuscript, or if it is for a clinical study report, you could offer to do it for the eventual submission to an agency or commit to provide to an agency as part of post-approval periodic updates.
Draw schedules containing all team projects. For ad-hoc unanticipated requests discuss the impact on other analyses in the pipeline. Pull up your excel or word schedule and pit an ad-hoc analysis for one project with ad-hoc or planned analyses on other projects.
For listings, avoid PDF format, especially if the request is ad-hoc or for a publication. Recommend excel spreadsheets (making sure column variables and sheets are properly labeled). Formatting to fit a lot of data on a landscape page is somewhat difficult for programmers and is not an issue with excel output. Besides the users prefer excel listings as they can sort and filter content.
Pre-planned, pre-approved and well thought out analyses should be retained through the course of a project to the extent possible. Often requests for changes to the analysis come from the requester’s hope that we will see nicer results with a different analysis. This is data dredging. Do a quick and dirty (no tables needed) sensitivity analysis (but a statistically valid improvement in line with the requester’s request) to demonstrate robustness of the planned analysis. Unless you made a gross error in initial statistical design retain pre-planned analysis as the core analysis and address any challenges to its validity through sensitivity analyses.
Even in a secondary publication setting, if there are too many endpoints requested, ask the requester to pick primary and secondary endpoints and commit on endpoints which he/she will report prominently. Consider adjusting for multiple comparisons. Help prevent cherry-picking as well as reduce your work load.
You are a scientist as well. Your statistical training as well as your clinical trial experience has given you the ability to smell out bad science. Seek a seat at the table at all discussions with key opinion leaders (KOL)/regulators and primary authors. Use your expertise it to steer questions and hypotheses in directions which are statistically meaningful and easily implementable. Always be pro-active in offering analyses and addressing questions. Try to avoid someone else telling you how you should be doing your work – that is more likely to happen if you are not part of the discussions and with team members who do attend telling you to ‘just do it’ because the KOL at MD Anderson said so. KOLs can be somewhat immutable on the clinical questions (even these are can be malleable under the scrutiny by a good statistician) to be answered, but they will give you a lot of leeway on the analyses to address them. When someone tells you how to do things as well, it usually results in repetitive work and a large volume of work which will later be difficult for the team and the requester to understand and interpret.
Our tables, listings and graphs (TLG) should have 100% accuracy on the data content and all efforts should be made to make them easily interpretable and well understood by the wide audience in our clinical teams. However, we should often push back on cosmetic changes to TLGs, especially for publications. Usually medical communications personnel can make any changes to documents which go directly into the manuscript such as moving legends, changing titles and footnotes etc. Programmed output are usually source tables from which data is extracted for end documents such as publications and study reports. So, we can always request that any change be made in the abstract, poster, or end document figures and tables rather than sending TLGs back to programmers for reruns.
Resist refreshes — it takes a lot of effort at the back end (stats and programming) and front end (Medical Communications). Further, two slightly differing pdf decks can lead to a lot of confusion once the data is distributed. Clinical clients take estimates such as median time to progression as writ in stone rather than randomly varying entities which can change with increased follow-up. A change in the median from say 18.3 months to say a lower number of 17.5 months can lead to considerable consternation and a complete re-evaluation of every data point and every heuristic. There may not be a statistically valid rationale for a refresh once the study has reasonable follow-up. Demonstrate the lack of incremental utility using a side by side comparisons of some simple data such as the overall PFS curves for the two data cuts (medians and % survival differ by only very small amounts). It may not be judicious to do a data refresh between an abstract and an eventual conference presentation/poster – the abstract is published and we want to stay consistent with that. The course of an analysis, especially those for publication, typically includes 2 to 3 pdf decks of collected analyses followed by disparate analyses by disparate programmers and statisticians to address odds and ends from the clinical team, reviewers and authors. So, refreshes collating and repeating all that, are complicated and take time, pushing back the start of succeeding projects.
Given a tight deadline for an abstract and poster, encourage your clinical team to circumvent additional analysis by the creative use of language – medical communications should be good of that. A descriptive statement can often be a good substitute for a statistic or a p-value. One does not want to send a quick and dirty and unQCed (and likely inappropriate) statistic or inference a day or so before a submission.
Conclusions: Communicating with your team
We must negotiate or we will just be road kill on the information processing super highway. Instant and consistent gratification of our clients’ needs, need to be sacrificed in favor of delayed gratification so that we can reach our critical internal and publication analysis goals.
We need to be careful about language when negotiating. Be positive and use expressions such as ‘of course’, ‘absolutely’ “great idea’ “understand your perspective’ — don’t follow this with ‘but” — you could say “and we will definitely’ do ‘that in time for the manuscript/poster’, ‘that as sensitivity analysis supporting the results’, ‘that in a statistically valid and resource conserving manner as follows …’. etc. Not all of us are mindful, engaged and good at ‘thinking on our feet’ during meetings. We have strong quantitative skills possibly at the detriment of verbal communication skills. To contribute during meetings, we may find ourselves mindlessly bringing up historical concerns (or even the issues that I raise in this note) which are not relevant to current discussions. Alternately one may have some hunches about an alternate solution which needs further exploration and which are difficult to articulate immediately. It may often be helpful to think through issues outside the meeting and write a note to the team describing a way forward which is advantageous to all.
For a good part, we acquiesce to requests, often, we try to steer requests in a direction palatable statistically and in terms of resource use. We do come across narcissistic, aggressive or inflexible people, and in such circumstances, we should try to build support within the rest of the team (a usual reputation of active co-operation helps) and consider recruiting the help of managers and the leadership in the statistical and programming functions.
Avoid getting agitated during meetings (or in your electronic communications) and be calm and rational when explaining your alternate takes. A little joke helps sometimes. When my workload gets heavy I sometimes tell clinical teams to be sure to send our team of statisticians and programmers our crate of whisky, cigarettes and aspirin to help us along!