Session 10 (QC Part 2) - Q&A
Questions for: Dr. David Mortimer
- What’s the most common mistake that people make when starting QM?
- What is the best KPI to measure success..per embryo transfer or transferred etc etc
- My personal preference for this would be the implantation rate per embryo transferred as it “corrects” for cases where more than one embryo was transferred.
- When values are consistently low below benchmark..what should you do? Lower the KPI value?
- Absolutely not – remember that benchmarks are rounded-down values that are obtained routinely by the best labs. So, undertake a full review of the process or system that is under-performing, identify all aspects where / how it is suboptimal in either its design (choice of method/products/etc) or performance (e.g. are all staff following the optimized SOP?) then create an Action Plan to bring it to optimal performance, including identifying all the (K)PIs that would allow you to monitor the effect of the changes. Then implement the changes and monitor the outcome. If this doesn’t achieve the goal of reaching the benchmark value then look again for something you missed, and/or seek help.
- Liesl Nel-Themaat: Dr. Mortimer, can you talk about QA as it pertains to andrology and endocrine testing, and how to use the westgard rules.
- When should there be another Vienna consensus? With advances and intro of freeze all, new PGT, AI..does it need a reboot?
- A consensus meeting can only take place when there is widespread experience of something, and should be reasonably focussed to avoid the situation arising that when a small part changes the whole earlier document needs to be updated even though most of it hasn’t changed.
This is why the Vienna Consensus updated the 2nd Alpha consensus on cryo KPIs regarding the new routine use of blastocyst vitrification. But I believe it is too soon to consider updating the Vienna KPIs consensus.
An example of this would be in the late 1990s when the WHO recognized ART for male factor there was a need to change the clinical guidelines for male infertility from what was in their “WHO Manual for the Standardized Investigation and Diagnosis of the Infertile Couple” (CUP, 1991), and hence the “WHO Manual for the Standardized Investigation, Diagnosis and Management of the Infertile Male” (CUP, 2000) was published.
So, regarding “PGT 2.0” there might be justification to hold a consensus meeting on that, but artificial intelligence is still not even here, so in my opinion this would be premature
- Celal Kaloglu: Dr. Mortimer do you think there are a lot of KPIs parameters and they should be revised to the most important and useful ones?
- Marlane Angle:The Vienna Consensus has 12 recommendations. I think it is up to each lab director to decide what to use.
- My personal opinion is that one should have as many KPIs as one finds useful to be able to monitor all the aspects of one’s lab processes. I showed the minimum list that we prefer, which is larger than the Vienna Consensus list that reports what the group accepted as the minimum list.
- Carlos Encinas: Are there particular KPIs for IVF programs that work continuously throughout the year compared to those that batch patients? Should we pay more attention to certain indicators depending on the approach? Thank you
- In this case the ART KPIs could be calculated for each series of cases. Use as many KPIs as you feel necessary to be able to feel confident that the lab’s processes and systems function properly during each series. The Andrology Lab would probably function continuously, so its KPIs could be calculated on a monthly basis.
- Panagiota Efstathiou: Thank you Dr Mortimer for the insightful presentation. If you saw a sudden decrease for a specific KPI (without having implemented a new method, culture media,etc) in the lab what steps would you take to ensure what went wrong?
- Answered Live
- Marlane Angle: Personally, I would have a chat with my physicians—LOL. But I would also look at media lots.
- Heba Aburabie: What is the exact blastocyst formation percentage rate in an ideal ivf lab?
- Alison Bartolucci: Great question! There are so many variables to consider including patient age, culture system, criteria for blastocyst culture and cryopreservation, grading system, etc. And, unlike pregnancy outcomes we don't have national averages to compare to. Some good benchmarks to use, other than those based on your own performance, are published in the most recent Vienna Consensus.
- Blastocyst development rate (“BDR”), especially if derived as a specific Day 5 BDR, will be affected by various aspects of the stimulation and the culture system, so it will vary between centres.
This is why the Vienna Consensus Group could not reach an agreed value for a BDR KPI. In fully optimized centres one might expect the great majority of blastocysts should be seen on Day 5, with some slow ones appearing on Day 6. We have used 60% (of zygotes reaching the blastocyst stage) for a Day 5 BDR. But different stimulation or trigger criteria or timing could generally slow down blastocyst development and/or cause blastocysts (particularly those created from post-mature oocytes) to have lower developmental potential.
Hence each lab should simply try to optimize their BDR KPI(s)
- ioana adina Rugescu In the context of Covid-19 pandemia, what is your opinion regarding the idea of an elevation of the positive pressure in the IVF lab ? And if you consider this to be an idea at which value to be established?
- Interesting question. But my opinion is that the design of the IVF Lab should be to protect the embryos, which requires positive pressure to prevent the dirty air entering the space. Using photocatalytic oxidation (PCO) devices in the HVAC system to eliminate the VOCs is the best idea as the u.v. will also kill viruses. Therefore management of infectious agents such as SARS-CoV-2 at the human level of patients and staff would be managed separately to the protection of embryos from contamination and toxic agents.
- Sung Tae Kim: What is a reasonable minimum cycle volume per month to analyze KPI?
- Answered Live
- Marlane Angle: You probably need at least 30 data points to start to see significant changes, so you might have to look at your data quarterly, or at some other frequency.
- Giles Tomkin How do you determine which cycles to include according to what outcome?
- Alison Bartolucci: What is likely the most important is that whatever criteria you decide to use, it is the same used each time you measure and compare your outcomes. Many groups choose patients that are 40 or younger or 35 and younger but as long as you are comparing the same groups that is what is important.
- Defining the dataset for each KPI should take into account confounding factors so as to give a KPI that monitors the intended endpoint as cleanly as possible. So for some KPIs the female partner’s age would be an obvious defining parameter. Since the fertilization rate can be adversely affected by the # of oocytes obtained, especially in cycles with very high numbers of oocytes, setting a limit on what is a “normal” response can make the Indicator less subject to patient casemix variability. In this we have often used a range of 3 to 15 oocytes to define the cycles included in fertilization rate calculations.
- Denis Islamgulov: How references of KPIs were calculated? it's is a percentage or a random parameter?
- I’m not sure what you mean by “references”.
If it’s the values included in the Vienna Consensus then they were derived from the literature, the Expert Group’s own experiences, and in some cases the values were clarified by large datasets provided by some of the Group members.
If it’s the control mean and warning / control limits then these must be derived from your own historical data since the purpose of the control chart is to monitor whether the process is operating within your own historical performance criteria.
- EFROSSINI KESSOPOULOU The charts you mentioned are freely available or do they need a subscription?
- Control charts can be plotted from your own data (e.g. collated in Excel) using software such as MedCalc (www.medcalc.org).
Some ART EMR systems include control charts in their internal reports, e.g. the Lab Performance Indicators report within the IDEAS system (www.mellowoodmedical.com), I’m not sure about other EMRs.
The “SMARTS” system that I showed in the final part of my talk is an analytics platform (currently in beta testing) that, while not actually being part of it, works with the IDEAS ART EMR, and can also integrate other external data sources with it. It is a value proposition add-on to the IDEAS system; at this time it doesn't work with other EMRs
- Praful Vaid Is there any software available to measure KPI in IVF field
- Marlane Angle: The dashboard he showed was in IDEAS which is an EMR so it is built into the system.
- Based on my statement that we should only ever enter a piece of information into a computer once, generation of KPIs and reports on them should ideally be integrated with the clinic’s EMR so it can use all the embryology (and andrology) lab data that are entered for each treatment cycle. I know that this is possible with the IDEAS EMR (www.mellowoodmedical.com), but I’m not sure about other EMRs.
I believe there are also some apps that can do this, but I’ve not used any of them. I think the data need to be entered separately into these apps for analysis.
- Liesl Nel-Themaat: What QC do you do on new media and reagents received?
- Shaakirah Moosa Do you believe that most KPI changes can be a cause human error?
- I think you are asking whether I believe that most variations in KPIs are caused by human error(s), not that changes in KPIs cause human error to happen.
I do believe that much of the variability in KPIs from period-to-period is caused by a combination of variable human practices and both equipment and environment fluctuations. Hence it is the Lab Director’s job to select and design systems that minimize such possibilities, and to train staff to be consistent in what they do and how they do it (which can be verified by ongoing IQC and PT [proficiency testing]).
Certainly it has been my experience auditing labs around the world that focus on quality in all its aspects does result in not only better overall success, but also lower variability (expressed as higher control means and narrower warning & control ranges in control charts).
- Jayaprakash Divakaran Shall we take all those different cases such as TeSA sperm and other severe sperm problems together for the KPI calculation?
- Answered Live
- Alison Bartolucci: Depending on what parameter you are looking at it, those types of cycles are typically excluded.
- Marlane Angle: The only time I would include these would be if I had a special KPI to look specifically at TESE outcomes. You can always create a KPI for these if you are concerned about this group of patients. But I would not include these in the KPI for fert rates for example.
- ioana adina Rugescu: Do you consider that the Istanbul consensum for Cryo KPi needs an update?
- My personal opinion (not speaking on behalf of Alpha) is that that document is now more of a historical reference, since slow freezing is now rarely used for oocytes or embryos. An update on the vitrification competency and benchmark values might be useful, but blastocyst cryosurvival was already updated in the Vienna Consensus. I’m not sure that trying to update the oocyte vitrification values would be productive since it would need to separate patient and donor oocytes due to their different clinical scenarios and applications – with still limited experience on warmed patient oocytes being available, and donor oocytes typically being very highly successful.
- Dr. Robert BADER: Is there any software ready to use to calculate all these KPIs ? thank you
- Control charts can be plotted from your own data (e.g. collated in Excel) using software such as MedCalc (www.medcalc.org).
Some ART EMR systems include control charts in their internal reports, e.g. the Lab Performance Indicators report within the IDEAS system (www.mellowoodmedical.com), I’m not sure about other EMRs. “SMARTS” is an analytics platform (currently in beta testing) that, while not actually being part of the IDEAS ART EMR, integrates with it, as well as with other related external data sources. It is a value proposition add-on to the IDEAS system and does not work with other EMRs at this time.
I believe there are also some apps that can generate control charts, but I’ve not used any of them. I think the data need to be entered separately into these apps for analysis.
Questions for:Dr. Elizabeth Hammond
- ioana adina Rugescu: How many cycles/month do you consider to be the minimum number in order to calculate KPI monthly? (David received a similar question)
- Answered Live
- 30 cases as a guideline.
- ioana adina Rugescu: In order to have a reliable KPI do you consider that we need to establish this/age category?
- Age is particularly important for pregnancy rates, we divide age by two subgroups, 37 years and younger versus 38 years and older
- Pedro Ferreira: Do you think Vienna Consensus should be reviewed taking account female age?
- Implantation rates at the cleavage and blastocyst stage are given in the Vienna consensus for patients aged 40 years and below. The rates could also differ from centre to centre according to the proportion of PGT-A tested embryos transferred and proportion of cycles using donor oocytes. Large data sets, with clearly described cycle characteristics like those described above will provide guidelines.
- Denis Islamgulov: Should we calculate references of KPI for different ethnic groups or age groups, countries?
- These can all play a role and again large data sets, with clearly described cycle characteristics like those described above will provide guidelines, in addition to ART registries within countries.
- ioana adina Rugescu: if there is a non constant number or cycles per month do you consider that the KPi monthly are relevant ?
- It depends, 30 cases per data point should be used as a guideline. If certain months are dropping below 30 cases, then quarterly rates could be used. Otherwise, if low volume months reach 30 or more cases, monthly tracking is appropriate, whilst keeping a record of swings in case load to incorporate into trouble shooting.
- Celal Kaloglu: Elizabeth should ıt will be more effective if KPIs rearranged based on the patient groups, for example in my labs, most of patients (70%) are poor responders, all these KPIs are valid for that group
- Answered Live
- A high proportion of poor responders could mean a lower average oocyte number. You could subgroup based on a certain number of oocytes as a threshold ( for example equal to or more than 4 oocytes), or for some KPIs, calculate the rates on a per oocyte basis, rather than a per case basis (for example ICSI fertilisation rates).
- EFROSSINI KESSOPOULOU: Do you think that the PR KPI is influenced by seasonality?
- Answered Live
- We haven't received any obvious seasonality effects when analysing KPIs. Wesselink et al., 2020 recently reported seasonality effects for spontaneous conception cycles. One thing to consider is that any seasonality effect would be even more complex for TER pregnancy rate by month as each month the original OPU date would differ.
- Alison Bartolucci: Pregnancy rate is influenced by an endless number of variables. It may be that seasonality is one of those variables - or even changes in HVAC conditions (air conditioning, heat, humidity, etc.) as a result of different seasons.
- Liesl Nel-Themaat: Elizabeth, when you record D5 blastocyst rate, do you pay attention to time of ICSI/Insem and time when you look at the embryos? Of course the earlier in the day you do ICSI and later in the day you look the more blastocysts you’ll have on Day 5.
- We aim for 116 hours for day 5, however as you point out there may be subtle differences for insemination time, checking time, and also freezing time according to clinical schedule. This is a limitation of the day 5 usable blastocyst rate.
- ioana adina Rugescu: do you consider that first we need to define ( to have definition) for what we want to compare ? All of us have as a success indicator the LBR but some consider a live birth any more than 22 weeks ; other 30 weeks.... so we do not have the same measure. Which is your opinion?
- A key driver of your definitions that you use is how easily you are able to access the information and how well it is coded in your system. Some sites may take a while to follow up live birth data for example, and you may have a preference for how you define clinical pregnancy. We use fetal heartbeat at 7-9 weeks gestation for clinical pregnancy - implantation and clinical pregnancy are important tracking KPIs because the data is available sooner than LB.
- Jayaprakash Divakaran: Do you consider the Blastocyst formation post D5 culture anywhere in the KPI?
- The total useable blastocyst rate - day 5 useable blastocyst rate will represent the useable blastocysts post day 5
- Heba Aburabie: what do you think about the unknown shifts to 25 % for blasto refers to
- Nothing had changed in the lab, so you might not always have an explanation for these types of shifts.
- Denis Islamgulov: when have we got new KPIs for IVF cycles with PGTa, IVM?
- For PGT-A pregnancy rates you could turn to the literature to devise your own benchmark value based on recent studies - be careful to look for similarities in cycle characteristics, including average number of embryos transferred, biopsy method ect
- Dr Sujala S: Does media lot have any major impact
- Answered Live
- Alison Bartolucci: Media lot is a very important variable and could have an impact on blastocyst development, fertilization and other lab KPIs.
- OLAJUMOKE DARAMOLA: how can KPI be calculated per individual in the embryology lab
- ESPERANZA CARBALLO: What about tracking the changes in the clinic, like switch in medication for stimulation?
- Dr. Hammond: Tracking during times of clinical protocol change is also important - depending on the change, they are likely to influence oocytes coming into the lab and the reference indicators such as the oocyte recovery rate.
- Seungwook Hong: Is there Day 6 blastocysts for KPI?
- Dr. Hammond: The total useable blastocyst rate - day 5 useable blastocyst rate will represent the useable blastocysts post day 5
- Seungwook Hong: Due to biopsy for PGT most of time we do biopsy on day-6 because more TE cells available
- Dr. Hammond: Proportion of PGTA cycles can shift the day 5 usable blastocyst rate, depending on which day biopsy tends to fall.
- Habib Azar: Is there any data to transfer blastocyst 3AB or 3BA
- Dr. Hammond: At our centre we prioritise TE grade (3BA) because we see a stronger link to pregnancy rates. This has been corroborated in the literature for some studies, but it is important to look at this effect within your own site.
- Sung Tae Kim: Do you have any recommendation to look at stimulation or egg quality regarding physician side?
- Dr. Hammond: A major reference indicator is the oocyte recovery rate by physician, this tests for stimulation effects and also procedural differences (ensuring everyone is going into the same size follicles for example).
- Dr. Mortimer: Definitely. One of the biggest issues I see when auditing or troubleshooting clinics is that an increased prevalence of late triggers reduces clinical outcomes (including perhaps an increased early pregnancy loss) due, I believe, to increased oocyte post-maturity. Trigger decisions are typically physician-specific practices.
- Murid Javed: KPI' take a lot of time. Whose task it should be; lab director or embryologist?
- Dr. Hammond: Both can take part. The important aspects are that data entry is correct and timely, and reviewed regularly for consistent quality management.
- Dr. Mortimer: In my opinion directing and overseeing the lab is the Lab Director’s responsibility, how they delegate the work is up to them. But ideally there should be little extra work involved since all data should only be entered into the EMR system once (by the operators, which are mostly embryologists), all subsequent analyses should be performed on those data.
- Denis Islamgulov: Why people use PR as KPI. It isn't a comparable value that strongly depends on the number of transferred embryos.
- Dr. Hammond: The data I presented from the paper was single embryo transfer. If you do have a higher average transfer you could divide the data into subgroups, SET and DET, or keep an eye on the average number transferred per month.
- ALISON BARTOILUCCI: It depends on what you are measuring. You are right, if you are measuring the "health" of the lab, PR might not be the best parameter. According to Dr. Hammond's work, the usable D5 blast rate may be one of the most sensitive lab KPIs.
- OLAJUMOKE DARAMOLA: can KPI result be based on the type of stimulation medication used in the different cycle batch
- Carol Curchoe: Total KPIs-score (C-KPIs+L-KPIs) can be correlated with the presence or absence of clinical pregnancy. Rosen et al. Day 3 FSH, number of mature oocytes retrieved, and E2 levels on the day of hCG were significant independent predictors of degeneration rate.
- Dr. Mortimer: This can definitely be the case as better outcomes have been related to the use of urinary menotropins compared to recombinant gonadotropins, see the MERIT and MEGASET studies in the literature.
- Dr Sujala S: when blastulation rate is good, but there is no pregnancy, what parameters need to be assessed
- Dr. Hammond: Look at blastocyst quality (i.e. proportion of top quality, aa grade embryos) and day 5 usable blastocyst rate to assess speed of blastocyst development. When the pregnancy rate is low check for any planned/unplanned changes in the lab and look at the cycle characteristics of the cycles coinciding with that period.
- Dr. Mortimer: One of the biggest issues I see when auditing or troubleshooting clinics is that an increased prevalence of late triggers reduces clinical outcomes (including perhaps an increased early pregnancy loss) due, I believe, to increased oocyte post-maturity.