Evaluating the diagnostic accuracy and clinical utility of 16S and 18S rRNA gene targeted next-generation sequencing based on five years of clinical experience

Abstract Background The use of 16S/18S rRNA targeted next-generation sequencing (tNGS) has improved microbial diagnostics, however, the use of tNGS in a routine clinical setting requires further elucidation. We retrospectively evaluated the diagnostic accuracy and clinical utility of 16S/18S tNGS, routinely used in the North Denmark Region between 2017 and 2021. Methods We retrieved 544 tNGS results from 491 patients hospitalised with suspected infection (e.g. meningitis, pneumonia, intraabdominal abscess, osteomyelitis and joint infection). The tNGS assays was performed using the Illumina MiSeq desktop sequencer, and BION software for annotation. The patients’ diagnosis and clinical management was evaluated by medical chart review. We calculated sensitivity and specificity, and determined the diagnostic accuracy of tNGS by defining results as true positive, true negative, false positive, and false negative. Results Overall, tNGS had a sensitivity of 56% and a specificity of 97%. tNGS was more frequently true positive compared to culture (32% vs 18%), and tNGS detected a greater variety of bacteria and fungi, and was more frequently polymicrobial. However, the total diagnostic turnaround time was 16 days, and although 73% of tNGS results were true positive or true negative, only 4.4% of results led to changes in clinical management. Conclusions As a supplement to culture, tNGS improves identification of pathogenic microorganisms in a broad range of clinical specimens. However, the long turnaround time of tNGS in our setting may have contributed to a limited clinical utility. An improved turnaround time can be the key to improved clinical utility in a future setting.


Introduction
Human infectious diseases are usually diagnosed by detection of a bacterium, virus, fungus, or protozoan from a clinical specimen in a patient with a clinical illness.Timely identification of the pathogenic microorganism is crucial for facilitating targeted antimicrobial therapy, avoiding adverse effects of broad-spectrum antibiotics [1], development of antibiotic resistance [2], and the risk of untreated infections and delayed diagnosis [3].Conventional microscopy, culturing, and biochemical testing remain the gold standard for detection of bacteria, fungi, and parasitic pathogens [4].However, culturing has low sensitivity, for slow-growing, fastidious or uncultivable microorganisms, and in general, decreased sensitivity in presence of antecedent antimicrobial therapy [5].
Molecular diagnostic methods are attractive alternatives as they do not require the presence of viable microorganisms in clinical specimens.Commonly used approaches include immunoassays or PCR assays; however, such methods are usually only able to detect a preconceived range of pathogens [6,7].Alternatively, broad-range gene sequencing of the bacterial 16S rRNA gene and the eukaryotic 18S rRNA (or other target genes) allows for detection of any organism present in genomic databases.Sanger (or chain termination) sequencing was previously used for this purpose [8], but has been replaced by next-generation sequencing (NGS), in which millions of primed DNA sequences are interrogated in parallel [4], enabling characterisation of complex microbial communities [3,[9][10][11], even in culturenegative specimens [12,13].The diagnostic turnaround time of NGS may be as short as a few days, while the main drawbacks are high costs, technical issues with contamination, and challenges in determining the clinical relevance of findings etc.Though the technical performance has been extensively studies, knowledge on the use of tNGS in a clinical setting is sparse.
This study aimed to evaluate the diagnostic accuracy and clinical utility of this tNGS assay in the North Denmark Region (NDR) over a five-year period from 2017 to 2021.

Design
This was a retrospective single-centre evaluation of the use of 16S/18S tNGS for diagnosing bacterial, fungal, and parasitic infections in a clinical setting.We evaluated findings of assays requested by physicians in the NDR during a five-year study period from 2017 to 2021.The evaluation was performed in cooperation between the Department of Clinical Microbiology (DCM) at Aalborg University Hospital, Denmark, and the Danish reference laboratory af Statens Serum Institut (SSI) in Copenhagen, Denmark.Data were collected via the Danish Microbiology Database (MiBa) maintained by SSI, and from two local registries of the DCM and NDR.Outcomes were diagnostic accuracy, clinical utility, and diagnostic turnaround time pertaining to tNGS.

Study population and setting
All patients for whom tNGS had been performed on request by physicians in the NDR during the study period were included.Patients were either hospitalised or seen in an outpatient clinic on suspicion of an infection.Patients were mainly residents of the region, however both Danish and foreign visitors were also included in the study.
The DCM at Aalborg University Hospital provides microbiological diagnostic services and consultancy on antibiotic treatment for all hospitals, general practitioners, and private specialist doctors in the NDR (catchment population of 591 724 citizens in 2021).
Specimens destined for SSI are shipped from all Danish DCMs on daily basis with arrival the following day.SSI performs tNGS once a week with a batch-up of all specimens collected nationwide.During the study period physicians in the NDR requisitioned tNGS as a supplement to the routine diagnostics (usually culture).
SSI is under the auspices of the Danish Ministry of Health and functions as both the national centre for disease control and prevention, and as a national reference laboratory.Since SSI sorts under the national government their clinical microbiologists are prohibited by law from accessing regional databases and accordingly, they often have limited clinical information about patients for which tNGS has been requested.

Microbiome sequencing
tNGS was performed at the SSI as described in detail by Hansen et al. [10].In brief, DNA extraction from clinical specimens was done using the DNeasy Blood and Tissue Kit (Qiagen Inc, D€ usseldorf, Germany).DNA was amplified using a 2-step polymerase chain reaction using custom 341 F/806R primers targeting the V3-V4 16S regions, as well as 3 primer sets targeting the hypervariable regions V3-V4 of the 18SrDNA gene.Amplicons were sequenced on the Illumina MiSeq desktop sequencer (Illumina Inc., San Diego, CA, USA) with the 500-cycle MiSeq reagent kit, V2, in a 2 Â 250-nucleotide setup (Illumina Inc.).For further details concerning primer design and library preparation, see supplementary material.16S and 18S rDNA sequence reads were annotated to taxonomic level using the software tool BION (Danish Genome Institute, Aarhus, Denmark).

Other microbial diagnostics
Most specimens were also examined in the laboratory at DCM using routine diagnostic methods as requisitioned by the physicians.This included culturing with bacterial identification by phenotypical testing or MALDI-TOF (Bruker, Bremen, Germany), immunoassays, and PCR assays, and as such no direct comparison between routine methods and tNGS was performed.

Data collection
SSI provided a list of accession numbers, pertaining to tNGS assays performed, along with personal identification numbers identifying patients in all three databases used.First, the MiBa database was reviewed for results of tNGS for every single patient and the results of other microbiological diagnostics.Next, the local microbiology database, WWBakt (Autonik AB, Sweden), was reviewed for results of tests performed at the DCM, including the result of culturing of the same specimen as send for tNGS and results of other relevant microbiological tests.Finally, the regional database of medical charts, Clinical Suite (DXC Technology Scandihealth A/S, Denmark), was reviewed for the following variables: age, sex, clinical presentation, blood inflammatory markers, descriptions of other diagnostic modalities, the diagnosis, and therapeutic interventions.In the overall evaluation of the definitive diagnosis, emphasis was put on the conclusions made by treating physicians, but in some cases, subsequent findings revealing an alternative aetiology for the clinical presentation overruled the diagnosis made by the treating physicians.
For patients with multiple tNGS performed on specimens of the same type, anatomical site and episode, we bundled the tNGS results, giving precedence first to positive results, then to the first assay result available.Specimens separated by type, anatomical site, or collection time (separate episodes) were considered independent.
We used no predefined 'gold standard' for genuine infection for each specimens.However, all positive tNGS or culture result was further reviewed by two consultant clinical microbiologists to determine whether the identified microorganisms constituted the most likely aetiology of the infection, for example with commensals to be designated as infectious.Positive cultures registered in WWBakt as contamination were reported as negative and not reviewed.The diagnostic accuracy of tNGS was defined as: True Positive (TP): There was evidence of infection at the site (and time) of sampling, and tNGS was positive for microorganisms considered the most likely aetiology of the infection.True Negative (TN): The clinical information did not suggest infection at site (or time) of sampling, and the tNGS analysis was negative or positive only for microorganisms identified as contaminants or nonpathogenic human commensals.False positive (FP): The clinical information did not suggest infection at the site (or time) of sampling, but tNGS was positive for well-established human pathogens.False negative (FN): There was evidence of infection at the site (and time) of sampling, but tNGS analysis was negative or positive only for microorganisms identified as contaminants or non-pathogenic human commensals.
For TP tNGS results it was registered whether the same pathogen was previously identified at the same anatomical site by other microbiological modalities.
The clinical utility was determined using the following definitions: Utility: The tNGS result was TP or TN, and none of the criteria listed under 'non-utility' were fulfilled.Non-utility: The tNGS result was FP or FN, or regarded as such by the treating physician, or there was no evidence in medical records that the treating physician had taken the tNGS result into consideration, or the patient died before the tNGS result was considered by physicians.

Statistical analysis
During medical chart review, results were registered in REDCap (Vanderbilt University, USA) and subsequently analysed using Microsoft Excel (Microsoft, USA) and IBM SPSS Statistics Version 28.0.1.1 (IBM, USA).Binomial proportion confidence intervals (CI), using a confidence level of 95%, were reported when relevant.Patient's age (years) and time intervals (days) were reported as median [1st quartile; 3rd quartile] (in tables stated as [Q1; Q3]).

Ethical approval
The study was approved as a quality assurance project by the Aalborg University Hospital Board of Executives (Project ID: K2022-050).Informed consent from patients was not required for this study.

Study population
During the five-year study period SSI performed 699 tNGS covering a total of 511 patients.We excluded a total of 20 patients for the following reasons: 1) lack of documentation about the specimen or the clinical presentation (n ¼ 11), 2) the specimen had been paraffinimbedded (n ¼ 5), 3) the specimen was collected postmortem (n ¼ 2), 4) the specimen was an inanimate object collected by the patient and mistaken for macroscopic parasites (n ¼ 2).
Among the remaining 491 patients we included 544 independent clinical specimens, see Table 1.The age of patients ranged from < 1 day to 96.5 years.Children (age < 18 years) contributed to 50 specimens of which 25 were from suspected CNS infections and 15 were from suspected joint infections.Notably, infants (age < 1 year) contributed with 20 specimens of which 17 were cerebrospinal fluid (CSF) specimens from suspected CNS infections.Immunosuppressive conditions or prescription of immunosuppressive medication were prevalent pertaining to 131 (21%) specimens.Among specimens from suspected joint infections and spine infections the prevalence of local prosthetic implants was 30% (n ¼ 37/124) and 28% (n ¼ 21/74), respectively.

Diagnostic turnaround time
The median turnaround time for tNGS at the SSI was 12 [10; 14] days, see Supplementary Table S1.Additionally, 2 [0; 5] days passed from sample collection to requisition of the assay, and 1 [1; 1] day passed transferring the sample from the DCM to SSI.Thus, the total turnaround time from sampling to a result being available in MiBa was 16 [13; 20] days.Among specimens from cases  S2).The detected bacteria sorted into 95 species of 49 genera, while fungi sorted into 9 species of 6 genera.Among TP specimens tNGS was positive for more than one agent in 26% (n ¼ 46) specimens.
Comparatively, culture was significant (i.e.true positive) for 89 specimens in which 102 bacterial, 10 fungal, and zero parasitic identifications were made (Supplementary Table S3).Among significant cultures, culture was positive for more than one agent in 9 specimens.An overview of TP pathogens detected by tNGS and culture is presented in Table 2.
Amplicon sequencing using Illumina sequencing (short read sequencing) can most often diagnose pathogens to genus level and may prove difficulty in diagnosing at the species level, however, tNGS detected Kingella kingae in four culture-negative specimens from four toddlers (age 1-3) with joint infection (elbow, hip, knee, and ankle joints, respectively).In all cases, effective antibiotics were administered empirically, and all four children were discharged before tNGS results became available.Furthermore, an unexpected case of tertiary syphilis was diagnosed when tNGS detected Treponema pallidum in one bone biopsy.Both K. kingae and especially T. pallidum are generally regarded as difficult to culture in vitro [14,15].

Clinical utility
tNGS results were concordant with culture in 73% (95%CI 69-77, n ¼ 398/544) of specimens and provided clinical utility in 35% (95%CI 31-39, n ¼ 189/544) of The Full list of identified microorganisms available in the supplemental tables.Ã Culture performed at the national TB-laboratory at Statens Serum Institut.
instances, including specimens were tNGS results did not spur changes clinical management, see Table 4.For 4.4% (95%CI 2.8-6.5, n ¼ 24/544) of specimens the test result did inform therapeutic interventions.These included targeted antibiotic therapy (either improved coverage or narrowed spectrum), appropriate cessation of antibiotics (in case of a TN tNGS result).'other interventions' included informed choice of peroral antibiotic for intravenous to oral conversion, prescription of a vaccine, and planning of elective surgery, see Table 4.
For 30.3% (95%CI 26.5-34.4,n ¼ 165) of specimens, tNGS test results provided clinically useful information by confirming of diagnoses or treatment decisions already made.This included ruling out infection (cessation of antibiotics not included), confirmation of an established antibiotic regimen by detection of pathogens not previously identified, and confirmation of previously identified pathogens.
For 65.3% (95%CI 61.1-69.3,n ¼ 355) of specimens, tNGS results provided no clinical utility, in most instances because no comment on the test result was recorded by physicians (n ¼ 263).

Discussion
We evaluated the diagnostic accuracy and clinical utility of 16S/18S tNGS used in a clinical setting, including specimens from almost 500 patients with suspected infection in the NDR from 2017 to 2021.We found that tNGS overall had a low sensitivity of 56% and a high specificity of 97%.Although prone to selection bias, the detection rate of causative pathogens was higher for tNGS than conventional culture.Compared to culture, tNGS detected a greater variety of bacteria and fungi, and results were more frequently polymicrobial.
False positive results are a concern and need consideration when applying the use of tNGS in a clinical setting [6,16].Our evaluation was based on tNGS results  Excluding cases where 16S/18S tNGS results led to cessation of antibiotics, including cases where antibiotics were either never started or ceased before the tNGS result became available, or continued against another site of infection.Of note, if the initial PCR was amplicon negative, tNGS was not performed.B tNGS confirmed presence of pathogen previously identified.C Change of drug or partial cessation of a multi-drug regimen.D Alterations to antibiotic regimen to broaden antibiotic treatment.E Including choice of antibiotic for intravenous to peroral conversion, prescription of vaccine, and planning of elective surgery.
subject to appropriate censorship by the reference laboratory, and additional interpretation by local clinical microbiologists advising the treating physicians.Accordingly, false positive detections of well-established human pathogens were made in only eight specimens, while detections of contaminants and human commensals were appropriately regarded as insignificant.
tNGS is an advanced technology that enables characterisation of microbial communities and diagnosis of infectious disease [3,[9][10][11][12][13].However, there is insufficient evidence of its clinical utility and how it is best employed.A recent study by Flurin et al. investigated the clinical application of tNGS, as part of a diagnostic algorithm, in which tNGS supplemented broad range 16S qPCR and Sanger sequencing [3].tNGS was mandated only when qPCR suggested the presence of bacterial rDNA, yet Sanger sequencing was (or was predicted to be) unable to make a bacterial identification.A total of 554 clinical specimens from normally sterile sites were examined with tNGS, yielding an overall sensitivity of 53%, which was comparable to our results.They concluded that tNGS markedly improved the sensitivity of their diagnostic algorithm, increasing positivity rates by between 33% to 231% depending on specimen type and antecedent antibiotic administration.It is relevant to ask, which specimen types tNGS may be employed on.We found that across the major specimen categories (CSF, abscess material, pleural fluid, synovial fluid, bone biopsies, and heart valves), tNGS was more frequently true positive than culture.However, there was high variation in sensitivities ranging from 27% to 100%.Others have evaluated the performance of tNGS focusing on specific specimen categories.However, the results are generally difficult to compare between studies due to differences in aims, patient populations, and methodologies, and the majority are of small sizes.
In prior studies of culture-negative CSF specimens from patients diagnosed with purulent meningitis (n ¼ 55), tNGS detected bacterial 16S rDNA in all specimens and provided the bacterial aetiology in 53 specimens [12,13].Also, in the diagnosis of primary cerebral abscesses (n ¼ 36), sensitivity is reported to be higher using Sanger sequencing than culture (94.4% versus 86%), and a substantial higher detection of pathogens (n ¼ 96 versus n ¼ 47) [10].Another study examined 62 samples from intracranial abscesses and found sensitivity of tNGS, culture, and Sanger sequencing to be 82%, 74% and 61%, respectively [17].In both studies, tNGS was superior to other methods in characterising a more diverse bacterial composition of the cerebral abscesses.
In a validation study of 16S tNGS on endotracheal aspirates from intubated patients with acute respiratory failure, tNGS results were largely concordant with culture in culture positive specimens, but also detected pathogens in one fifth of culture negative specimens [18].In agreement, use of 16S tNGS in pleural fluid may increase the positivity rate [3], and may also be used to distinguish between microbial communities found in empyema and in non-infectious malignant pleural effusions [19].In our data, BAL was the specimen which had most false positive results when comparing tNGS with culture.
Regarding suspected joint infection, sensitivity of synovial fluid was 44%, whereas it was only 27% in joint tissue in this study.However, we had a small sample size and low positive rate in the joint tissue, and selection bias regarding the choice of which specimen type to send for tNGS may play a role.
Multiplex PCR assays are increasingly used in routine diagnostics; in a study of 60 patients with knee arthroplasty failure, 16S tNGS had higher sensitivity than the PCR assay (93% versus 56%), whereas specificity was high in both methods [7].However, the lower sensitivity of the PCR was due to a high prevalence of Staphylococcus epidermidis, a common pathogen in periprosthetic joint infections, which was not included in the assay.In a study including 36 synovial fluid from patients with non-infectious arthroplasty failure or periprosthetic joint infection, sensitivity of tNGS and culture were comparable (72% versus 69%) [20].In addition, the sensitivity of tNGS in sonicate fluid for diagnosing prosthetic joint infection, was also found to be high in two studies (n ¼ 47, sensitivity of 85%; n ¼ 208, sensitivity of 78%) [21,22].However, results are difficult to compare as only about one fifth of synovial fluid specimens in our study were sampled from joints with a prosthetic implant.
In agreement with our results for intraabdominal abscesses, one study reported sensitivities of both tNGS and culture to be high (88% and 75%) in samples from liver, pancreatic and renal abscesses (n ¼ 59), however the detection of bacteria by tNGS largely exceeded that of culture (n ¼ 227 versus n ¼ 69) [11].
Finally, although our study did not include a meaningful sample size of blood or heart valve specimens, others have found tNGS to add to the diagnostics of sepsis (n ¼ 60) [23], whereas there was no added value in patients with infective endocarditis (n ¼ 27) [24].
On the matter of clinical utility we found that despite tNGS results were true (TP or TN) for 73% of specimens, just 35% of results provided clinical utility and only 4.4% of results informed a change in clinical management.For 48% of specimens there was no acknowledgement of the tNGS result recorded in medical charts.These findings may be explained by relatively long median total turnaround time of sixteen days.By this time most cases were resolved without tNGS, e.g. by identification of pathogens using other microbial diagnostics, by patient recovery on empirical antimicrobial therapy, or by discovery of a non-infectious aetiology.For 56% of specimens from hospitalised patients, the patient was discharged before tNGS results were available.As such, when tNGS results were utilised, it most often confirmed conclusions already reached.Likewise, prolonged delays may have lowered the likelihood of physicians acknowledging the result or recording their acknowledgement of the result.
Comparatively, Flurin et al. found that 16S tNGS had impact on clinical decision-making in 8.7% of cases [3].They stated hesitancy by treating physicians to trust the results of this novel assay as a limiting factor for clinical utility.However, such concerns were not encountered in our evaluation.
Thus, in our setting the median total turnaround time of sixteen days constituted a major limiting factor on clinical utility of the assay.In an ideal setting with use of optimised and validated workflows the turnaround time of tNGS could be two to three business days, perhaps even shorter [6].However, at present the SSI performs tNGS on app.3000 specimens per year, and batching specimens for a weekly tNGS run is more costeffective.
We recommend that when implementing tNGS into routine microbiological diagnostics, treating clinicians should be provided with clear guidelines for its usage which encourages a decision at the time of sampling.Furthermore, logistics and laboratory services should be appropriately funded and optimised to facilitate a turnaround time, not substantially longer than an ideal tNGS workflow.
Strengths of this study include the evaluation of realworld diagnostic accuracy and clinical utility of tNGS in a large study population representing the entire usage of this assay by physicians in the NDR within a five-year period.
However, there are also limitations which needs consideration: The retrospective observational design does not allow for control of selection bias.During the fiveyear period, tNGS was employed on a highly selected group of patients and the assay was requested without clear guidelines.This limits the generalisability of results.
Notably, tNGS was often requested for culture-negative specimens, for specimens affected by antecedent antibiotics, and for specimens from immunocompromised patients.Furthermore, selection bias prohibited valid comparison between tNGS and culture in terms of diagnostic accuracy.These challenges may be overcome by use of a prospective design investigating a clearly defined concept for employment of tNGS.
The diagnostic accuracy of the of tNGS was dependent on the analytic abilities of SSI clinical microbiologists, which must differentiate true pathogens from human commensals and contaminant microorganism when submitting tNGS results to MiBa.As they are prohibited from accessing medical charts, limited clinical information may be detrimental to the diagnostic accuracy of tNGS.Arguably, this was technically not a weakness of the study, but rather a characteristic of how tNGS was implemented.
In conclusion, this study shows that, as a supplement to conventional microbiological culturing, the 16S/18S tNGS assay does improve identification of pathogen microorganisms in a broad range of clinical specimens.With appropriate counselling by clinical microbiologists, detections of human commensals and contaminant organism can be appropriately censored.However, in our setting the logistic setup and lack of guidelines prolonged diagnostic turnaround times substantially beyond what tNGS could technically offer in a setting with more ideal implementation.For this main reason, tNGS has yielded limited clinical utility during the study period.

Table 1 .
Demographic overview by suspected site of infection.
A: reported as n (%).B: reported as median [Q1; Q3].C: Any immunodeficiency was defined as the total prevalence of immunosuppressive drugs and any immunosuppressive conditions (often overlapping).D: Abscess of the skin, muscle, or connective tissue, but not of any internal organs.E: Including heart valves of suspected infective cardiac device-related endocarditis.F: Including abscess/cyst of peritoneum, kidney, and spleen, but excluding hepatic abscess/cyst which sorts under 'Liver'.G: Including suspected infection of aortic prosthetic implants and of periaortic haematoma.H: Other sites sorting into several small-sized groups including the conjunctiva and corpus vitreum of the eye, outer and middle ear, lymph nodes, skin (not abscess) and nails.-:Omitted to protect patient anonymity due to low numbers.

Table 3 .
Diagnostic accuracy and value of 16S/18S tNGS for specimen types with a minimum of five samples included.Abscess of the skin, muscle or connective tissue, or around orthopaedic implants, but not of any internal organs.E: See supplementary material.Includes several small-sized (n 4) specimens types including brain biopsy, eye swab, corpus vitreum aspirate, swabs/biopsy/lavage of the outer or middle ear, tympanic drain fluid, sputum, mucus from trachea, lung abscess aspirate, lymph node biopsy, skin biopsy and soft tissue biopsies from various sites, nails, wound swab, various prosthetic implants (aortic/orthopaedic/heart valve/pacemaker electrode), haematoma around infected aortic prosthetic implant.
B: Similar definition as for TPs tNGS, with percentages in parenthesis.C: Defined as TP 16S/18S analysis identifying a pathogen not previously identified by other diagnostic tests.D: