CSEG3060 · Research Methodology in Computer Science

Approaches to Literature Review

Systematic, narrative, scoping and meta-analytical methods

Given my research question, which review method can legitimately answer it?

Unit III · Lecture 2 of 8 · 60 minutes · CO2

Dr. Mohsin Furkh Dar · School of Computer Science, UPES

→ / Space next · ← back · O overview · N speaker notes · F full screen

Learning objectives · CO2

By the end of this session, you can…

01

Define and distinguish the four approaches: systematic, narrative, scoping, meta-analytical

02

Identify the methodological steps each approach requires

03

Evaluate which approach suits a given CS research question

04

Apply reporting standards — PRISMA, PRISMA-S, PRISMA-ScR

05

Critically appraise published reviews for methodological rigour

CSEG3060 · Unit III · Lecture 2 — Approaches to Literature Review2

Foundations

A review is not a summary — it is a method

A literature review is a systematic, replicable and transparent synthesis of existing scholarly contributions within a defined domain.

Your choice of approach determines…

  • The breadth and depth of evidence included
  • The reproducibility of search and synthesis
  • The type of inference permitted — qualitative or quantitative
  • The defensibility of claims under peer review

What every review must do

  • Maps the intellectual terrain
  • Identifies gaps, contradictions, open questions
  • Justifies the contribution of new work
  • Grounds hypotheses theoretically and empirically

Why this matters in computing

CS research output roughly doubles every five years. Expert intuition alone can no longer guarantee coverage — which is why protocolised methods migrated into the discipline.

CSEG3060 · Unit III · Lecture 2 — Approaches to Literature Review3

Foundations

Protocols, question formats and reporting standards

The review protocol

A pre-specified plan stating the research question, inclusion/exclusion criteria, search strategy, data-extraction method and synthesis approach. The primary defence against post-hoc bias.

Question formats

PICO — Population, Intervention, Comparison, Outcome · systematic reviews and meta-analyses

PEO — Population, Exposure, Outcome · qualitative phenomena

PCC — Population, Concept, Context · scoping reviews

PRISMA

Systematic reviews and meta-analyses

PRISMA-S

Search reporting

PRISMA-ScR

Scoping reviews

Protocol ≠ reporting standard. A protocol governs what you will do; a reporting standard governs what you must disclose. PRISMA compliance does not make a weak review strong — it makes a weak review visible.
CSEG3060 · Unit III · Lecture 2 — Approaches to Literature Review4

Approach 1 of 4

Systematic Literature Review: the methodological anatomy

A review using explicit, reproducible methods to collate and synthesise all studies meeting pre-defined eligibility criteria, in answer to a specific, bounded question.
Protocol
→
Search
→
Screen
→
Appraise
→
Extract
→
Report

Pre-registered protocol

Published or institutionally archived before screening begins, fixing criteria in advance.

Exhaustive search

IEEE Xplore, ACM DL, Scopus, Web of Science, ScienceDirect, plus backward and forward citation chasing.

Screen and appraise

Two-phase screening; quality appraisal of every included study using validated instruments (JBI, CASP).

Extract and report

Standardised extraction forms; transparent reporting via a PRISMA flow diagram.

In computing the canonical procedural reference is Kitchenham & Charters (2007), Guidelines for performing Systematic Literature Reviews in Software Engineering — itself the product of a systematic review across seven digital libraries.

CSEG3060 · Unit III · Lecture 2 — Approaches to Literature Review5

Approach 1 of 4

The PRISMA flow, and the SLR trade-off

Identification
→
Screening
→
Eligibility
→
Included
Ne = Ni − Nd,s − Nd,e − Nd,r   with   Ne ≤ Ni

Worked counts: 3,142 identified → 2,801 excluded on title/abstract → 294 excluded at full text → 47 included. Every exclusion at eligibility must carry a stated reason.

Strengths

  • Reproducible and transparent
  • Materially reduces selection bias
  • Licenses evidence-based conclusions

Limitations

  • Time- and resource-intensive: 6–24 months
  • May exclude valuable grey literature — arXiv, technical reports
  • Quality appraisal contested in CS: designs are heterogeneous
CSEG3060 · Unit III · Lecture 2 — Approaches to Literature Review6

Approach 2 of 4

Narrative review: interpretive synthesis and expert judgement

A comprehensive, discursive synthesis without rigidly defined search protocols — interpretive rather than aggregative.

Methodological profile

  • Sources selected on the author's expertise and judgement
  • No registered protocol
  • Thematic or chronological organisation
  • Quality appraisal informal or absent
  • Aims at breadth, contextualisation, theoretical framing

Editorial

Short, opinion-driven

Critical

Evaluative, argumentative stance

State-of-the-art

Current developments and future directions

Where it appears in CS: survey articles in ACM Computing Surveys; tutorials in IEEE Transactions introducing an emerging sub-field; the framing chapter of a thesis.

Strengths

Flexible · theoretically deep · viable for emerging topics · 2–8 weeks

Limitations

Susceptible to selection bias · hard to replicate · often methodologically opaque

CSEG3060 · Unit III · Lecture 2 — Approaches to Literature Review7

Approach 3 of 4

Scoping review: mapping a heterogeneous field

Maps the key concepts, types of evidence and gaps in an area — the method of choice when a topic is broad, heterogeneous or not previously reviewed. Arksey & O'Malley (2005), refined by the Joanna Briggs Institute.
1 Identify the question
→
2 Identify studies
→
3 Select studies
→
4 Chart the data
→
5 Collate and report

Optional stage 6 — stakeholder consultation. Practitioners routinely identify grey literature and deployment concerns absent from indexed venues.

Question framing

PCC — Population/Participants · Concept · Context. Reported under PRISMA-ScR.

Strengths

  • Broad, exploratory questions
  • Maps heterogeneity across a field
  • Identifies gaps for future SLRs · 3–6 months

Limitations

  • Does not appraise quality of included studies
  • Findings are less definitive
  • Can blur conceptually into a narrative review
CSEG3060 · Unit III · Lecture 2 — Approaches to Literature Review8

Approach 4 of 4 · part 1

Meta-analysis: quantitative synthesis and effect sizes

A quantitative synthesis that statistically combines results from independent studies to produce a pooled effect estimate. A systematic review is its precondition, not its alternative.

Continuous outcomes — standardised mean difference

SMD = (X̄1 − X̄2) / spooled
spooled = √[ ((n1−1)s1² + (n2−1)s2²) / (n1+n2−2) ]

Binary outcomes — odds ratio from a 2×2 table

OR = (a · d) / (b · c)

Feasibility condition. Studies must report commensurable outcomes with dispersion statistics. Incompatible metrics make pooling illegitimate, not merely difficult.

Applications in CS: empirical software engineering (pair programming and defect density) · educational data mining (intelligent tutoring) · HCI (interface design and task-completion time).

CSEG3060 · Unit III · Lecture 2 — Approaches to Literature Review9

Approach 4 of 4 · part 2

Fixed-effect and random-effects pooling

Fixed effect

Assumes a single true effect; all divergence is sampling error.

θ̂FE = Σ wiθ̂i / Σ wi    wi = 1 / Var(θ̂i)

Random effects

Assumes true effects vary; adds between-study variance τ² to every weight.

wiRE = 1 / (Var(θ̂i) + τ²)
τ̂² = max( 0 , (Q − (k−1)) / C )    C = Σwi − Σwi² / Σwi  (DerSimonian–Laird)

The estimand differs

Fixed effect asks what is the effect? Random effects asks what is the average of a distribution of effects?

Model selection rule

CS studies differ in datasets, hardware, hyperparameters and participants — so random effects is the appropriate default. Fixed effect under real heterogeneity gives intervals that are too narrow.

CSEG3060 · Unit III · Lecture 2 — Approaches to Literature Review10

Approach 4 of 4 · part 3

Heterogeneity and publication bias

Q = Σ wi (θ̂i − θ̂FE)²
I² = max( 0 , (Q − (k−1)) / Q ) × 100%

Interpretation bands

I² < 25%  ·  low heterogeneity

25% ≤ I² < 75%  ·  moderate

I² ≥ 75%  ·  high — demands subgroup analysis or meta-regression, not a louder pooled claim

Publication bias

The preferential publication of statistically significant results. Diagnosed by funnel-plot asymmetry; tested formally by Egger's regression.

t = b / SE(b)

A symmetric funnel: precise studies cluster near the pooled estimate, imprecise ones fan out below. A missing lower-left corner indicates small negative studies that were run but never published.

CS-specific pressure: benchmark culture rewards improvements over null results, so negative findings on a popular architecture are systematically under-published.

CSEG3060 · Unit III · Lecture 2 — Approaches to Literature Review11

Choosing an approach

Four approaches across eight dimensions

DimensionSystematicNarrativeScopingMeta-analysis
Research questionSpecific, focusedBroad, contextualBroad, exploratorySpecific, quantitative
ProtocolPre-specifiedOptionalOptionalPre-specified
Search strategyExhaustiveSelectiveComprehensive, iterativeExhaustive
Quality appraisalMandatoryInformalNot requiredMandatory
SynthesisQualitative or quantitativeQualitative (thematic)Descriptive mappingQuantitative pooling
ReproducibilityHighLowModerateHigh
OutputEvidence-based conclusionsConceptual synthesisConceptual mapPooled effect size
Typical duration6–24 months2–8 weeks3–6 months3–12 months
Decision heuristic. Quantitative pooling feasible and metrics compatible → meta-analysis. Precise question needing exhaustive, reproducible synthesis → systematic. Mapping an emerging or heterogeneous field → scoping. Conceptual or theoretical framing → narrative.

Fastest diagnostic pair: Is the question broad or precise? Are the outcome metrics commensurable? These two answers isolate the correct method almost every time.

CSEG3060 · Unit III · Lecture 2 — Approaches to Literature Review12

Practical application

Executing an SLR in computer science — and where bias enters

StepWhat you doBias risk
1–2 Question and protocolPICO/PEO; fix strings, databases, criteria, appraisal toolsA question so narrow it guarantees a pre-decided answer
3–4 Search and screenIEEE Xplore, ACM DL, Scopus, ScienceDirect, Scholar; two-phase screenSingle-database reliance; over-restrictive terms; unjustified date limits
5–6 Extract and appraiseStandardised forms; JBI or CASP checklists, two reviewersExtraction drift; weak studies treated as equal evidence
7–8 Synthesise and reportThematic synthesis, pooling where legitimate, PRISMA 2020Selective citation; silent method reclassification

Boolean search string

(term1 ∨ term2) ∧ (term3) ∧ ¬(exclusion term)

Seven pitfalls

  1. Review written as an annotated bibliography
  2. No clear research question
  3. Search bias
  4. Absent quality appraisal
  5. Selective citation
  6. Misclassified review type
  7. Unjustified publication window
CSEG3060 · Unit III · Lecture 2 — Approaches to Literature Review13

Case study

One question, four review lenses

“What is the effectiveness of deep learning models in detecting zero-day network intrusions?”
ApproachWhat you actually doWhat you legitimately obtain
NarrativeSynthesise seminal deep-learning IDS work by expert judgementConceptual framing; dominant architectures identified
SystematicExhaustive PRISMA-governed search across CS databasesReproducible evidence across 47 primary studies
ScopingMap all ML applications across cybersecurity subdomainsGaps surfaced — adversarial robustness in real deployments
Meta-analysisPool F1-scores and detection rates across comparable studiesQuantitative estimate of average improvement

Complementary, not competing

A mature programme uses them in sequence: scoping to locate a defensible gap → systematic to answer it rigorously → meta-analysis where metrics permit pooling.

Feasibility caution

Pooling F1 across NSL-KDD, CICIDS2017 and a proprietary trace invites exactly the heterogeneity that inflates I². Report it, stratify by dataset, and resist a single triumphant number.

CSEG3060 · Unit III · Lecture 2 — Approaches to Literature Review14

Consolidation

What to retain, and what the examiner asks

Four takeaways

  • Systematic — reproducible, protocol-driven; gold standard for a bounded question
  • Narrative — flexible, theoretically deep; suits emerging topics
  • Scoping — maps heterogeneous literatures, exposes gaps
  • Meta-analysis — pools results, assesses heterogeneity and bias

Exam focus

  1. Master the comparative table — especially quality appraisal and reproducibility
  2. Memorise PRISMA stages and the Arksey & O'Malley five(+1) stages
  3. Contrast fixed-effect vs random-effects; interpret I² conceptually
  4. Know PRISMA vs PRISMA-ScR and their purposes
  5. Apply the decision heuristic to unseen research questions
  6. Practise critiquing a published review for rigour

Numerical practice. Three studies: d = 0.45 (SE 0.10), d = 0.62 (SE 0.15), d = 0.38 (SE 0.08). Compute fixed-effect weights wi = 1/SEi², the pooled effect, Q and I², then interpret the heterogeneity.

Method follows from the research question, the available evidence and the intended inference — never from convenience or prestige.

1 / 15

Slides — click to jump Esc to close

Speaker notes