1. Learning Objectives

1. Distinguish clearly between identifying a research problem and investigating solutions to it.
2. Enumerate the principal investigative approaches adopted in computer science research.
3. Match a given research problem to an appropriate approach using decision criteria such as the nature of evidence required, feasibility and validity.
4. Recognise how each approach shapes data collection, instrumentation and analysis choices.
5. Critically evaluate the strengths and limitations of each approach in the context of computing artefacts.
6. Justify, in writing, the selection of an investigative approach for a sample problem.

2. Key Terminology

3. The Eight Investigative Approaches

These are not mutually exclusive — contemporary CS research commonly combines them as mixed-methods investigations. Nonetheless every design privileges one or two as the primary engine of evidence.

3.1 Experimental Investigation

Definition. Manipulates one or more independent variables, controls extraneous factors, and measures the effect on dependent variables using quantitative metrics. This is the dominant approach for evaluating algorithms, systems and software products under controlled conditions.

1. Hypothesis statement — H₀: no difference; H₁: a statistically significant difference exists.
2. Variable identification — independent (algorithm variant), dependent (runtime), controlled (hardware, dataset).
3. Experimental design — randomised block, factorial, Latin square or repeated-measures.
4. Measurement — calibrated instruments: profilers, hardware counters, benchmark suites.
5. Statistical analysis — t-test, ANOVA or non-parametric equivalents, reporting effect size and confidence intervals, not just p-values.

3.2 Analytical / Theoretical Investigation

Definition. Relies on logical and mathematical reasoning to derive properties of constructs without empirical measurement, producing proofs, complexity bounds or formal specifications. It fits algorithm analysis, formal verification, cryptography, complexity theory and theoretical machine learning.

1. Statement of claim — a theorem, conjecture or formal property.
2. Assumption set — the pre-conditions under which the claim holds.
3. Reasoning apparatus — lemmas, proofs by induction, contradiction or construction.
4. Bound derivation — worst-case, average-case or amortised complexity, expressed asymptotically.
5. Counter-example or proof — either disproves the claim or establishes it.

Example: proving that a proposed cache replacement policy is starvation-free using an invariant-based argument over the request queue. Strengths: definitive within the assumption set, immune to measurement noise, foundational. Limitations: results may not predict real-world performance, and the formal model may not reflect reality.

3.3 Simulation-Based Investigation

Definition. Builds a computational model of a system and runs experiments on the model when the real system is inaccessible, unsafe or too costly to operate at scale. It fits network protocol evaluation, distributed systems, large-scale ML pipelines, urban computing and epidemic modelling on graphs.

1. Model formulation — identify entities, attributes and behaviours.
2. Calibration — tune parameters against historical or pilot data.
3. Validation — confirm the simulator's output matches reference behaviour (face validity, predictive validity).
4. Scenario execution — run synthetic scenarios varying key parameters.
5. Sensitivity analysis — examine how output changes under input perturbation.

Example: using a discrete-event simulator such as ns-3 to evaluate congestion-control behaviour across 10,000 simulated TCP flows, infeasible on a physical testbed. Strengths: scalable, repeatable, safe for edge cases. Limitations: garbage in, garbage out — fidelity is bounded by the model, and there is publication bias toward optimistic results.

3.4 Survey Investigation

Definition. Cross-sectionally captures the attitudes, practices or self-reported behaviours of a population sample, typically through questionnaires or interviews. It fits software engineering practice, developer experience, cybersecurity awareness and paradigm adoption.

1. Population and sampling — define the target population; choose probability or non-probability sampling.
2. Instrument design — question construction, pilot testing and validation; Cronbach's alpha of α ≥ 0.7 is typically regarded as acceptable.
3. Data collection — online, telephone or in-person distribution.
4. Analysis — descriptive statistics, factor analysis, regression, or thematic coding for open-ended items.
5. Reporting — response rate, confidence intervals and discussion of non-response bias.

Example: a study of how Indian IT firms adopted DevOps practices between 2018 and 2024, surveying 250 project managers. Strengths: captures breadth; useful for phenomena not directly observable. Limitations: self-report bias, and the cross-sectional nature rules out strong causal claims.

3.5 Case Study Investigation

Definition. Investigates a contemporary phenomenon in its real-world context, relying on multiple sources of evidence, when the boundaries between phenomenon and context are unclear. It fits industry pilots, large-scale deployment studies and post-mortem analyses of system failures. Following Yin's protocol:

1. Case definition — single holistic, single embedded, multiple holistic or multiple embedded.
2. Case selection — literal, theoretical or extreme-case replication logic.
3. Data triangulation among interviews, documents, direct observation and physical artefacts.
4. Pattern matching between empirical findings and theoretical propositions.
5. Case report with thick description.

Example: investigating the failure of a multi-region cloud deployment by interviewing SREs, reviewing incident logs and reproducing the failure in a sandbox. Strengths: ecological validity, preserves context, supports theory-building from data. Limitations: limited generalisability, researcher subjectivity, difficulty of replication.

3.6 Design Science (Constructive) Investigation

Definition. Design science research creates and evaluates innovative IT artefacts intended to solve identified problems; the artefact may be a construct, model, method or instantiation. It fits HCI prototyping, novel algorithm design, new language features and system architectures.

The canonical cycle is build → evaluate → reflect → iterate: construct an artefact, observe its utility in a realistic setting, refine it, and repeat. This maps onto the Hevner–Chatterjee framework of three cycles — the relevance cycle (problem), the rigour cycle (theory base) and the design cycle (build–evaluate).

Example: designing a privacy-preserving federated learning aggregator, instantiating it as a prototype, and demonstrating utility on three benchmark datasets. Strengths: concrete contribution, immediate applicability. Limitations: requires both engineering and evaluation rigour, with a risk of conflating innovation with validation.

3.7 Comparative / Benchmarking Investigation

Definition. Positions a new solution against established baselines on shared workloads and metrics to demonstrate relative merit. It fits ML model evaluation, database engine comparison and framework-versus-framework studies.

1. Selection of baselines — state-of-the-art and credible baselines, not strawmen.
2. Standardised workloads — public datasets with documented characteristics.
3. Common evaluation protocol — identical hardware budget, identical preprocessing.
4. Multiple metrics — beyond accuracy alone: latency, fairness, energy.
5. Statistical rigour — multiple seeds, paired tests, reported variance.

Limitations: risk of benchmark overfitting and metric gaming.

3.8 Mixed-Methods Investigation

Definition. Strategically combines qualitative and quantitative strands so the strengths of one offset the weaknesses of the other. Increasingly the norm for industry–academic collaborations. Creswell and Plano Clark identify four major designs:

• Convergent parallel — both strands run independently, then merge.
• Explanatory sequential — quantitative first, qualitative deepens the findings.
• Exploratory sequential — qualitative first, quantitative generalises.
• Embedded / nested — one strand plays a supportive role inside the other.

Example: quantitatively benchmarking a new IDE refactoring plugin, followed by qualitative interviews with five developers to interpret usage patterns. Strengths: triangulation, richer findings. Limitations: demands dual competence and longer timelines.

4. Choosing an Approach

Additional pragmatic filters:

• Resource envelope — compute budget, time, human participants.
• Skill mix — statistical literacy, proof-writing capability, fieldwork access.
• Ethical clearance — human-subject protocols, data privacy.
• Publication venue norms — theory venues favour analytical work; systems venues favour experimental and design-science work; software engineering venues favour case studies and surveys.

5. Instrumentation Across Approaches

Instrumentation is the operational expression of measurement, and it is tightly coupled to the approach.

6. Worked Examples

7. Common Errors in Choosing an Approach

1. Tool-led selection. Picking a survey because Qualtrics is easy, rather than because a survey answers the research question.
2. Single-method tunnel vision. Treating one approach as universally applicable to all sub-questions.
3. Confusing method with approach. Running a questionnaire is a method; the approach decides what kind of evidence is sought.
4. Ignoring ecological validity. A clean lab experiment that fails to reflect deployment realities.
5. Neglecting venue rigour expectations. A theory paper without proofs; a systems paper without statistical comparison.
6. Over-claiming beyond the approach. Drawing causal conclusions from cross-sectional survey data.
7. No instrumentation validation. Trusting an ad-hoc timer or an unverified rubric.

8. How the Approach Cascades Downstream

The investigative approach is not a stand-alone decision. It determines:

• Data collection design — what to record, at what granularity, over what time horizon.
• Data analysis plan — descriptive, inferential, qualitative coding, or hybrid.
• Interpretation strategy — confirmatory, exploratory or theory-driven.
• Reporting format — replication package, open dataset, qualitative codebook.

9. Summary

Investigative approaches in computer science are plural by necessity. Experimental, analytical, simulation-based, survey, case study, design science, comparative and mixed-methods investigations each contribute a distinct evidentiary logic. The rigorous researcher does not ask which approach is best but which approach is justified for this problem, given these constraints, to support these claims.

Selection criteria include the nature of the construct, accessibility of the system, causality requirements, resource envelope and venue expectations. The chosen approach then orchestrates downstream decisions in data collection, instrumentation, analysis and interpretation — the subjects of Lectures 6 and 7.

10. Exam Preparation

10.1 Short-Answer Questions

1. Differentiate between investigative approach and research method, with a computing example. 5 marks
2. Under what conditions would a simulation-based investigation be preferable to an experimental one? Illustrate from networking or distributed systems. 6 marks
3. Explain the role of the relevance, rigour and design cycles in the Hevner–Chatterjee design science framework. 6 marks
4. Why are cross-sectional surveys unable, on their own, to establish causality in software-engineering research? 5 marks

10.2 Medium-Answer Questions

1. Compare and contrast experimental and case-study investigations along six dimensions of your choice. Conclude with the kind of problem best suited to each. 10 marks
2. Describe the principal threats to validity in a controlled experiment measuring the runtime of a new database index, and propose mitigations. 10 marks
3. For the problem “How do Indian product-based startups evaluate the cost-effectiveness of migrating monoliths to microservices?”, propose an investigative approach and justify it with reference to validity, resources and venue expectations. 10 marks

10.3 Long-Answer and Project Questions

1. Design a mixed-methods investigation for a research problem of your choice. Include research questions, an approach diagram, an instrumentation plan, a sampling strategy, an analysis plan, threats to validity and a reproducibility statement. 20 marks
2. Critically review a published CS paper for the appropriateness of its investigative approach. Highlight mismatches, missing evidence and recommended revisions. 15 marks
3. Construct a decision rubric — table or flowchart — mapping a research problem to an investigative approach, and apply it to three diverse problems drawn from algorithmic, systems and HCI sub-fields. 20 marks

11. Suggested Readings

• Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012). Experimentation in Software Engineering. Springer.
• Yin, R. K. (2018). Case Study Research and Applications: Design and Methods (6th ed.). SAGE.
• Hevner, A., & Chatterjee, S. (2020). Design Science Research in Information Systems. In Design Research in Information Systems. Springer.
• Creswell, J. W., & Plano Clark, V. L. (2018). Designing and Conducting Mixed Methods Research (3rd ed.). SAGE.
• Kitchenham, B., & Charters, S. (2007). Guidelines for Performing Systematic Literature Reviews in Software Engineering. EBSE Technical Report.
• ACM Code of Ethics and Professional Conduct (current edition).