Have you ever seen a study claiming that e-cigarettes help people quit smoking, and another one saying the opposite? Nothing surprising. Not all research carries the same weight. Understanding the difference between a cross-sectional study and a randomized controlled trial completely changes how results should be read.
Key takeaways
Not all studies are equal: some only observe, and only one type can prove causality—randomized controlled trials (RCTs).
Cross-sectional and ecological studies show associations, not cause-and-effect links.
Cohort and longitudinal studies track changes over time but remain observational.
RCTs establish causality thanks to randomization and control of external factors.
Systematic reviews and meta-analyses offer the strongest overall picture, but their reliability depends on the studies included.
Table of contents
Observational studies
oCross-sectional studies
oLongitudinal studies
oCohort studies
oEcological studies
Experimental studies
oRandomized controlled trials (RCTs)
Synthesis studies
oSystematic reviews
oMeta-analyses
The hierarchy of evidence
oComplementary methods
Practical guide: how to read a scientific study on e-cigarettes
o5 steps to read a study on vaping
oPotential biases
Ranking of scientific study types by strength
Your questions
Understanding methodologies to better interpret results (and avoid pitfalls)
Scientific research on e-cigarettes advances every day. Yet very few studies are capable of answering certain questions that might seem simple. For example: do e-cigarettes help people quit smoking? Does vaping increase the risk of certain diseases? And many others.
After our article “Can We Still Trust Scientific Studies?”, which highlighted several flaws in the current scientific system, let’s now take a closer look at the different types of research. What are the differences between a cross-sectional study and a longitudinal one? Can an RCT establish causality? Can conclusions from an ecological study be applied at the individual level? Here is an overview of the main types of scientific studies, how they work, their advantages and their limits.
To remain clear and accessible, this article simplifies certain scientific notions. The figures presented are not from real studies: they are made-up examples, used only to illustrate how each research method works.
Observational studies
Observational studies are those in which researchers do not intervene but merely observe what happens. This type of research could be compared to the work of a journalist who observes, takes notes, analyzes, but does not alter events. Among observational studies are cross-sectional, longitudinal, cohort, ecological studies, etc.
Cross-sectional studies
Snapshot of a population at a given moment, like a survey. Participants are questioned only once about their characteristics and health status. Allows measurement of prevalence of a disease or behavior, but impossible to establish a cause-and-effect relationship because the chronology of events is unknown. Quick and inexpensive, but limited to statistical associations.
How it works: a sample of people is surveyed at a given moment: “Are you still smoking? Are you using e-cigarettes? Did you quit thanks to them?”
Results: 40% of respondents who use a vape device have quit smoking, compared to 15% among those who do not use e-cigarettes.
Conclusions: an association can be observed, since there are more ex-smokers among vapers than among non-vapers. But that’s it! We cannot know if vaping actually caused smoking cessation. Maybe these people also used other cessation tools in addition, or maybe they started vaping after quitting to avoid relapse, etc.
Longitudinal studies
Tracking a group of people over time without a predefined structure at the start. Participants evolve naturally: some adopt risk behaviors, others do not. Allows observation of evolution and establishing a temporal link, but cannot prove causality since many confounding factors may influence results. More informative than cross-sectional studies but still observational.
How it works: a group of smokers is followed over time. Some begin vaping, others do not, and researchers observe how many quit after x months or years.
Results: after 12 months, 25% of smokers who started vaping quit smoking, versus 10% of non-vapers.
Conclusions: a temporal link can be established, but still not causality. As with cross-sectional studies, perhaps smokers who started vaping also joined cessation programs, had better medical support, or belonged to social groups more inclined to quit, etc.
Cohort studies
Structured version of a longitudinal study: two groups are deliberately formed at the start (exposed / non-exposed) and followed over time. For example: smokers vs. non-smokers tracked for 10 years to see who develops cancer. More rigorous than a simple longitudinal study, but still cannot eliminate all confounding factors (age, lifestyle, genetics…).
How it works: two cohorts are recruited. Smokers who use e-cigarettes and smokers who do not. Both groups are followed over time to see who quits smoking.
Results: similar to longitudinal study.
Conclusions: also similar. The cohort of vaping smokers may have been younger, more motivated to quit, etc. It still cannot be affirmed that vaping itself caused cessation.
Ecological studies
Compare aggregated data between countries, regions, or time periods. Example: “In countries where chocolate consumption rises, Nobel prizes also increase.” Main pitfall: ecological fallacy. What is true at the population level is not necessarily true at the individual level. High risk of confusing correlation and causation due to massive external factors.
How it works: compare the number of smokers and vapers in several countries over several years.
Results: in Germany, smoking prevalence decreased by 15% between 2010 and 2015. At the same time, the number of vapers increased by 5%.
Conclusions: the decline in smoking alongside a rise in vaping may suggest a link, but once again, causality cannot be demonstrated. Perhaps Germany also raised tobacco taxes, launched prevention campaigns, etc. The drop in smoking cannot directly be attributed to vaping. Just as an increase in ice cream consumption in a country coinciding with more drownings does not mean ice cream causes drownings.
But observing is not always enough. To test a hypothesis, researchers must intervene directly.
Experimental studies
Unlike observational studies, researchers intervene directly with participants. A simple analogy would be a baker who, when preparing a cake, replaces one ingredient with another and watches what happens.
Randomized controlled trials (RCTs)
A randomized controlled trial (RCT) is an experimental study where participants are randomly assigned to two groups: one receiving the intervention being tested (vapes) and one control group receiving an alternative (nicotine patches, for example). This random distribution eliminates confounding factors and allows causality to be established.
How it works: smokers are recruited and then randomly split into two groups. The first receives e-cigarettes, the second nicotine patches.
Results: 20% of smokers given an e-cigarette quit smoking, compared to only 10% of smokers who received patches.
Conclusions: since the study is controlled by researchers, all smokers had equal access to the same care, tools, etc. No external factor could alter the results. It is therefore possible to establish causality. In this case, yes, vaping caused more smoking cessation than nicotine patches.
But RCTs are not perfect. All smokers who received an e-cigarette received the same device, vaped the same liquid, with the same nicotine strength. E-cigarette models evolve very rapidly. RCTs conducted in 2013 are less relevant today, since newer technologies have made current devices more effective than those used back then. For comparison, it would be like comparing the use of two mobile phones: a Nokia 3310 and an iPhone 17.
This is the paradox of RCTs: the more rigorous they are, the less they reflect real life.
Synthesis studies
Synthesis studies involve analysis work. Researchers compile and analyze the results found in other studies.
Systematic reviews
Exhaustive and methodical analysis of all available scientific literature on a specific topic. Follows a strict protocol: searching all databases, predefined selection criteria, critical appraisal of study quality. Offers a global view of scientific consensus, but its quality depends entirely on the studies included. Qualitative rather than quantitative synthesis.
How it works: researchers search medical databases for all studies investigating whether vaping helps people quit smoking.
Results: out of 20 studies, 15 concluded that e-cigarettes help people quit smoking.
Conclusions: the majority of studies show that vaping helps smoking cessation. But this result depends directly on the studies included. Were they high-quality? If biased studies were included, then the systematic review will be biased as well. It is also difficult to merge conclusions of studies with very different methodologies.
Meta-analyses
Mathematically combine the numerical results of several similar studies (ideally RCTs) to obtain a more precise overall outcome. Example: combining 20 studies with 1,000 participants each to reach the statistical power of 20,000 participants. Highest level of scientific evidence because it maximizes robustness, but still dependent on the quality of the included studies.
How it works: 20 comparable studies are selected (ideally RCTs), each reporting how many smokers quit in the vaping group and in the nicotine patch group.
Results: the total number of smokers was 25,000 (1,250 per study). Of these, 3,750 managed to quit. In the vaping group, 18% quit, versus 12% in the patch group—a 6% difference. P < 0.001.
Conclusions: Odds ratio (OR): 1.8 [95% CI: 1.4–2.3], meaning e-cigarettes increase the chances of quitting smoking by 80%. The term “95% CI” means researchers are 95% confident the true OR lies between 1.4 and 2.3 (since you can never be 100% certain).
And finally, P < 0.001 means there is less than 1% chance the result is due to chance. (See our previous article on P-Hacking for more details.)
Meta-analyses are among the most reliable studies, though once again, their conclusions depend directly on the quality of the studies included. The GRADE method, often used in meta-analyses, was developed to assess the overall quality of scientific evidence, from “very low” to “high.” Thus, a meta-analysis including mostly low-quality studies will itself be of low quality.
Meta-analyses, because of the smoothing they apply to data, can also hide subgroup differences. Mixing results from a 2013 RCT using cigalikes with those from a 2024 study using modern pods may also be problematic. And sometimes, meta-analyses create an artificial average that does not correspond to any real-life situation.
The hierarchy of evidence
All the studies described above use different methodologies, each leading to specific types of conclusions. These differences determine their place in the hierarchy of evidence—from the highest level to the weakest:
Meta-analyses – the highest level of evidence
Why the highest?
Quantitative synthesis: statistically combine several randomized trials
Maximum statistical power: 25,000 smokers vs. 1,250 per study
Exceptional precision: OR = 1.8 [95% CI: 1.4–2.3] → narrow range
Statistical certainty: P < 0.001 = less than 0.1% chance it’s random
Strengths:
Eliminates peculiarities of each individual study
Detects modest effects invisible in single studies
Robustness: 20 convergent studies = strong result
Limitations:
Quality depends on studies included
Technological evolution: mixing cigalikes (2013) + pods (2024) = problematic
Artificial average may not reflect real life
Publication bias: negative studies less published
Randomized controlled trials (RCTs) – the only ones able to establish causality
Why level 2?
Only design capable of establishing real causality
Controlled conditions: everyone receives the same care and tools
Randomization: eliminates confounders
Direct comparison: vape vs. patch under identical conditions
Strengths:
Controlled intervention: researchers decide who gets what
Controlled temporality: treatment → effect
External factors neutralized
Limitations:
Artificial conditions: same model, liquid, nicotine strength
Fast technological evolution: RCT from 2013 outdated today
Not real life: standardized use ≠ actual use
Limited generalization: works only under those conditions
Systematic reviews – an exhaustive qualitative synthesis
Why level 3?
Global view of all scientific literature
Rigorous methodology: exhaustive database search
Critical appraisal: assess quality of studies
Transparency: predefined selection criteria
Strengths:
Exhaustive: no relevant study missed
Identifies trends: convergence/divergence
Critical appraisal: separates good from bad studies
Why not higher?
No combined statistical calculation like meta-analyses
Qualitative synthesis → more subjective
Dependent on existing studies: cannot fix flaws
Heterogeneity: difficult to compare very different methodologies
Cohort studies – the best structured observational design
Why level 4?
Clear structure: 2 defined cohorts (vapers vs. non-vapers)
Prospective follow-up: over time
Temporality respected: exposure → effect
Risk calculation possible
Strengths:
More structured than simple longitudinal studies
Groups comparable (in theory)
Avoids some selection bias
Why not ranked higher?
Confounding factors: vaping cohort may be younger, more motivated…
No randomization: self-selection
Observational = cannot prove causality
Expensive and long duration
Longitudinal studies – unstructured time-based tracking
Why level 5?
Tracks over time: chronology established
Temporal link observable: vaping → quitting
Natural evolution: observes real-world behaviors
Less artificial than RCTs
Strengths:
Temporality (unlike cross-sectional)
Natural conditions: no intervention
Can detect complex patterns
Why only level 5?
Less structured than cohort: groups not predefined
Multiple confounders: cessation programs, medical support…
Self-selection bias: vapers may already differ
Cannot establish definitive causality
Ecological studies – simple population-level comparisons
Why level 6?
Macro view: country-level trends
Useful for public policy
Hypothesis generating: finds associations worth testing
Uses existing data: fast and cheap
Strengths:
Population-level scale
Longitudinal data possible: track over years
Inexpensive
Why so low?
Major ecological bias: cannot infer individual behavior
Huge confounders: taxes, prevention campaigns, laws…
Correlation ≠ causation (ice cream/drowning analogy)
Aggregated data: individual detail lost
Cross-sectional studies – weakest level of evidence
Why last?
Snapshot only: no temporality
Association only: no causality
Many possible selection/confounding biases
Strengths:
Very fast: results in weeks
Inexpensive
Good for prevalence: “how many vapers quit?”
Generates hypotheses
Why the weakest?
No temporality: don’t know what came first
Reverse causality possible: those wanting to quit choose vaping
Confounders: cessation programs, support…
Selection bias: who answers surveys?
Complementary methods
While some types of studies may seem more important than others, observational, experimental, and synthesis studies actually complement each other very well.
Observational studies are often the starting point of more advanced research. They allow detection of associations and hypothesis generation. “It seems vapers quit smoking more often.”
Then comes the experimental phase: “Let’s test whether giving smokers a vape actually helps them quit.”
Finally, synthesis studies gather the results of different research efforts, confirm or refute theories, and help guide public policy decisions.
Practical guide: how to read a scientific study on e-cigarettes
5 steps to read a study on vaping
Go to the source: always consult the original study. If results are reported in the media, ignore them! Check the actual study, usually cited in the article.
Check funding (listed at the bottom in the funding section): who paid for the study? Do the authors have industry ties? Does the study align with the funder’s interests? If yes, keep in mind results may be biased.
Identify the type of study: cross-sectional, cohort, RCT, etc.
Check question/method fit: does this type of study answer the research question?
Contextualize findings within existing literature: is it the first to show this? Do other studies confirm it?
Which study for which question?
Prevalence questions: “How many vapers quit smoking?” → Cross-sectional sufficient. Snapshot suited to measure a percentage.
Trend questions: “Is the number of vapers rising in country X?” → Ecological studies using national data.
Risk factor questions: “Does vaping during pregnancy affect the fetus?” → Prospective cohort minimum. Must ensure exposure precedes effect.
Causality questions: “Does vaping cause smoking cessation?” → RCTs only. Only design that eliminates confounders. Exception: if no recent RCT, a high-quality cohort may be acceptable.
Synthesis questions: “What does science say about vaping’s effectiveness?” → Meta-analyses or systematic reviews.
Population-specific: “Does vaping help smokers over 65?” → Sub-analyses of RCTs or targeted cohorts. Avoid extrapolating from general populations.
New products: “Are latest-generation pods effective?” → Recent RCTs prioritized. Old meta-analyses may be irrelevant due to outdated devices.
Warning signs
A cross-sectional study claiming causality or temporal effect.
An RCT using outdated devices.
RCT without proper control (vape vs. nothing instead of vs. another substitute).
Cohort losing >50% of participants without explanation.
Study funded by stakeholders in the market studied.
Generalizing results from a specific group (teens, psychiatric patients, etc.) to the whole population.
Too small a sample size.
Follow-up duration too short.
Potential biases
A bias is an error in reasoning or procedure that risks distorting the results of a clinical study. The error becomes systematic and is not due to chance or random inaccuracies. To illustrate, imagine a photographer who wants to take a picture of the crowd waiting before a concert. Without bias, he would stand in the middle and take his shot. However, if he stands in front of the VIP entrance, he will only photograph well-dressed people. His picture will therefore be biased and not representative of all those present at the event. In a study, the principle is the same: it is a systematic error in the methodology that makes the sample unrepresentative of the target population. Below are the three main biases (though there are many others):
Selection bias: problem in the constitution of the sample, which ends up not being representative of the target population. For example, recruiting participants only in a vape shop. They will necessarily be better informed or more motivated than the general population.
Measurement/classification bias: error in measuring the risk factor or in establishing the presence of the disease. For example, asking vapers if they quit smoking, but not asking patch users. In short, measuring differently depending on the groups.
Confounding bias: presence of a confounding factor, meaning a difference between compared groups other than the treatment tested: age, comorbidities, stage of disease, etc. Example: the vaping group is younger than the nicotine patch group → perhaps age influences the success of that type of aid.
Ranking of scientific study types by strength
Hard to find your way through the jungle of scientific methods? This table gives you an overview: at a glance, the strengths and weaknesses of each type of study, from the strongest to the weakest.
In summary, no single study is perfect on its own. But when placed within the hierarchy of evidence, they form a coherent puzzle.
Your questions
Why don’t all studies on e-cigarettes have the same scientific value?
Because each type of study is based on a different methodology. Some, such as cross-sectional or ecological studies, only observe associations at a given moment or at the population level. They are useful to detect trends but cannot prove a cause-and-effect link. By contrast, randomized controlled trials (RCTs) introduce an intervention and randomly assign participants, which makes it possible to establish real causality. The hierarchy of evidence exists precisely to distinguish the strength of conclusions depending on the method
What does a cross-sectional study provide, and what are its limits?
A cross-sectional study provides a “snapshot” of a population. It is very quick to carry out and inexpensive, which makes it common in public health surveys. For example, it can show how many smokers use e-cigarettes at a given point in time. But it cannot tell us whether vaping caused smoking cessation or, on the contrary, if ex-smokers started vaping afterwards. It is therefore a descriptive tool, not proof of causality.
How is a longitudinal study more informative than a cross-sectional study?
A longitudinal study follows the same participants over months or years. It introduces a temporal dimension, showing whether one behavior precedes another. For example, researchers can observe if smokers who start vaping are more likely to quit than those who don’t. This type of study therefore provides stronger evidence than a simple snapshot. But since researchers do not intervene, uncontrolled external factors (personal motivation, medical support, social context, etc.) still limit the strength of its conclusions.
What do cohort studies demonstrate?
Cohort studies are structured follow-ups: two distinct groups are recruited from the outset (e.g. smokers who vape vs. smokers who don’t), and their evolution is observed over time. They allow relative risks to be calculated and group trajectories to be directly compared. This is one of the best observational designs. But without randomization, it can never be certain that the observed difference is due solely to vaping. Cohort participants may differ in many other respects (age, motivation, health, etc.).
Why are ecological studies useful but limited?
Ecological studies compare aggregate data between countries or regions. They are useful for identifying general trends and informing public policies. For example, observing that smoking declines as vaping increases in a country. But this is a macro level: it is impossible to conclude that an individual quit smoking because they vaped. This is the “ecological fallacy”: we cannot infer what happens at the individual level from statistics that summarize an entire population.
Why are randomized controlled trials (RCTs) considered the gold standard?
RCTs are the only type of study that can prove a cause-and-effect link. By randomly assigning participants to a group receiving the intervention (e.g. an e-cigarette) and a control group (e.g. nicotine patches), the influence of other factors is neutralized. If the results differ, the difference can be attributed to the tested intervention. Their limitation is that conditions are highly controlled and standardized, which may poorly reflect real-world use.
What is the difference between a systematic review and a meta-analysis?
A systematic review provides a critical inventory of all existing studies on a given question. It ensures that no relevant data are left out. A meta-analysis goes further: it mathematically combines the results of several comparable studies to produce a global numerical estimate (for example, the average percentage of smokers who quit thanks to vaping). The former provides the overall picture, the latter provides statistical power.
What is the highest level of evidence in medical science?
Meta-analyses of randomized controlled trials. They combine many independent results, reducing the risk of a single bias skewing conclusions. They provide very high statistical precision and often serve as a reference for health recommendations. But their reliability directly depends on the quality of the studies included and the relevance of the data (e.g. mixing trials using obsolete devices with studies on modern pods can be misleading).
What are the main warning signs when reading a study?
A sample size that is too small, a follow-up that is too short, a high dropout rate among participants, or funding from an actor with a direct interest are all warning signs. Likewise, be cautious with cross-sectional studies that claim to prove causality, or clinical trials conducted with outdated equipment. These do not make a study useless, but they do (sometimes greatly) reduce confidence in its conclusions.
How can a non-specialist assess the reliability of a vaping study?
A few simple habits are enough: check the type of study (a survey does not carry the same weight as an RCT), look at who funded the research, make sure the chosen method matches the research question, and compare the findings with those of other studies. If several solid studies converge, the conclusions are more reliable. If, on the contrary, only one study reports a spectacular result, caution is warranted.