Some thoughts about mapping studies

Systematic Mapping are intended to be used to define the coverage of a research field¹, and recommended for PhD candidates to form a better understanding of the potential topic or topics for their thesis², specially when they do not know well the particular area.

I must admit the idea is good. When you are intending to do some research on a given area, and when there is just a fuzzy idea of the orientation, having some methodology to tackle the issue is definitely a plus.

Even more, I’d say it is a good idea even if you believe you know the area. It will force you to find and review, even if superficially, existing papers that already did something. Or maybe didn’t managed to do anything, but that will be, nevertheless, useful too.

There is a “but”. Kitchenham³ reports three significant problems about systematic reviews, but I think this applies to mappings as well. I cite literally:

1. Digital libraries are not well-suited to complex automated searches (mentioned five times). In addition the lack of stan- dardized keywords was mentioned twice.
2. The time and effort needed for SRs (mentioned four times). In addition the time taken for protocol construction was men- tioned twice.
3. The problem of quality assessment of papers based on different research methods (mentioned four times).

I can say I deeply relate to 1 and 2. If the area to be researches happens to have very well defined and specific keywords, good for you. Otherwise, you will be flooded by tons of irrelevant papers, which will make finding the relevant ones harder.

For instance, I searched about “queries” and “raw data”. Hardly any other specific keywords. More than 5000 results I got from the digital libraries, of which the relevant ones are a minority. Even worse, some previously known relevant papers didn’t show up, but adding more keywords would have likely increase the number of results.

In summary, terrible signal/noise ratio.

Later on, Kitchenham³ considers this change appropiate:

To remove the proposal for constructing structured questions and using them to construct search strings. It does not work for mapping studies and appears to be of limited value to SRs in general since it leads to very complex search strings that need to be adapted for each digital library.”

From my inexperience, I do agree. Not only that, it will give you a lot of noise.

Either to include more advice on mapping studies or produce a separate set of guidelines for mapping studies.

Pretty please 🙃

To mention the use of citation-based search strategies (i.e. snowballing).

Gotcha. More papers to the list, but well, that will solve blind spots.

It goes on, but mainly those are my main gripes. Don’t get me wrong, I definitely think it is an interesting and worth technique. It just can be painful sometimes.

Conclusion

In summary, Systematic Mappings are nice to get an idea on who is working on what, and to get familiar to deal with references and paper search.

However, if you are unfortunate enough not to have clear and well defined keywords, to have an orthogonal research area, or to be trying to enter a very covered (I am not saying well) field, you will have to dig through tons of papers.

Disclaimer

I am quite green, and just halfway through the systematic mapping. Take everything I say with a pinch of salt.