No solo existen palabras en los textos escritos: algunas teorías y modelos de comprensión de textos multimodales o multisemióticos

Giovanni Parodi, Cristóbal Julio


Words have been the most representative system of human language and they have captured historically the focus of attention given to communication. In this vein, an important group of researchers have addressed systematically the investigation of a number of variables related to the comprehension of words, given the intrinsic complexities of the verbal system (e.g., van Dijk & Kintsch, 1983; De Vega, Carreiras, Gutiérrez-Calvo & Alonso-Quecuty, 1990; Kintsch, 1998, 2013; De Vega & Cuetos, 1999; Parodi, 2014; Parodi, Peronard & Ibáñez, 2010). Theories and models of language comprehension offer a number of important developments and debates (van Dijk & Kintsch, 1983; Kintsch, 1998; De Vega, Graesser & Glenberg, 2008; Alvermann, Unrau & Ruddell, 2013; Parodi, 2014). However, there have comparatively less advances in the study of multiple semiotic systems that contribute to the construction of textual meanings. Words together with other semiotic systems, as constitutive elements of written texts, have only recently begun to be the focus of study. Thus, in an effort to contribute to the reflection on the understanding of texts composed by diverse systems (among others, words, charts, diagrams, graphics), this article carries out a critical review of four theories of multisemiotic text comprehension. Special attention is paid to the understanding of static texts with words and graphics, and two particular proposals are also reviewed. The paper is closed with a set of reflections and projections for this research area.

Texto completo:



Acartürk, C., Habel, C., Cagiltay, K. & Alacam, O. (2008). Multimodal comprehension of language and graphics: Graphs with and without annotations. Journal of Eye Movement Research, 1(3), 1-15.

Ainsworth, S. (2006). DeFT: A conceptual framework for considering learning with multiple representations. Learning and Instruction, 16, 183-198.

Alvermann, D., Unrau, N. & Ruddell, R. (Eds.) (2013). Theoretical models and processes of reading. Newark, NJ: IRA.

Andrá, C., Lindström, P., Arzarello, F., Holmqvist, K., Robutti, O. & Sabena, C. (2013). Reading mathematics representations: An Eye-Tracking Study. International Journal of Science and Mathematics Education, 13(2), 237–259.

Benveniste, E. (1977). Problemas de lingüística general. Madrid: Siglo Veintiuno Editores.

Bateman, J. (2014). Text and image: A critical introduction to the visual/verbal divide. London: Routledge.

Bertin, J. (1983). Semiology of graphics: Diagrams networks maps. Madison, WI: University of Wisconsin Press.

Bondi, M. (1995). World of Fact and world of hypothesis in Economics textbooks. En V. Negri Zamagni (Ed.), Annale 1994/1995, Università degli Studi di Bologna, Facoltà di Economia, Sede di Forlì (pp. 341-354). Cesena: Società Editrice Il Ponte Vecchio.

Bondi, M. (1999). English across genres: Language variation in the discourse of Economics. Modena: Il Fiorino.

Boudon, E. & Parodi, G. (2014). Artefactos multisemióticos y discurso académico de la Economía: Construcción de conocimientos en el género Manual. Revista Signos. Estudios de Lingüística, 47(85),164-195.

Brünken, R., Steinbacher, S., Plass, J. & Leutner, D. (2002). Assessment of cognitive load in multimedia learning using dual-task methodology. Experimental Psychology, 49(2),109-119.

Canham, M., & Hegarty, M. (2010). Effects of knowledge and display design on comprehension of complex graphics. Learning and Instruction, 20(2), 155-166.

Carpenter, P. & Shah, P. (1998). A model of the perceptual and conceptual processes in graph comprehension. Journal of Experimental Psychology: Applied, 4(2), 75-100.

Chandler, P. & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8, 293-332.

Dehaene, S. (2010). Reading in the brain: The new science of how we read. New York: Penguin Books.

Dehaene, S. (2011). The number sense: How the mind creates mathematics. New York: Oxford University Press.

De Vega, M. & Cuetos, F. (Coords.) (1999). Psicolingüística del Español. Madrid: Trotta.

De Vega, M., Carreiras, M., Gutiérrez-Calvo, M. & Alonso-Quecuty, M. (1990). Lectura y comprensión: una perspectiva cognitiva. Madrid: Alianza.

De Vega, M., Glenberg, A. & Graesser, A. (Eds.) (2008). Symbols and embodiment: Debates on meaning and cognition. Oxford: Oxford University Press.

Gidlof, K., Holmberg, N. & Sandberg, H. (2012). The use of eye-tracking and retrospective interviews to study teenagers’ exposure to online advertising. Visual Communication, 11(3), 329–345.

Goldberg, J. & Helfman, J. (2011). Eye tracking for visualization evaluation: Reading values on linear versus radial graphs. Information Visualization, 10(3), 182–195.

Habel, C. & Acartürk, C. (2007). On reciprocal improvement in multimodal generation: Co-reference by text and information graphics. En I. van der Sluis, M. Theune, E. Reiter & E. Krahmer (Eds.). Proceedings of the Workshop on Multimodal Output Generation: MOG 2007 (69-80). University of Aberdeen, U.K.

Hiippala, T. (2012). The interface between rhetoric and layout in multimodal artefacts. Literary and Linguistic Computing, 28(3), 461-471.

Holsanova, S., Holmberg, N. & Holmqvist, K. (2009). Reading information graphics: the role of spatial contiguity and dual attentional guidance. Applied Cognitive Psychology, 23(9), 1215-1226.

Julio, C. (2016). Congruencia entre el sistema verbal y el sistema gráfico: experimento de lectura en textos multisemióticos con el uso de en la técnica de eyetracking. Tesis para optar al grado de licenciado en lengua y literatura hispánica. Pontifica Universidad Católica de Valparaíso.

Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge: Cambridge University Press.

Kintsch, W. (2013). Revisiting the Construction-Integration Model of text comprehension and its implications for instruction. En D. Alvermann, N. Unrau & R. Ruddell (Eds.), Theoretical Models and Processes of Reading (pp. 807-839). Newark, NJ: IRA.

Koch, C. (2012). Consciousness: Confessions of a romantic reductionist. Cambridge, MA: MIT Press.

Lemke, J. (1998). Multiplying meaning: Visual and verbal semiotics in scientific text. En J.R. Martin & R. Veel (Eds.), Reading Science (pp. 87-113). London: Routledge.

Liu, Y. & O’Halloran, K. (2009). Intersemiotic texture: Analyzing cohesive devices between language and images. Social Semiotics, 19(4), 367-388.

Louwerse, M. (2010). Symbolic or embodied representations. A case of symbol interdependency. Topics in Cognitive Science, 1-30.

Manghi, D. (2013). Géneros en la enseñanza escolar: Configuraciones de significado en clases de historia y biología desde una perspectiva multimodal. Revista Signos. Estudios de Lingüística, 46(82), 236-257.

Martin, J. (2012). Multimodal semiotics: Theoretical challenges. En S. Dreyfus, S. Hood, M. Stenglin (Eds.), Semiotic Margins: Meaning in Multimodalites (pp. 243-270). London: Continuum.

Mason, L., Tornatora, M. C. & Pluchino, P. (2013). Do fourth graders integrate text and picture in processing and learning from an illustrated science text? Evidence from eye-movement patterns. Computers & Education, 60(1), 95–109.

Mayer, R. (2005). Cognitive theory of multimedia learning. En R. Mayer (Ed.), The Cambridge Handbook of Multimedia Learning (pp. 31-48). Cambridge: Cambridge University Press.

Mayer, R. (2009). Multimedia learning. Cambridge: Cambridge University Press.

Mayer, R. (2011). Applying the science of learning to multimedia instruction. En J. Mestre & B. Ross (Eds.), The psychology of learning and motivation (pp. 77-108). New York: Academic Press.

McNamara, D. (Ed.) (2007). Reading comprehension strategies: Theories, interventions, and technologies. Mahwah, NJ: Erlbaum.

McNamara, D. & Magliano, J. (2009). Toward a comprehensive model of comprehension. En Ross, B. (Ed.), The psychology of learning and motivation: advances in research and theory (pp. 297-372). Illinois: Academic Press.

Moya, J. & Parodi, G. (2017). ¿Existe influencia del sistema verbal en la ‘jugabilidad’ de un videojuego?: registro de movimientos oculares con eyetracker. Revista Círculo de Lingüística Aplicada a la Comunicación, 69, 276-305.

O’Halloran, K. (2006). Mathematical discourse. Language, symbolism and visual images. London: Continuum.

Parodi, G. (2003). Relaciones entre lectura y escritura: una perspectiva cognitiva discursiva. Antecedentes teóricos y resultados empíricos. Valparaíso: EUV.

Parodi, G. (2010). Multisemiosis y lingüística de corpus: Artefactos (multi)semióticos en los textos de seis disciplinas en el corpus PUCV-2010. Revista de Lingüística Teórica y Aplicada, 48(2), 33-70.

Parodi, G. (2014). Comprensión de textos escritos. La Teoría de la Comunicabilidad. Buenos Aires: Eudeba.

Parodi, G. (2015). Variation across university genres in seven disciplines: A corpus-based study on academic written Spanish. International Journal of Corpus Linguistics, 20(4), 469-499.

Parodi, G. & Burdiles, G. (Eds.) (2015). Leer y escribir en contextos académicos y profesionales: géneros, corpus y métodos. Santiago de Chile: Ariel.

Parodi, G. & Julio, C. (2015). Más allá de las palabras: ¿Puede comprenderse el género discursivo Informe de Política Monetaria desde un único sistema semiótico predominante? Revista ALPHA, 41, 133-158.

Parodi, G. & Julio, C. (2016). ¿Dónde se posan los ojos al leer textos multisemióticos? Procesamiento de palabras y gráficos en un estudio experimental con Eye Tracking. Revista Signos. Estudios de Lingüística, 49, supl.1, 149-183.

Parodi, G., Boudon, E. & Julio, C. (2014). La organización retórica del género Manual de Economía: Un discurso en tránsito disciplinar. Revista de Lingüística Teórica y Aplicada (RLA), 52(2), 133-163.

Parodi, G., Peronard, M. & Ibáñez, R. (2010). Saber leer. Madrid: Aguilar.

Paivio, A. (1971). Imagery and verbal processes. New York.

Paivio, A. (1986). Mental representation: A Dual Coding Approach. New York: Oxford University Press.

Paivio, A. (2007). Mind and its evolution: A Dual Coding Theoretical Approach. Mahwah, NJ: Erlbaum.

Peronard, M., Gómez, L., Parodi, G. & Núñez, P. (1998). Comprensión de textos escritos: De la teoría a la sala de clases. Santiago: Editorial Andrés Bello.

Pinker, S. (Ed.). (1985) Visual cognition. Cambridge, MA: MIT Press/Bradford Books.

Pinker, S. (1990). A theory of graph comprehension. En R. Freedle (Ed.), Artificial intelligence and the future of testing (pp. 73-126). Hillsdale, NJ: Erlbaum.

Plass, J. Moreno, R. & Brünken, R. (Eds.) (2010). Cognitive load theory. Cambridge: Cambridge University Press.

Radach, R. & Kennedy, A. (2013). Eye movements in reading: Some theoretical context. The Quarterly Journal of Experimental Psychology, 66, 429–452.

Rayner, K., Ashby, J., Pollatsek, A. & Reichle, E. (2004). The effects of frequency and predictability on eye fixations in reading: implications for the E-Z Reader model. Journal of Experimental Psychology & Human Perception Performance, 30(4), 720-32.

Rau, M., Michaelis, J. & Fay, N. (2015). Connection making between multiple graphical representations: A multi-methods approach for domain-specific grounding of an intelligent tutoring system for chemistry. Computer & Education, 82, 460-485.

Ratwani R., Trafton J. & Boehm-Davis D. (2008). Thinking graphically: Connecting vision and cognition during graph comprehension. Journal of Experimental Psychology: Applied, 14(1), 36-49.

Reichle, E. & Drieghe, D. (2015). Using E-Z Reader to examine the consequences of fixation-location measurement error. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41, 262–270.

Reichle, E., Pollatseck, A., Fischer, D. & Rayner, K. (1998). Toward a model of eye movement control in reading. Psychological Review, 105, 125-157.

Royce, T. (1999). Visual-verbal intersemiotic complementarity in the Economist magazine. Tesis doctoral, The University of Reading, Reading: Great Britain.

Rummer, R., Schweppe, J., Fürstenberg, A., Scheiter, K. & Zindler, A. (2011). The perceptual basis of the modality effect in multimedia learning. Journal of Experimental Psychology: Applied, 17, 159-173.

Sadoski, M. (1992). Imagination, cognition, and persona. Rhetoric Review, 10, 266-278.

Sadoski, M. & Paivio, A. (2001). Imagery and text: A dual coding theory of reading and writing. Mahwah, NJ: Erlbaum.

Sadoski, M., & Paivio, A. (2004). A dual coding theoretical model of reading. En R. R. Ruddell & N. J. Unrau, (Eds.), Theoretical models and processes of reading (pp. 1329–1362). Newark, DE: IRA.

Sadoski, M. & Paivio, A. (2007). Toward a unified theory of reading. Scientific studies of reading, 11, 337-356.

Sadoski, M., Paivio, A. & Goetz, E. (1991). A critique of schema theory in reading and a dual coding alternative. Reading Research Quarterly, 26, 463–484.

Sadoski, M., & Willson, V. (2006). Effects of a theoretically based large-scale reading intervention in a multicultural urban school district. American Educational Research Journal, 43, 137–154.

Sadoski, M., Willson, V., Holcomb, A. & Boulware-Gooden, R. (2005). Verbal and nonverbal predictors of spelling performance. Journal of Literacy Research, 36, 461–478.

Samuels, W. (Ed.) (1990). Economics as discourse. An analysis of the language of Economics. London: Kluwer.

Sánchez, M. (1993). Los textos expositivos. estrategias para mejorar su comprensión. Madrid: Santillana.

Segers, E., Verhoeven, L. & Hulstijn-Hendrikse, N. (2008). Cognitive processes in children's multimedia text learning. Applied Cognitive Psychology, 22, 375–387

Schnotz, W. (2002). Towards an integrated view of learning from text and visual displays. Educational Psychology Review, 14(2), 101-120.

Schnotz, W. (2005). An integrated model of text and picture comprehension. En R. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 49-69). Cambridge: Cambridge University Press.

Schnotz, W. & Bannert, M. (2003). Construction and interference in learning from multiple representations. Learning and Instruction 13, 141-156.

Schnotz, W., Bannert, W. & Seufert, T. (2002). Towards an integrative view of text and picture comprehension: Visualization effects on the construction of mental models. En A. Graesser, J. Otero & J. A. León (Eds.), The psychology of science text comprehension (pp. 385-416). Mahwah, NJ: Erlbaum.

Schnotz, W. & Horz, H. (2010). New media, learning from. En E. Baker, P. Peterson & B. McGaw (Eds.), International Encyclopedia of Education (pp. 140-149). New York: Elsevier.

Schüler, A., Arndt, J. & Scheiter K. (2015). Processing multimedia material: Does integration of text and pictures result in a single or two interconnected mental representations? Learning and Instruction, 35, 62-72.

Shah, P. (1997). A model of the cognitive and perceptual processes in graphical display comprehension. AAAI Technical Report, 94-101.

Shah, P. & Freedman, E. G. (2011). Bar and Line Graph Comprehension: An interaction of top‐down and bottom‐up processes. Topics in Cognitive Science, 3(3), 560-578.

Shah, P., Mayer, R. & Hegarty, M. (1999). Graphs as aids to knowledge construction: Signaling techniques for guiding the process of graph comprehension. Journal of Educational Psychology, 91(4), 690-702.

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12: 257-285.

Sweller, J. (1999). Instructional design in technical areas. Camberwell, Australia: ACER Press.

Sweller, J. (2003). Evolution of human cognitive architecture. En B. Ross (Ed.), The psychology of learning and motivation (pp. 215-216). San Diego, CA: Academic Press.

Sweller, J. (2005). Implications of Cognitive Load Theory for multimedia learning. En R. Mayer (Ed.), The Cambridge Handbook of Multimedia Learning (pp. 19-30). Cambridge: Cambridge University Press.

Sweller, J., Ayres, P. & Kalyuga, S. (Eds.) (2011). Cognitive load theory. New York: Springer.

Taboada, M. & Habel, C. (2013). Rhetorical relations in multimodal documents. Discourse Studies, 15(1), 65-89.

Tufte, E. (1990). Envisioning information. Graphics Press, Cheshire, Connecticut.

Tufte, E. (1997). Visual explanation. Graphics Press, Cheshire, Connecticut.

Tufte, E. (2001). The visual display of quantitative information. Graphics Press, Cheshire, Connecticut.

van Essen, D. & Dieker, D. (2007). Surface-based and probabilistic atlases of primate cerebral cortex. Neuron, 56, 209-224.

van Dijk, T. & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic.

van Leeuwen, T. (2005). Typographic meaning. Visual Communication, 4(2), 137-143.

van Leeuwen, T. (2006). Towards a semiotics of typography. Information Design Journal + Document Design, 4(2): 139-155.

van Leeuwen, T. (2011). The language of colour. An introduction. London: Routledge. van Gog, T. & Scheiter, K. (2010). Eye tracking as a tool to study and enhance multimedia learning. Learning and Instruction, 20, 95–99.

Viramonte, M., Peronard, M., Gómez Macker, L. Carullo, A. & Velásquez, M. (2000). Comprensión lectora. Dificultades estratégicas en resolución de preguntas inferenciales. Argentina: Ediciones Colihue.

Wilkinson, L. (2005). The grammar of graphics. New York: Springer-Verlag.

Winn, W. (1994). Contributions of perceptual and cognitive processes to the comprehension of graphics. En W. Schontz & R. W. Kulhay (Eds.), Comprehension of graphics (pp. 3-28). Amsterdam, North-Holland: Elsevier.

Zwaan, R. (2008). Experiential traces and mental simulations in language comprehension. En M. De Vega, A. Glenberg & A. Graesser (Eds.), Symbols and embodiment: Debates on meaning and cognition (pp-165-180).

Oxford: Oxford University Press.

Zacks, J. & Tversky, B. (1999). Bars and lines: A study of graphic communication. Memory and Cognition, 27(6), 1073-1079.

Enlaces refback

  • No hay ningún enlace refback.

Copyright (c) 2017 Investigaciones Sobre Lectura


Además, está indexada en los catálogos UBUCAT (Universidad de Burgos), Universidad Nebrija, Universidad de La Rioja, CRAI (Universidad Pablo de Olavide), Universitat de Barcelona, Universidad de Navarra, Linceo+ (UNED), Universitat de Girona, AlcorZe (Universidad de Zaragoza), Universidad Autónoma de Madrid, Universidad de Alicante, y Universidad de Granada

Investigaciones sobre Lectura (ISL)

ISSN 2340-8685


Información de contacto: