jim.shamlin.com

Chapter 7: Images and Propositions

Recall from the previous chapter that sense data is not stored in its raw form, but is organized into a mental representation that can be compared to present sensation to recognize that it has been encountered previously, but knowledge is a separate form of memory that does not require sensory stimulation to activate.

For example, when you form a mental image of what someone looks like, in their absence, or recall the sound quality of their voice, you are referencing a knowledge representation of a person that is an interpreted form of sense data. You do not literally have an image or a sound in memory, but information that enables your mind to reconstruct it.

Mental Representation of Knowledge

The phenomenon of knowledge can only be indirectly studied. There is to date no method of directly examining the information in the mind of a subject nor is there signs that such a method has even been conceived. We are limited to asking subjects to describe their knowledge representations and accepting the inaccuracies of self-reporting.

There is an overlap between psychology and philosophy, epistemology in specific, in that both apply a rational approach and deductive reasoning to develop a theory of the nature and scope of knowledge. Psychologists generally begin with a philosophical approach and then seek to design experiments to obtain empirical evidence that supports their theories, but the evidence is as yet scant and oblique.

Two kinds of knowledge structures are declarative knowledge (the ability to state facts) and procedural knowledge (the ability to do things, based on implicit understanding). Both are leveraged in the learning process - understanding how a car is driven and knowing how to drive a car are two different things, but each can be gleaned from the other.

External Representations

it is generally considered that knowledge consists of words and pictures, and the method used to convey information depends on the nature of the information. The easiest way to covey a concept such as "pigeon" is simply to draw an image of one, but you cannot draw a picture of "justice," as it is an immaterial concept best described in words.

Neither images nor words are a perfect means of conveying information, and they tend to focus on the most obvious distinguishing characteristics of a thing or a concept. This is the notion of symbolic representation: the difference between a thing as we understand it and the thing as we describe it (and presumably, the thing as we understand it) is that knowledge is a symbolic representation of the thing itself.

Moreover, symbolic representation is highly idiosyncratic. Ask a dozen or a hundred people to draw a picture of "a cat under a table" and no two will draw exactly the same picture, though there will be certain features in common. Likewise, ask the same number of people to describe the concept of "fairness" and they will each describe it in their own way.

The idiosyncrasies of symbolic representation stem from the unique and subjective view people take of the world they perceive. The functional application of a "knife" (as a cooking tool, a cutting tool, a survival tool, or a weapon) shape the way that a person understands the object - and asked to draw or describe a knife, a soldier and a chef will have different representations. The emotional portent likewise influences which observable features are emphasized - a person who has experienced a traumatic incident in which he was bitten by a dog has a different representation of the animal than a person who has not.

Back to the distinction: pictures aptly capture physical and spatial information, whereas words more flexibly capture abstract and categorical information. And the way in which a given concept is illustrated or described reflects the knowledge and experience of the subject.

Mental Imagery

As a basic definition, "imagery" pertains to the mental representation of things that are not currently in sensory perception. Our notion of a person's face, the sound of their voice, or their scent in the absence of that person, is a mental image.

As such we are said to be "imagining" things when we must reference mental imagery to bring a concept to our conscious mind. More specifically, it is accessing the symbolic representation of things we have experienced - the defining details that distinguish it from other things.

Most research into this area focuses on visual stimuli - the ability to draw or describe in visual terms an object that is not visible in the present environment. A distinction is drawn between the general representation of a concept (a woman) versus the recollection of a specific instance (your mother). It's also suggested that human beings have a preference for vision - subjects asked to describe a scene focus mostly on the visual elements, and it is rare other sensory elements are included at all.

Imagery is often used to solve problems and answer questions - to build something requires us first to imagine what it should be when the task is completed, and to solve any problem means imagining the future state in which the problem has been solved. This requires projecting our imaginations into a future state that does not yet exist, but which we wish to effect.

It's noted that not everyone is equally facile in creating mental images, and correspondingly there are individuals who excel or fail to varying degrees to achieve outcomes in the real world: it is not the ability to hold a brush that differentiates a bad artist from a great one, but the ability to imagine the painting that this action, among others, will bring into existence.

There is no realistic way to assess the extent to which mental images are maintained. By an extreme theories, all images of everything we ever sense, whether we are conscious of sensing it are not, are stored in granular detail and with perfect fidelity in our memory. More realistically, however, it is doubtful that even my current estimations that the human brain has such infinite storage capacity - and while it can hold am amazing amount, it is not infinite in its fidelity, scope, or duration.

Dual-Code Theory

Dual-code theory (Pavivo) maintains that subjects use a combination of visual and verbal elements in forming symbolic representations.

Visual data is considered to be analogue code, a form of representation that preserves the main features of whatever is being represented - in essence, an analogue code is a sketchy outline or stick-figure representation containing only as much detail as necessary to have a sense of the thing we have perceived.

Verbal date, meanwhile is a symbolic code. That is, our conception of a principle such as "fairness" has a the most basic possible definition, and the various details by which we assess whether a given arrangement is "fair" are filigree that are closely related to, but not contained within, the basic definition of the principle.

One key difference that has been supported by research is that verbal data is most often recalled, and most accurately recalled, in a linear fashion: the words we have heard are in the order in which they were heard. Meanwhile, order seems to be entirely arbitrary for cataloging visual data: an image is perceived as a unit, all at once, and the order of sequence of images is arbitrary in terms of recollection.

Neuroscience further supports this theorem, demonstrating that the different parts of the brain that are active when images or words are perceived are also active when they are recalled.

It's also been observed (Brooks 1968) that perceptual data from the environment commingles with and interferes with recollection of knowledge stored in memory - which can be readily observed in everyday life when a person who is recalling a phrase from memory is disrupted by words that are spoken in the present environment, but the level of interference is less evident when a visual stimulus is presented during verbal recollection or vice-versa.

Conceptual-Propositional Theory

The propositional theory provides an alternative to the dual-code theory, which suggests that the way in which the mind handles sensory data is more abstract than words and images - these are merely the ways in which mental representations are expressed. In long-term memory, we store the concepts of things rather than the sensory data of actual things

There is some finnickiness in the way people describe the relationships among things. There are multiple ways to describe things and the relationship between them (the same physical relationship is stated by indicating "your nose is above your mouth" and "your mouth is below your nose") and the way in which you consider them gives focus to one or the other. Essentially, the mind "proposes" this relationship.

By this theory, it is our understanding of the relationship that shapes our knowledge, hence this relationship is more significant that the qualities of the things we describe. The author describes four kinds of relationships:

The arbitrary and sometimes incorrect choice of proposition can be witnessed in the way that things are conceptualized. Consider that a hexagram may be perceived ass a single unit, or seen as two overlapping triangles, or six triangles radiating from a hexagon, or three pairs of parallel lines, or as a religious symbol.

Some "optical illusions" present us with ambiguous figures - a sketch may resemble a rabbit or a duck (depending on whether you see the protrusions as ears or a rather rounded beak), or an old or young woman (depending on whether the line on one side is perceived as a cheek or a rather distorted nose). The simple practice of cloud-gazing can lead to many interpretation of what a given formation "looks like" to a specific person.

This is not about the sensory perception, but about the way that the data we perceive visually is matched against our conceptions: the ambiguous rabbit-duck drawing will look like a rabbit to someone who has never seen a duck, because the absence of "duck" from memory means that it cannot be proposed as a solution. To those who have seen both, the result will depend on the stronger representation in memory, on whether the ambiguity is biased to one interpretation over the other, or some combination of both factors.

The creation of propositions, like perception itself, is also influence by gestalt. We recognize certain defining features of an object or class of objects, but at the same time recognize that a real-world object has many additional qualities that are not considered to be defining characteristics. We more readily process both sensation and concept on the broadest level. A capital "E" is recognizes as such quickly, and it is only with some thinking that we consider that the shape might be a capital "F" superimposed on a capital "L."

So to be more specific about the way information is conceptualized, consider that we use propositions to determine what it might be, but the choice of proposition is guided by "hints" we notice:

It's also to be noted that the mind comes to expect further ambiguity once ambiguity has been witnessed. Shown a sequence of ambiguous drawings, a subject's interpretation becomes more deliberated before he noticed the ambiguity. Up to that point, he is not expecting ambiguity, but afterwards he begins to look for ambiguity in the rest.

Not everyone in the field considers the propositional theory to hold, and there is some indication that the responses of participants may be more to do with perception than memory, but there is evidence to support the theory as well as contradict it.

Mental Manipulation of Images

The functional equivalence hypothesis holds that, which visual imagery is not identical to perception, it is treated as functionally equivalent. That is to say that the mention of the word elephant conjures a mental image of an elephant that does not correspond to the sense data we have taken from any specific real-world elephant - but the mental image we have is handled in the same way as a sense memory.

Five distinct principles have been derived (Finke 1989):

  1. Our mental images and concepts are similar to physical objects and percepts in terms of their properties and behaviors
  2. The spatial relationships among mental imagery is similar to and follows the same basic rules as spatial relationships in physical reality
  3. Mental images may also include information that was not perceived in physical reality and exclude some information that was
  4. The construction of a mental image is similar to drawing: the drawing represents the physical object in many ways but is not a perfect representation
  5. Mental imagery is utilized in the same way as sensory perception in cognitive processes.

Mental Rotations

Mental rotations of objects are effected by the imagination: when we perceive a two-dimensional image, or even a three-dimensional object from a single angle, the mind is able to conceive what the appearance of the object would be if it were viewed from a different angle.

An often-repeated experiment is a visual test that presents the subject with a two-dimensional image of an object, and asks him to indicate whether a second image is or is not the same shape that has been rotated.

The response time is directly correlated to the degree to which an object had been rotated - i.e., less than two seconds if the object had been rotated less than 40 degrees, but more than four when the object was rotated 140 degrees or more - and this is regardless of whether the subject's response was correct or incorrect.

Response time, as well as accuracy, was also impacted according to the complexity and familiarity of the shape. For example, people can quickly and accurately respond when the object is a letter or numeral that has been rotated, as well as when the object is geometrically simple, such as a cube or a box shape.

It is also noted that the practice effect became evident in these experiments: that is, the speed and accuracy of responses were better on later puzzles than on the initial ones as their minds settled into the task of performing mental rotation.

The difference between individual participants' response times is proposed to be linked to their intelligence, though this quickly degenerates into a tautology (they are considered to be more intelligent because they are faster and more accurate at the task).

Some preliminary research indicates a neurological link between mental rotation and the visual and motor cortexes within the brain - in essence, reinforcing the notion that the parts of the brain that are used for performing tasks and interpreting sense data are likewise engaged when the exercise is purely imaginary.

While this indicates the same mechanisms may be used, the question remains as to whether the same processes are used - there is some indirect evidence that suggests that it is so, but not sufficient to sway consensus distinctly to one or the other conclusion.

Image Scaling

Aside of the shape, color, and texture of objects we perceive, there is also the matter of their size. In general, we have a predilection of larger objects: when describing an environment, we relate the larger objects first, consider them to be of greater importance. We perceive larger objects to be closer to ourselves, and give them greater attention.

While size is fairly easy to assess when objects are perceived in relation to one another, it is more difficult to assess how we perceive an object to be in itself. There is the definite sense that things ought to be of a certain size, and that a given object I unusually large or small compared to our mental representation of it.

The details of a given object (how large an elephant's ears should be) are fairly easy to observe because the parts are seen in relation to the whole. We can immediately recognize its ears are 10% larger than they ought to be, but do not have the same facility in recognizing whether the entire object is slightly disproportionate - shown a mouse and a rabbit in the same picture, we do not immediately notice (or notice at all) that the mouse is 10% larger or rabbit is 10% smaller than it ought to be - but if the mouse's head is 10% larger than it ought to be, it seems immediately peculiar, provided we have an existing mental representation of both.

Image Scanning

Everything within our field of vision, whether an image or a real environment, is unconsciously perceived at a glance, but only as raw sense data without recognition. We recognize things by scanning - moving our gaze around the environment and separating out individual items for the purpose of recognizing them and testing them against memory.

The author refers to a study in which subjects were shown a depiction of several objects and asked to press a button when they saw a specific object, on the assumption that the length of time it took to press the button corresponded to its order in which subjects gave attention to various items in the image.

Results suggested that when objects are depicted in a line, horizontally and vertically, subjects are naturally inclined to scan them in a given order (top to bottom, left to right) - but this is mediated by the preference for certain objects (larger instead of smaller, brighter instead of paler, etc.) such that we will start with the object that is most effective at drawing attention and scan the remainder relative to it.

(EN: The shortcomings of to the study methodology are fairly obvious, but later experimentation with eye-tracking systems support the evidence of less sophisticated studies.)

Synthesizing Images and Propositions

The dual-code theory and propositional theory are distinctly at odds with one another. There is evidence for and against each and various attempts have been made to develop a synthesis of the two.

It's noted that experiment design has an influence on the results, specifically in the demands placed on the test subjects. That is to say that depending on the design of a given experiment, participants leverage certain mental skills, such that when the results support the hypothesis, it means that the participants demonstrated certain behavior that seems to support the notion that a given skill exists.

However, it is also significant to note the things that an experiment does not prove:

Essentially, an experiment presents a demand to the participant, and the meeting this demand by whatever means necessary becomes the objective of the participant. This is even true in a blind experiment in which the participant does not know what the research is meant to prove - he will substitute his own assumptions and act accordingly. At the very least, participants are given some level of instruction in most experiments: they have an idea of what they are meant to do, and it should be little surprise when they find a way to do so.

In sum, the reason that evidence exists to support two opposing theories is that the experiment for each theory was designed in such a manner as to gather supporting evidence.

Johnson-Laird Mental Models

A third alternative to images and propositions is that of mental models that reflect an individual's implicit theories of knowledge. A mental model is in essence a theory, which may be supported by observational evidence, that defines a concept and its associated qualities and features. A person who has never encountered an object is capable of forming a mental model of it based on a text description.

For example, consider the mental model of the Earth. Very few people have been into space and had the ability to regard the planet with their own eyes, but instead rely on descriptions and images provide by others to develop a mental model of something they have never personally observed. The most common model of the shape of the earth is primarily as a sphere, but it is based on no more evidence than the previous model of the earth as being flat. It is entirely conceptual and based on belief rather than experience.

In a similar way, when a subject interprets the phrase "The cat is under the table," he has not seen a cat or a table, and refers to his mental model. In effect, he imagines a specific cat underneath a specific table (images) with the cat in a specific posture and the table in a specific angle (propositions).

Experimental evidence suggests that most instances in which a subject has fabricated details, the details are drawn from their mental model to substitute for gaps in their memory or information that was withheld from the description they are given. It is not an intentional act of embellishment or distortion, merely the mind's attempt to gather sufficient detail to imagine or remember - even if certain details must be fabricated to do so.

The author then refers to a series of experiments (Kerr 1983) done with blind patients - the born blind who have never had visual capabilities - such that their mental models are not tainted by experience or their immediate perception. While the blind subjects responded more slowly than sighted ones, their responses showed the same kinds of disparities; they were similar in the latency of describing elaborate versus simple scenarios; they seemed focused on larger versus smaller objects in a scenario; etc. Granted, while they did not "see" they still exercised perception, both in terms of sense memory and the verbal description given by the proctor.

Additional testing into auditory compositions (Intons 1992) showed similar distinctions. Given a sound clip in which multiple distinct sounds were combined, the intensity of sound (loud/soft) substitutes for visual perspective (large/small); sounds at the same level of volume and pitch were more difficult to distinguish; etc. such that it becomes evident that the findings related to visual stimuli are also evident in auditory stimuli.

There's a stray note that our mental models govern much of our thinking, and that a faulty mental model (one that does not correspond to demonstrable reality) leads to faulty conclusions. Consider that children and adults alike have incorrect concepts: they think that heat and cold flow through air and over objects much as smoke driven on a breeze; they think that plants obtain all their food from the soil in which they grow; they think that an object dropped from an airplane will fall perpendicular to the ground, and so on.

Neuropsychological Evidence for Multiple Codes

Neuropsychology proposes to address the problem of bias in experience, based on the principle that a subject can alter his behavior, but cannot alter the functioning of his brain as an organ.

(EN: An interesting premise, but the problem is that the significance of electrochemical activity is clearly enough understood and must be interpreted - and this biases not only the outcome, but the recording process. Also, the brain is not like other organs in that its functioning has to do with behavior - such that if a subject consciously modifies his behavior, speech, or even his thoughts, this will in turn modify the activity in his brain, and researchers are still misled.)

Some researchers look to deformed brains - those with injuries or lesions in specific locations - to suggest the functioning of the mind. This is valued because it is assumed that the damage is the cause of cognitive anomalies, and that the subject has neither knowledge nor control of the anomalies that should result from a particular kind of damage.

In specific, it is noted that lesions in certain areas of the brain seem to affect symbol-manipulation functions such as language, whereas others influence information storage and retrieval, such as is necessary to recognize faces. From such studies we have evidence that the right hemisphere is associated to visual memory and perception whereas the left is associated with verbal memory and comprehension.

It's also suggested (Corballis 1989) that the left side has the ability to manipulate imagined concepts and symbols and to generate entirely new information, whereas the right is limited to representing knowledge in a manner that is consistent with the physical environment. Specifically, it is believed to be a distinctly human, and distinctly left-brained characteristic, to conceive what has never been perceived.

Visual and Spatial Images

The author revisits the notion that visual and spatial imagery may be represented differently in the brain - the difference being that visual imagery corresponds to the qualities of a perceived item (its color and shape) and is processed in the occipital lobe, whereas spatial imagery corresponds to its relation to other things (size, position, and distance as well as qualities pertaining to its position relative to a light source such as shadows and textures).

The case of an anonymous patient is considered, a man who suffered an accident at the age of 18 that resulted in lesions in the occipital lobe. He did not lose his ability to see, but lost his ability to describe and remember information pertaining to color and shape. He meanwhile showed relatively normal abilities in rotation, mental scanning, scaling, and relative locations.

This patient, along with others whose limitations have been linked to injury to specific parts of the brain, suggest that the various tasks involved in "seeing" are quite disparate and are handled by various parts of the brain. That is the ability to perceive, the ability to remember what was perceived, the ability to comprehend what is perceived, and the ability to imagine what is not perceived, are distinctly different capabilities.

Spatial Cognition and Cognitive Maps

It is generally considered that mental maps represent a mental process of comparing sense data against an internal (imagined) representation, which requires the application of intelligence and judgment to determine how the models are constructed and how sense data is collocated.

But there is also the theory that mental maps are created without an intentional process - merely by moving though space and encountering objects, we created models based on our perception, in a subjective manner: everything is experienced in reference to oneself, and each person's knowledge is formed by his own limited perception.

"Cognitive Maps" is the term used for the internal representation of a physical environment, particularly focusing on the spatial relationships among the things we perceive.

In the 1920s, Edward Tolman conducted the rats-in-mazes series of experiments, which observed that reward was essential to learning: i.e., rats who were rewarded (with food) for completing a maze learned to get thought it without making errors, those who received no reward merely explored the environment, and that when a reward was offered to rats that had explored the maze without an incentive, they quickly began to learn to run the course.

His methodology was the essence of cognitive mapping: considering experience in a sequential manner, in which a series of twelve right/left decisions are memorized in order to achieve a goal.

Frisch (1962) observed the same of bees, who memorized the location of food sources relative to the location of the hive, and the ability to "dance" to communicate the map to other members of the hive, such that even bees that had not been rewarded can follow the directions to the goal.

It has generally been observed that humans use the same basic processes to gain knowledge of environments. Three specific types of knowledge are leveraged:

  1. Landmark Knowledge - Having a sense of the location of a goal relative to present position, which enables us to move in a straight line
  2. Route Knowledge - Having a sense of a specific path from present to desired location, which enables us to travel around obstacles following a specific path
  3. Survey Knowledge - Having a sense of the lay of the land, including the position of various obstacles, which enables us to travel toward the goal without a specific path

Succeeding at getting from A to B may be the result of having multiple routes to the goal and choosing among them, or it may be the result of having survey knowledge to plot a course that changes dynamically. Or, as is often observed, we may have both facilities and change between them in any given situation.

Mental Shortcuts

The combination in landmark, route, and survey knowledge of a region work in combination to create "mental shortcuts" in our estimation of distance. Generally, the more landmarks that we know in a region, the larger we believe the region to be and the greater the distance we believe to exist between them.

This has been observed not only in the real experience of being in a place, but also evidences itself when looking at maps: a region with many roads and townships is assumed to be larger than a different region that covers more land, but has fewer landmarks. And when demarking cities on a map, those that are most familiar are placed closer together.

A number of common distortions are described:

An experiment (Saarinen 1987) that asked participants to draw a world map on a blank sheet of paper demonstrated biases in the representation of geography. Most individuals placed their own country in the center of the map (curiously, Asians were less likely to do this and centered on Europe), increased the size of familiar locations and decreased the size of unfamiliar ones, and distorted distances between them. The effects were more pronounced when asked to draw the maps upside-down (inverting north and south poles). The same kinds of distortions were observed in mapping states in a country, or even drawing a city map.

(EN: IN all fairness, a "map of the world" includes many distortions because a spherical area is flattened out, but the example shown in the book demonstrates even further distortion than would arise from this factor alone.)

A stray note: the ability to navigate physical spaces, and to develop the ability to accurately depict the terrain, is a basic survival skill - in some locations, it is more critical than others (nomadic desert tribes must recall the locations of water sources on a largely featureless terrain). It is by necessity we must be able to do this accurately.

There is also a gender bias that supports the man-hunter woman-gatherer stereotype, in that men are more accurate at long-distance navigation whereas women tend to remember more landmarks.

(EN: This notion of distance has broader applications: While a "map" is most literally tied to one of a geographic area, a "map" is also a metaphor that describes the relationships between immaterial things, such as two terms, and mental frameworks consider the similarity or dissimilarity of concepts in the same terms as physical differences. That is, "like" is said to be closer to "love" than it is to "hate" in the same way Baltimore is closer to New York than it is to Chicago. In this sense, our idea of the relative sizes and positions of landmarks in a geography reflects similar bias in the relation of concepts within the realm of experience.)

Text Maps

The bias evident in diagramming locations is also reflected in verbal or written description of environments.

(EN: More detail is provided, and while I find this plausible, the detail itself causes it to seem more questionable - whether it is measuring he perception of size and distance or merely reflecting the subject's ability to describe spatial relationships in words, or even the researcher's ability to interpret what the subject wrote or said, undermines the hypothesis. My sense is that this is too great of a distraction to accept the findings.)

Development of Visuospatial Skills

Young children develop spatial orientation in a geometric manner as they learn to orient themselves in a given environment, with a demonstrable preference for pure forms - a perfect square, a perfect triangle, etc. such that objects are represented as being equal in their distance from one another.

Experiments done with children of varying ages show their ability to recognize common symbols (letters and numbers that have been rotated) increases with age. A separate experiment with aged patients (ages 55-71) also showed a decrease in this facility.

There is some suggestion that the difference in speed is one of familiarity with the symbols, such that younger children take longer to recognize a shape rather than to recognize its rotation. The two factors may be inseparable.

It's noted that both children and adults tend to increase speed an accuracy of recognition during an experiment, suggesting that visuospatial skills may be learned behaviors rather than natural inclinations. While it stands to reason that recognizing things, regardless of the angle at which they are seen, is a practical skill, the degree to which it develops in nature may be difficult to assess by means of a laboratory exercise, as skills are developed in performing the task in a laboratory setting in a way they are not otherwise exercised.

JIM.SHAMLIN.COM
DISCLAIMERS · CONTACT