Mapping Image Properties into Shape Constraints: Skewed Symmetry, Affine-Transformable Patterns, and the Shape-from-Texture Paradigm

In this paper we demonstrate two new approaches to deriving three-dimensional surface orientation information (“shape”) from two-dimensional image cues. The two approaches are the method of affine-transformable patterns and the shape-from-texture paradigm. They are introduced by a specific application common to both: the concept of skewed symmetry. Skewed sym-nietry is shown to constrain the relationship of observed distortions in a known object regularity to ;I sniall subset of possible underlying surface orientations. Iksidcs this constraint. valuable in its own right, the two methods are shown to generate other surface constraints as well. Some applications are presented of skewed symmetry to line drawing analysis, to the use of gravity in shape under-\tdnding. and t o glotxil h i p c recovery.

We begin with the exploration of how one specific image property, "skewed syninietry."can be defined and formulated to serve as a cue to the determination of surface orientations.Then we will distuss the issue from two new, broader viewpoints.One is the class of affine-transformable patterns.It has various interesting properties, and includes skewed symmetry as a special case.The other is the computational paradigm of shape-from-texture.Skewed symmetry is derived in a second, independent way, as an instance of the application of the paradigm.Also, it is proven that the same skewed-symmetry constraint can arise from greatly different image conditions.This paper further claims that the ideas and techniques presented here are applicable to many other properties under a general framework of the shapefrom-texture paradigm with the underlying meta-heuristic of non-accidental image properties.

Skewed Symmetry
to image, and a knowledge of the gradient space (see Mackworth, 1973).
In this section we assume the standard orthographic projection from scene

Definition, Assumption and Constraints
Symmetry in a 2-D picture has an axis for which the opposite sides are reflective; in other words, the symmetrical properties are found along the transverse lines perpendicular to the symmetry axis.The concept skeumi sjmmefry was introduced by Kanade (1979) by relaxing this condition a little.It means a class of 2-D shapes in which the symmetry is found along lines not necessarily perpendicular to the axis, but at a fixed angle to it.Fornially, such shapes can be defined as 2-D affine transforms of real symmetries.Figures lac show a few exaniples.' Stevens ( 1980) presents a number of psychological experiments which suggest that hunian observers can perceive surface orientations from figures \+,ith this propen!..This is probably because such qualitative ~y i i i i i w [ I !~ in thc image is often due to real symmetry in the scene.Thus let us associate thc following assumption with this image property: A skewed symmetry depicts a real symmetry viewed f'roiii soiiic unknown view angle.
Notc that thc con\ crse of this assumption is always true undcr o i 1 h o ~r i q d ~i ~~ pIojection.
We can transform this assumption into constraints in tlic gi-adicnt spacc.As shown in Figure 1 , a skewed symmetry defines two directions: Ict 11s call them the skewed-symmetry axis and the skewed-transverse axis, and dcnotc 'The mouse hole example of Figure IC is due to K. Stevens (1980).The 3-11 vectors on the plane corresponding to the directions a and p are (cosa,sina, -p cosa-9 sina) and (cosp,sinp,-p cosp-9 sinp).
The ahsunipt ion demands that these two vectors be perpendicular; their inner prcduc-1 vmishcs: c o s ( a -~) + @ c o s a + 9 s i n a ) ( p c o s ~+ 9 s i n ~) = 0. (1) By rotating the p -9 coordinates into the p '-4 ' coordinates so that the new p 'q .I\c\ .uc the bisectors of the an& nude by the skewed symmetry and A c u d -t r m h v c r s e axes, it is easy to show that Thus, the (I,, 4 ) ' s are on the hyperbola shown in Figure 2.That is, the skewed symmetry defined by a and p in the picture can be a projection of a real symmetry if and only if the gradient is on this hyperbola.The skewed symmetry c 240 Tahco K.;;inxic.;riid Jotin K. Kendcr thus imposes a one-dimensional family of constraints on the underlying surface orientation ( p ,4).As we will see in Section 5. other constraints can be exploited for the unique determination of surface orientation.
The tips or vertices GT and GT of the hyperbola represent special orientations with interesting properties.First, since they are closest to the origin of the gradient space, and since the distance from the origin to a gradient represents the magnitude of the surface slant, GT and GT correspond to the least slanted orientations that can produce the skewed symmetry in the picture from a real symmetry in the scene.
Second, since they are on the line (the axis of the hyperbola) which bisects the obtuse angle made by OL and p, they correspond to the orientations for which the rates of depth change along the directions of OL and p in the picture are the same.In other words, the apparent ratio of length to width of the object in the picture represents the real ratio in the scene (see Kanade [ 19791 for the proof.),

Rationale and Justification
Skewed symmetry has straightforward applications to scenes containing objects that have been manufactured, whether naturally or artificially.Many constructed items exhibit symmetry, occasionally about many axes.Some symmetries are introduced due to economies of the manufacturing process: an object is often composed of identically formed component parts (fibers, cells, bricks, etc.).The symmetries result from the three-dimensional tessellation of the components into the whole.Often the tessellation is effectively two-dimensional, in laminae (cloth, honeycombs, walls, etc.), and the application of the s k c n d symmetry method is straightforward.Further, the requirement for a close symmetric packing of the coniponents occasionally imposes a local symmetry on the individual components, too.The method can then be applied to individual parts (such as the bricks themselves).Notice the method does no( a s s u m 3-L) symmetry of the whole object; what is assumed is loc.nl2-D symmetry.
A further source of symmetry is the bilateral symriictry that results from biological manufacture (growth).It not only contributes symmetric objects to the environment: it may also be responsible for an imitative esthetic bias in huriian riianut'acturc.11. thc extent of a bilaterally s!fniriictric patrc'rii into thc third dimcnsion is not too great (a face, a leaf, an airplane), the skewed symnietry method can be approximately applied also. .

Affine-Transformable Patterns
In texture analysis we often consider small patterns (texels= texture elements) whose repetition constitutes "texture."Suppose we have a pair of texel  and ( r i i i i d i i t i o i i ) .
Note that, as in the case of' skewed symmct?. thc convcrsc o f this assumption is always true under onhographic projection.Thc iibtlvc a\\uriiption can be schematized by Figure 3. Consider t\vo texel pattcrnh I' I and P z in the picture, and place the origins of the .r -\ coordinates at thcir center\.respectively.The transform from P to f' , i .i n hc i i t n i c\prc*N\c-ti h\ .irc-gul;ir 2 N 7 iiiatris A = ( a j j ) .P and P arc p r q c c t i o m ot.pattcrr\\ I' , : i d I': \vhich arc dra\vn on the 3-D surfaces.W e ~w r i i c th:rt f' I anti f ' : ;ire w i a l l cnouyh SO that \vc can regard them as being dranm on small planex.I.ct I I ~ dcnotc thc gradients of those small planes by G I = ( p l .4 , ) and G 2 = ( .J 1 2 .q 2 ) .respectively; i.e., P I is d r a w n o n a p l a n e -z = p l -r + q l !~ a n d P S on -: = p l -v + y L v .Now, our assumption amounts to saying that P I is transformable from (We can omit the translation trom our conxideration.xiiicc' t.or ciicti pattern the origin of the coordinates is placed at its g i r v i t > ccntcr.uhich i \ presewed under the affine transform.)Thinking about a pattern cfra\\m o n ;t sinall plane, -z = p x + q y , is equivalent to viewing the pattern froin directly o\*crhcad; that is, rotating the x -y -z coordinates so that the nomial vector of the plane is along the new z-axis (line of sight).For this purpose we rotate the coordinates first by + around the y-axis and then by 0 around the x'-axis (Figure 4).We have (4) The plane which was represented as -z = p x +y? in the old coordinates is. of course.n o w rcprcsented as -: ' = 0 in the new coordinates.
Let us dcnote the angles of the coordinate rotations t o obtain P I and 1): in Figurc 3 by ( + , , 0 1 ) and (&,e2), respectively.By eliminating u and a and substituting for sin+,, cos+,, sinO,, and * ' cosei from (3), we have the following equations in p 4 p 2 .and y: We thus find that the assumption of affine-transformable patterns yields the constraint represented by (6) on surface orientations.The constraint is determined solely by the matrix A = (a ij ), which is determined by the relation between P2 and P I observable in the picture without knowing either the original patterns ( P In order to have an idea about the degree of the constraint represented by ( 6), if we assume that the orientation of P 2 is known (i.e., G I = (p2.q2) is known), then (6) gives two simultaneous equations for G I = 0) I.y ,).The system appears to be of degree 4, but it can be shown that there are only tu'o solutions; they are of the form (po,qo) and (-po,-qo), which are symmetrical around the origin of the gradient space (see the Appendix).and P i ) or their relationships (a and R ) in the 3-D space.

I
From ( 5 ) we can also derive the following relationship:'  If we assume that a , p, T , and p are known, then ( p 1 , y I ) has two possible solutions.This is essentially the case which Ikeuchi ( 1 980b) investigated in his shape recovery method by assuming a known standard pattern, even though he used the constraint only partially.
Let us consider the case where (Y and p are known, but T and p are not. One can substitute ujj in ( 9) by (8), and eliminate T and p.Then we obtain (plcosa+qIsina)(plcosp+qIsin~)+cos(a-~) = 0 which reduces to the same as the hyperbola (1).This can be interpreted as follows.
As was noted in the previous subsection, a pair of affinc.-rransforrnablepatterns impose the constraints (6) between their surface orientations, in which, if one is fixed, the other has only two possible orientations.Hotvcvcr.if we loosen the transform in such a way that the angular (rotational) corrcspondence (a and p) is known while the length relationship is not known ( o r arbitraq).
then the one-dimensional constraint of the skewed-symmetry hyperbola is obtained.

The Shape-from-texture Paradigm
This section derives the same skewed-synlilletry copstraints from a second theory, different from that of the affine-transformable patterns.The shapefrom-texture paradigm is a method of relating image texture properties to scene object properties, by explicitly incorporating assumptions about the imaging phenomenon into a computational framework.The paradigm is briefly presented here, but a fuller discussion can be found in (Kender, 1980).
The paradigm has two major portions.In the first, a given image textural property is "normalized" to give a general class of surface orientation constraints.In the second, the normalized values are used in conjunction with assumed scene relations to refine the constraints.If there are sufficiently many textural elements ("texels") in the image to be normalized, and if enough assumptions are made about their scene counterparts, then the underlying surface's orientation can be specified uniquely.Somewhat more weakly, only two tcxels are required, and only one assumption (equality of scene textural objects, or some other simple relation), to generate a well-behaved onedimensional family of possible surface orientations.The method of skewed symmetrythe use of qualitative symmetries in the image to create a perspectively distorted right angleis an example of such a weak method.
The first step in the paradigm is the normalization of a given texel property.The goal is to create a normalized texture property map (NTPM), which is a representational and computational tool relating image properties to scene properties.The NTPM summarizes the many different conditions that may have occurred in the scene leading to the formation of the given textural element.In general, the NTPM of a certain property is a scalar-valued function of two variables.The two input variables describe the postulated surface orientation in the scene (top-bottom and left-right slants: 0, , 4 ) when we use the gradient space).The NTPM for a horizontal unit line length in the image summarizes the lengths of lines that would have been necessary in 3-D space under \*;triou\ oricntations: at surface oricntation ( p .(1 ). it would have to be m.hlorc specifically, the NTPM is formed by selecting a texel and a texel property.hack-projecting the texel through the known imaging geometry onto all conccitrahle surface orientations.and nieasuring the texel property there.The representation chosen for the two-dimensional space of orientations is irnportlrnt; u'c w i l l .houcvcr.only use the sradicnt space here.
In the scccinci phase ot thc paradigm.the NTPM is refined in the following w .3 ~.7'csc.I~ usually hat-1) various orientations in the image, and there are Illany diffcrcnt tcscl types.Each texel gencrxtcs its own image-scene relationships, suniniarizcd in its NTPM.If, however, assumptions can be made to relate one texel to another, then their NTPMs can also be related; in most cases only a few scene surface orientations can satisfy both texels' requirements.Some examples of the assumptions that relate texels are: both lie in the same + -htapping image propertics inlo sliapc constraints .247 plane, both are equal in textural measure (length, area, etc.), one is k times the other in measure, etc. Relating texels in the manner forces more stringent demands on the scene.If enough relations are invoked, the'orientation of the local surface supporting two or more related texels can be very precisely determined.

Skewed Symmetry from the Paradigm Applied to Slope
What we now show is that the skewed symmetry method is a special case of the shape-from-texture paradigm; it can be derived from considerations of texel slope.
To normalize the slope of a texel, it is back-projected onto a plane with the postulated orientation.The back-projected texel now has a new shape on this new surface.Its exact value, however, depends upon the coordinate system on this surface plane.Many coordinate systems are possible; we chose here a coordinate system whose x-axis lies along the gradient direction.Thc normalized slope is then the angle that the back-projected texel makes with respect to the surface coordinate system x-axis.The calculation is a bit involved, especially under perspective, which requires a knowledge of both the location of the center of focus and the length of the focal distance.
Using the construction in Figure 6, together with several lemmas relating surfaces in perspective to their local vanishing lines, slope is normalized as follows.Assume a slope is parallel to the p-axis; the image and gradient space can always be rotated into such a position.(If rotation is necessary, the resulting NTPM can be de-rotated into the original position using the standard twoby-two orthonormal matrix.)Also assume that the slope is somewhere along the line y = y,, where the unit of measurement in the image is equal to one focal length.The normalized value of the slope is equal to the tangent o f the 3-D space angle -q, whose base (of length R ) is parallel to the surface plane, and is in the direction of the gradient.R is determined from the focal distance, and from the point of the nearest approach o f the v a n i h i n g linc 0 1 [tic plane.This line has equation px +qy = I (or G .P = I ) and its nearest approach is GAlG 112.The distance d is given by the intersection of the linc \ ' = J, with the vanishing line.Then, the nomiali7cd slope valuethc NoniiaIi/cd Texture Property Map -is given by This normalized value can be exploited in w\.eral ways.Most iriiportant is the result that is obtained when one has t i i .0 s10pc.s in the innage that iirc a~sumed to arise from equal slopes in the scene.Under this assumption, their normalized property maps can be equated.The resulting constraint, surprisingly.is a simple straight line in the gradient space.It is intimately related to the vanishing point formed by the intersection of the extensions of the two image slopes (Kender, 1980).
The constraint equations resulting from assuming that the two slopes arose from perpendicular lines in the scene is, however, enormously complex.It unfortunately does not appear to have many tractable forms or special cases.Considering two image slopes to have arisen from parallel lines in a trivial solution.If the image slopes are parallel, the entire gradient spacc is a solution.If they are not, there is no solution at all.This corrcyxmds to the projective geometry theorem that under orthography, parallels are taken into parallels regardless of surface orientation.
In the case where the scene slopes are assumed to be perpendicular, we again get a simplification, but this time a useful one.Not only is the solution tractable, it is the skewed symmetry method of Section 2. We derive it as fol- lo\\ \ Consider Figure 7.Note that under orthography, texels can be translated arbitrarily, sincc the focal length is infinite and the focal point is effectively everywhere; there is no information in image position.Given the angle that the two texels forni, rotate the gradient space so that the positive p-axis bisects the angle.Call this adjustment angle A; we will use it to de-adjust our results into the original position after they have been computed.Let the angle that is bisected be 26.The normalized value of either slope is obtained directly from the standard normalized slope formula, corrected for the displacement of +S and -6 respectively.That is, for the posi- tive 6 orientation, instead of formula (ll), we use the formula under the substitution pcosS+qsinS for p , -psinS+qcosS for We proceed similarly for transformation (it is the length of the normal vector of the surface).
The fact that the normalized slopes are assumed to be perpendicular in the scene allows us to set one of the normalized values equal to the negative reciprocal of the other.The resultant equation becomes (12) the slope at -6.Note that the factor q + l+p + q is invariant under this This is exactly the hyperbola in Section 2 with 26 = I a -p I .

Skewed Symmetry from the Paradigm Applied to Length and Angle
The paradigm is similarly applicable to other texture measures.Using texel length as the property to be normalized, we find that under perspective, lengths must lic on thc s m w line in order for the resultant equations to be simpler than the fourth order.If they are collinear, again the resultant gradient space constraint is a simple straight line.
Under orthography and the assumption that image lengths have arisen from equal scene lengths.the constraint equation is again a hyperbola -the skewed-symmetn hypcrhola.somewhat offset.In fact, the geometric construct i o n in Figurc 8 shoir.5 that the assumption of equal length can be madc cq u i va 1 en t to skewed s y ni m c t ry .
First, a triangle is f'ormed by translating one or the other of the lengths so that they meet at a common endpoint.Under orthography, such translations do not affect the resulting constraints.Connecting the remaining endpoints creates a triangle which must be isosceles in the scene.Further, under orthography, midpoints of lines are preserved (the midpoint of the base of the scene triangle is imaged as the midpoint of the base of the image triangle).The line connecting the vertex and this midpoint has the property that, in the scene, it must form a right angle with the base.Its distortion to something other than a right angle in the imagethe induced angle 26 -is precisely the distortion which characterizes skewed symmetry.Therefore, the same methods apply.
One other case is worth mentioning.Supposc the image has two angles such that one leg of the first is parallel to one leg of the second.See Figure 9.
In this case, again the constraint is equivalent to skewed symmetry, as the construction shows.Choosing one of the angles, extend its non-parallel leg until it intersects both legs of the other angle.(If it cannot do so, then first translate the angle before extending.)The resulting triangle must be isosceles in the scene, since the angles are assumed equal in the scene.However, this is the same situation encountered above with the construction involving lengths.Therefore, the altitude from the midpoint of the base (here, the midpoint of the parallel side) to the vertex must form a right angle.Again, the distortion observed in the image is thc skewed s y n m c t ~' distortion.world gives the labeling shown in Figure lob, which signifies that the three edges meeting at the central FORK vertex are all convex, i.e., the object is a convex comer of a block.However, it does riot specify a particular quantitative shape.In fact, the labeling indicates only that the gradients of the three surfaces should be placed in the gradient space so as to form the triangle shown in Figure IOc.The edges of the triangle should be perpendicular to the picture edges separating the corresponding regions, but the location and size of the triangle are arbitrary in the gradient space.Therefore, the object is not necessarily right-angled.

Applications of
We can use skewed symmetry here to provide additional constraints.The three regions are skewed-symmetrical with the axes shown in Figure 1 la.The hyperbolas corresponding to thcsc regions are shown in Figure 1 Ib.Thus the problem is now how to place the triangle of Figure 1Oc in Figure 1 Ib so that each vertex is on the corrcspondiny hyperbola.Kanadc (1979) proves that the combination of locations sh0js.n in Figure 1 Ib is the only possibility.and that the resultant shape is a right-angled block.tion o f the drawing as a trapezoidal block in this case.

Skewed Symmetry under Gravity
tural necessity to oppose the gravitational field.
One principal influence toward symmetry seems to be an object's struc-Objects that must themselves tend to have structural nieniDcrs alignea parallel l o ttie directLon ot force, that is, vertically.Such members are niutually parallela typc of symmetry.The base of such an object is often perpendicular to gravity to distribute weight and provide balance.Together, then.the hase and structural niembers provide a local symmetry franie that can also bc cxploitcd by the skewedsymmetry method.One can sliou that in thi\ la\( c-;i\c it I \ u\uully possible t o specify surface orientation uniquely. We will assume that the direction of the p v i t y ficld is knokvn  14.Using skewed symmetry (or even direct observation), it is not hard to obtain an angle in the image that corresponds to a right angle in the scene.Suppose one of the legs of the angle is parallel to the known gravity field as in Figure skewed-symmetry method generates the following constraint hyperbola: p = -(q + l/q)coty.(O,q, ,I).@,, ,l/4, , I ) = 0.) Note that the value of 4 for m y vertical plane is fixed at -l/qg.Thus, in our example, p is also determined: it is ,c -(4g + l/q,)coty.Since qR is a constant, p vanes simply with y .Figure 14 shows the constraints graphically.

Shape Recovery of an Object with Many Patterns Stamped
Consider the problem of recovering the shape from a picture of a ball with a number of patterns stamped on it (see Figure 15).For each pair of texel patterns, if they are affine-transformable, we compute a transformation matrix A .Thus we obtain many constraints on the gradients of texels.From these, however, we cannot uniquely determine the surface orientation of each texel.
We need more assumptions or data.We will suppose we know the gradients of some particular texels, and assume that the surface is smooth (together, maybe, with an assumption of global concavity or convexity).Then a relaxation or cooperative technique similar to the one for shape-from shading (Woodham, 1977;Ikeuchi, 1980a) will allow us to determine consistent assignments of gradients to the texels which satisfy those many constraints.Notice that we need not assume that the original pattern is known, nor that the patterns are stamped in a particular manner.Even other pattcrns can be mixcd together with them.
One of the plausible methods of determining the gradient of one particular texel is to use equation ( 7).Assuming u = 1.we order the texels by the magnitude of d m , and assign ( p .q ) = (0.0) (the orientation that is directly facing the viewer) to the least slanted tcscl.This is arialogous to a similar hypothesis in shape from shading.That is.we tend to assign to the brightest point the orientation directly facing the light source.even though under the assumptions of parallel light3 ;mtl ;I nuttc wrf:icc.onc c<:in only say that the brightest pixels have the minimum incident angle o f light.not necessarily 0".
This provides a useful meta-heuristic for exploiting iniagc propcrticb: wc can call it the meta-heuristic of nort-accidental image propc.r-tic..s.It c a n bc regarded as a generalization of general view directions, often used in the blocks world, to exclude the cases of accidental line alignments.
terns, and texture analysis can be scncrali/cJ i t > .

Figure 2 .
Figure 2. The hyperbola dctcrmincd by a skewed symn1ctt-y Jctincd by a and (3

A
Figure 3. transformable patterns.A schematic diagram showing the assumption about the affine-

Figure 4 .
Figure 4. Rotation of the xyz coordinates.
the ratio of cosines of the slant angles of the patterns is equal to the ratio dct(A )/(I?.If we assume u = 1 (the original patlcrnr ; I I -~ 01 t hsame size) or that (T is known, (7) shows that we can order ttic tcxcl p;it1cr-n\ according to the magnitude of slant, Ti or d m , usins the \ ~; l l u c ~ 01.det(A ).
%his indicates that det(A ) should be positive.But if it is negative, then u'c can assume that f 1' and P2 are mirrored patterns, and put R = (sina -, Symmetry from Affine-transformable Patterns The affine transform from P to P I is more Intuitively u,ndcrhttx)d by how a pair of perpendicular unit-length vectors (typically along the s and ~9 coordinate axes) are mapped into their transformed vectors.As shown in Figure 5 , two angles ( a and p) and two lengths (7 and p) can characterize the transform.The components of the transformation matrix A = (a;,) are represented by a I = TCOSOL a 12 = pcosp = Tsincu a22 = psinp.Suppose, for simplicity, the orientation of P 2 in Figure3is known to be (p2,q2) = (0,O).This simplifies equation (6) to all(p:+l)+a*1PI91

Figure 5 .
Figure 5.An affine transform (without translation) as characterized by two angles and two lengths.
of a texel becomes Under orthography, nearly everything simplifies.The normalized slope a It is independent of F .~; in effect, all slopes are at the focal point.

Figure 7 .
Figure 7. Two image texels assumed to be perpendicular in the scene.

\Figure 8 .
Figure 8. Assuming lengths are equal generates the skewed symmetry constraint.

Figure 9 .
Figure 9. Assuming angles are equal generates the skewed symmetry constraint.

Figure 10 .
Figure 10. (a) A line drawing of a block; (b) Huffman-Clowes-Waltz labeling; (c) constraints in the gradient space.

Figure 11 .
Figure 11.(a) Axes of the skcwcd symmctry of the reyions of Figurc 10a; (b) corresponding hyperbolas and allocations of the gradients.
Figure 12.A line drawing of a rhomboid: this cumoi be a right-angled block.Notice that Figure IOa can be a rhomboid.

Figure 13 .
Figure 13.Shapc rccovery of a trapezoid block: (a) axes; (b) gradient alloca- This constraint is somewhat interesting: it expresses p (left-right slant) as If gravity points in the -4 direction, the ground plane must have as its Since all a frcricfiori o f 4 (top-bottom slant).The value o f q itself is easil!. obtained.orientation ( O .Y , ~) , for a value of qg deterniinable through sensing.

Figure 14 .
Figure 14.Assumptions about gravity can uniquely specify surface orientations.
vertical planes are perpendicular to the ground plane, all vertical planes must have the orientation (p,, .-Q),for variable p,. .( A quick check shows