Sketch-based Interfaces: Exploiting Spatio-temporal Context for Automatic Stroke Grouping

Authors:

Lutz Dickmann, Tobias Lensing, Robert Porzel, Rainer Malaka, and Christoph Lischka

Summary:

Goal: Grouping strokes in sketches based on spatio temporal data.

feature list:

- proximity, time between strokes, duration of the stroke, horizontal and vertical velocity.

- short term memory - using the features

- SVM and echo state networks.

ESN - neural network technique.

FNN - no memory

RNN - memory but convergence not gauranteed.

ESN - kind of linear combination of many RNN. (read more about this)

tested on the naive landscape scenes and hatching and arrows.

Performance score was based on precision and recall analysis.

Discussion:

What is the significance of the vertical and horizontal velocity of the stroke?

Example based automatic font generation

Authors :

Rapee Suveeranont

Takeo Igarashi

similar work: http://www.pilothandwriting.com/en/

Summary:

Goal: editing and creating a new font with font editor is difficult. correcting spline requires expertize. So reduce the work in generating a font.

Steps to create a font:

- sketch the alphabet sample with pen.

- overlay the alphabet skeleton to correct the sample.

- correct the sample(also called skin) using the overlaid skeleton.

Blending/ modifying the font.

- using various properties the skin/ sample can be modified.

- transform (shear, shrink...) the skeleton to change the skin and see the change in the UI.

Algorithm:

Once the sample alphabet is drawn by the user. The application fits the simplest skeleton to the skin drawn. (topological problems - Q). After fitting the skeleton, the system automatically corrects the skin based on the skeleton.

While blending, 2 weights are generated - one for the skeleton and one for the skin. the font is then mapped to the nearest font skin in the available database.

results:

- generating skeleton was fairly accurate.

- generating skin gave mixed results

- constraints due the linear morphing from the existing font.

- user study with 3 users and 3 trials for each.

- font generation took lesser time with each iteration

Discussion:

interesting technique. When the user gives a sketch, why match it or morph it to an existing font.

A nonlinear approach for face sketch synthesis and recognition

Authors:
Qingshan, Liu
Xiaoou, Tang
Hongliang, Jin
Hanqing, Lu
Songde, Ma

Summary:
This method uses LLE to probe sketch based on the pseudo sketch generated from the photos. The idea behind LLE is to compute neighbhor preserving mapping between high dimensiion data to the low dimension feature space (usually based on simple geometric intuition in this case this is complex). In the case of faces, patch based strategy is followed - the photos are divided into overlapping patches. The pseudo sketch and artist sketch is normalized by fixing the position of the eye. KNN is used to calculate the weights of patches - to provide smooth transition between patches k=5 is chosen. KNDA - non linear version of LDA is used to classify the sketch. KNDA provides a better classification rate than LDA and PCA.

Discussion:
Need to read the paper in detail and understand how the patch based transformation works. How does KNN help in the transformation.

Face Sketch Recognition System to Support Security Investigation

Authors:
Setiawan Hadi,
Iping Supriana Suwardi,
Farid Wazdi

Summary:
The three ideas behind component based face feature detection First, some object classes can be described well by a few characteristic object parts and theirgeometrical relation. Second, the patterns of some object parts might vary less under pose changes than the pattern belonging to the whole object. Third, a component based approach might be more robust against partial occlusions than a global approach. The method is performed in two stages. On the first stage, component classifiers independently detect components of the face. In the example shown these components are the eyes, the nose and the mouth. On the second stage, the geometrical configuration classifier performs the final face sketch detection by linearly combining the results of the component classifiers. Then SVMs are used to classify the faces based on the components in the sketch.

Discussion:
The interesting idea here is to divide the face into component features and compare them with features of the sketch which is kind of finding the middle ground between photos and the sketches.

Face sketch synthesis and recognition

Authors:
Xiaoou, Tang
Xiaogang, Wang

Summary:
The algorithm uses eigenfaces on sketch of the faces to find the weight vectors. This algorithm assumes that the transformation from the image of face to its sketch is linear. This is important because the mean representation of the face can converted to mean sketch using the linear transformation matrix. This linear assumption is also reasonable because the high pass filter can give rise to good sketch approximation.
The fiducial points in the sketch may not be the same when compared to the fiducial points on the face since the artist drawing the sketch may have exaggerated some features. This makes it difficult to find the linear transformation for the mapping of features and mapping of gray areas around the fiducial points. In order to remove this exaggeration and map the different fiducial points of the photos, The images are warped to mean image. affine transformation is performed on the face images and sketches to fix the different fiducial points. Eigenface method is then applied on the reduced sketch. To minimize the effect of transformation errors on the classification, the artist drawn sketch is classified using a bayesian classifier using the mahalanobis distance from the artist sketch vector to the eigenfaces of the reduced sketches of photos. The method than uses eigenfaces + bayes works better than PCA alone and PCA + eigenfaces.

Discussion:
This method again needs training data to find the eigenfaces.

Face sketch recognition

Authors:
Xiaoou, Tang
Xiaogang, Wang

Summary:
To compare geometric features in ideal condition, a fiducial grid model is designed. 35 fiducial grid points are identified.
The eigenface method uses Karhunen–Loeve Transform (KLT) transform to transform images to vectors. This gives a highly compressed representation of images but when compared with sketches, the distance between vectors of same image may be larger than the vectors of different images. To overcome this difficulty, the images are first transformed to sketches and KLT is performed on them. Once the face image is converted to a sketch then comparison sketch is done using the eigenface method. The face to sketch comparison works better than theface image comparison with sketch (ordinary eigenface method) and geometry method.

Discussion:
This method needs training data to calculate the average and weight vectors of each image. We cannot afford this in IcanDraw since we have just one sketch drawn by user and the photo.

Face Recognition by Expression-Driven Sketch Graph Matching

Authors:
Zijian, Xu
Jiebo, Luo

Abstract:
We present a novel face recognition method using automatically extracted sketch by a multi-layer grammatical face model. First, the observed face is parsed into a 3-layer (face, parts and sketch) graph. In the sketch layer, the nodes not only capture the local features (strength, orientation and profile of the edge), but also remember the global information inherited from the upper layers (i.e. the facial part they belong to and status of the part). Next, a sketch graph matching is performed between the parsed graph and a pre-built reference graph database, in which each individual has a parsed sketch graph. Similar to the other successful edge-based methods in the literature, the use of sketch increases the robustness of recognition under varying lighting conditions. Furthermore, with high-level semantic understanding of the face, we are able to perform an intelligent recognition process driven by the status of the face, i.e. changes in expressions and poses. As shown in the experiment, our method overcomes the significant drop in accuracy under expression changes suffered by other edge-based methods

Summary:
The face features are divided into 3 layers - face ( whole - high level feature) , part layers ( features like mouth,eyes,...) and sketch layer( features like line segment, blob, ...) .The sketch layer which is the low level features carries the knowledge from the higher level parent ( like position, length, orientation,...). This is helps in differential weighting of the features where the mouth can be given lower weight (to minimize difference in open and closed mouths) while giving greater weights to the eyebrows. This paper also provides a similarity measure to find the similarity between the sketches and photos.