|Publication number||US7502017 B1|
|Application number||US 11/943,260|
|Publication date||Mar 10, 2009|
|Filing date||Nov 20, 2007|
|Priority date||Nov 20, 2007|
|Publication number||11943260, 943260, US 7502017 B1, US 7502017B1, US-B1-7502017, US7502017 B1, US7502017B1|
|Inventors||Eugene Henry Ratzlaff, Yin-Min Chee|
|Original Assignee||International Business Machines Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (21), Non-Patent Citations (7), Referenced by (19), Classifications (7), Legal Events (4)|
|External Links: USPTO, USPTO Assignment, Espacenet|
The present invention relates to a handwriting recognition device and, more specifically, relates to a handwriting recognition device including a user interface and display area.
Handwriting recognition user interfaces (HWRUIs) for computing devices, e.g., personal digital assistants (PDAs) or handheld computers, allow the user to enter handwritten strokes, for example, via a handwriting stylus or pen, and then facilitate correction of recognition errors. HWRUIs typically transfer recognized characters directly to the user application (for example email or SMS) as would happen with a keyboard interface. Typical HWRUI systems try to optimize certain aspects of this interaction, but for many systems the engineering emphasis has been on recognition accuracy and not on the entire user experience.
One problem with known HWRUIs is that users cannot easily review and confirm that their output is correct, and cannot easily correct errors when they occur. Typically, the output appears in an application window that is far from the writing area, so that the user must divert his attention back and forth between the writing area and the distant application window. In order to make a correction, several steps are typically required. First, the user must move the stylus from the writing area to the application window and then select the character or word that contains the error. Unfortunately, proportional fonts often make the process of selecting a narrow character difficult. Then the user must re-write the character in order to replace the error with a correct result, which is cumbersome and time consuming. Further, the user must wait for the recognizer to return the corrected result to check if the result is in fact correct, and if not, repeat the process.
Typical handwriting recognition systems do not allow users to correct errors as they happen or “on the fly”. Correcting errors as they happen is natural for many users especially users who work with current word processing programs that allow easy correction when typing using a keyboard. The user can see the output in real time, and correct the output immediately. The user can either confirm the accuracy of the output by seeing the keys they strike, or the user may be a touch typist who is able to watch the output at the same time she enters the input. By contrast, as few writers write legibly and therefore need to watch the recognized text to check and correct their input, it is inconvenient to have the output appear both distant from the writing area and in a continuously changing location. Current HWRUIs do nothing to ameliorate this problem.
Another problem with known HWRUIs is that writers must either enter each character (or word) into a separate box or must create distinctly wide spaces between characters and words to ensure that the recognizer correctly segments these character or word units. If a separate box is used, the user typically must move back and forth between different boxes in order to write. If a single box is used, the user must write sequential characters in the same location. When a single box is used the user must wait for the system to process and clear the writing before entering new text. Further drawbacks of current systems include problems related to putting spaces between characters or words. Writing with natural spacing often results in merge errors, e.g., the recognizer merges characters together. Further, exaggerating spacing to compensate for merge errors results in an unnatural and uncomfortably wide spacing style that also wastes space, bringing the writer to the end of the writing space prematurely. As a result of entering fewer characters on a single line, the efficiency of the recognition system is reduced since a new line must be started more frequently. Starting new lines not only means additional stylus relocation, but also generally results in disrupting the user's writing.
Therefore, a need exists for a handwriting recognizing system and interface which allows a user to readily view the recognized entered text and make corrections with minimum strokes and shifting of a stylus. It would also be desirable to provide a system and interface, which improves the ease of character and word spacing while minimizing the space used.
In an aspect of the invention, a handwriting recognition device includes a handwriting recognizing component, and a user interface including a display. The handwriting recognition device further includes a stroke entry field for entering strokes forming written characters in a current character field of the stroke entry field which form a current character stroke set. The current character stroke set is written by multiple strokes, and the handwriting recognizing component interprets the multiple strokes as comprising a single character. A new character field adjacent to the current character field indicates a position in the stroke entry field for entering a next character. A new word entry field adjacent to the stroke entry field is for entering strokes for a new word. A recognized character display area on the display substantially adjacent to the stroke entry field displays at least one recognized character by the handwriting recognizing component. The recognized character and the locations of the new character and new word entry fields are changed in response to adding strokes to the current character stroke set, and the handwriting recognizing component includes an output recognition buffer (ORB) for holding the at least one character in the buffer while the at least one character is being displayed in the recognized character display area.
In another aspect of the invention, a method for recognizing handwriting includes: writing characters using multiple strokes in a stroke entry field on a display of a user interface; displaying the digitalized strokes in a current character field of the stroke entry field; interpreting the multiple strokes as comprising a single character; recognizing the written characters; displaying recognized characters substantially adjacent to the current character field of the stroke entry field; changing the recognized characters in response to changing the strokes in the stroke entry field; displaying a new character field adjacent to the current character field indicating a position in the stroke entry field for entering a next character; and displaying a new word entry field adjacent to the stroke entry field for entering strokes for a new word.
These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof which is to be read in connection with the accompanying drawings, in which:
An exemplary embodiment of a handwriting recognizer user interface 10 according to the present invention is shown in
The ORB display 15 reflects the contents of the ORB 86. The buffer 86 may also include additional information, such as, the temporal order in which the characters were generated, the alternates list for each character, the alternates list for each word, and a record of the original strokes that resulted in the character/word.
When edits are made in the ORB display 15 using a menu 1000, as shown in
The ORB display 15 facilitates the display and correction of characters and words. The ORB display 15, in one embodiment depicted, is a single-line text display window and appears adjacent to the stroke entry and display field 20, which location is convenient for reviewing and correcting results as they are written. The ORB display 15 renders the text right-justified (or left-justified for right-to-left languages such as Arabic) using a constant-width font in an exemplary embodiment. Right justification means that as each character is sent to the ORB display 15 subsequent to recognition, the most-recently added character appears in the rightmost display location of the window, and any and all of the antecedent characters are moved one character position to the left. By this means, the user has a single point of focus in the ORB display 15 for new characters. This point of focus is also close to where the user is looking, because the text is right-justified and the ORB display 15 is near the writing fields. This is in contrast to left-justified display outputs, where the user's point of focus must move as characters are sequentially entered and added to the text output being generated by the user. As a result of the point-of-focus being close to the stroke entry field 20 and being stationary, the writer who frequently views and confirms the output while writing can write more quickly without the need to seek a distant, moving target.
It is an advantage of the ORB display 15 interface method to provide both character and word alternatives. In an example of word correction by revising only selected characters, the word “CODE” (a hexadecimal number) is written, where the second character is a zero. The recognizer misinterprets the first two digits and consequently the displayed result is “6ODE”, where the second character is the letter “O”. Using the ORB display 15 with both character and word alternates the user could find character corrections for both the erroneous “6” and the “O” and complete the correction in only four taps. Note that if the ORB display 15 interface method had instead been constrained to offer only whole world alternatives the user would have been required to rewrite the entire word in order to correct it. Alternatively, if the desired outcome was the English word “code” the user could instead choose to view full word alternates, where it is likely that the list would include “CODE” and “code”. It is understood that, in some embodiments, correct results for English and non-English characters or words may be displayed if the handwriting recognition software is so programmed.
Another advantage of the ORB 86 is that it allows simple access to alternates-list corrections on the ORB display 15 without the need to correct each character or word immediately after it is entered. This also obviates the need for the software developer of the user application to provide special software interfaces between the user application and the recognizer in order to allow the user to make changes to the user's content in the user application. Further, allowing one or more characters to be written before corrections are made allows the user to focus on writing an entire phrase without interruption. Resultantly, the user can more quickly enter text and data because the corrections can be done after a word or phrase is written by using the ORB display 15. The ORB 86 allows corrections to be performed using the ORB display 15 so that all entries in the buffer can be changed.
The handwriting recognizer device 78 can also be programmed to use the original misrecognized stroke data to retrain the recognizer to more accurately return the correct result. The user can instruct the device 78 to retain written strokes with associated user-initiated corrections either manually or automatically. This retained information can then subsequently be used to retrain the recognizer for improved recognition accuracy.
In another embodiment according to the present invention, a pop-up virtual keyboard is programmed into the device 78 to allow the user to immediately key in the selected character (or word) for correction. Further, the device 78 can support different user selectable methods of displaying character or word alternates. For example, presenting alternate characters in a pop-up virtual keyboard with the likely alternates highlighted or enlarged.
In another embodiment, an enhanced display of the character(s) displayed in the ORB display 15 provides a clear, visual feedback for selecting the desired character for correction. The character and their neighbors may be expanded or magnified when touched by a stylus. This would, for example, be helpful for properly selecting for correction narrow characters such as “i” or “I” amidst other characters. Expansion or magnification would allow the user to subsequently move the stylus over the desired character within the magnifier and then lift the stylus to initiate the pop-up alternates menu.
In another embodiment of the invention, the device 78 can be programmed to support right-to-left languages such as Arabic.
Further, in another embodiment, special spacing indicator symbols could be inserted between the buffered text characters in the ORB display 15. The special spacing indicator symbols may be displayed approximately above or below the text line, for example, and have a special color or appearance otherwise distinct from the recognized characters, and allow sufficient space between characters to permit easy selection.
The user interface 10 uses a variable spacing (VariS, pronounced “varies”) entry method that continuously monitors stroke entry and dynamically adjusts the location of the new character field spacing indicator 24 and new word spacing indicator 48. These indicators show the writer the areas where strokes may begin or should not cross in order to avoid or alternatively create valid character and/or word separation. By dynamically adjusting the location and/or appearance of such indicators the HWRUI more efficiently uses the available space, since the alternative is to implicitly require that the user be overly conservative in generating wide spaces between characters or words in order to maintain optimal recognition accuracy. VariS allows the user to write more quickly while simultaneously helping improve recognition accuracy. Indicators other than the new character field spacer 24 and the new word spacing indicator 48 can be used to provide useful indications to the user, such as, different colors or sizes of words or characters.
VariS works in conjunction with the recognizer and ORB 86 and ORB display 15 to allow for continuous recognition of characters. After the first stroke of a character is completed (with stylus subsequently raised), all characters in the ORB display 15 are shifted to the left and the recognized result corresponding to the aforementioned stroke is presented at the (right-justified) end of the ORB display 15. Each time an additional stroke is added in the area overlying the original stroke in the process of completing the written character, the device 78 replaces the last character with a new recognized output in the ORB display 15. The advantage of continuous recognition is that it allows a writer to instantly confirm the recognition result after the writer has finished writing a character without having to wait the conventional timeout period. This is more comfortable, less distracting and allows faster input because less time is consumed waiting for recognition to complete before the user can confirm that a result is correct.
In use, the user writes in the stroke entry field 20 at the new character entry field bracket 24. When the user lifts the stylus, the appearance of the written stroke is changed to indicate the current character set segmentation. Character segmentation refers to how strokes are “grouped” to be members of different character stroke sets. In this case, those strokes that are part of the current character are distinguished from those that are considered part of previous characters. Each grouped stroke set is separately recognized as a single character. At the same time, the stroke entry and display field 20 displays the character and the new character entry field bracket 24, and the new word spacing indicator 48, respectively, based on, for example, the rightmost coordinate of the last stroke, as shown in
The ORB 86 is typically cleared when the user generates a new line. The method for generating a new line is not part of the present invention, but is typically through the use of a special stroke gesture or button or by interacting directly with the user application. The ORB 86 memory functions as a FIFO buffer (first-in, first-out buffer). When the ORB display 15 fills, the “oldest” (left-most) characters are typically cleared from the ORB 86 and thus the ORB display 15. Clearing typically takes place at the word-level, that is, when the first character of a word would extend outside the window then the entire word is removed from the ORB 86 and display 15.
While the present invention has been particularly shown and described with respect to preferred embodiments thereof, it will be understood by those skilled in the art that changes in forms and details may be made without departing from the spirit and scope of the present application. It is therefore intended that the present invention not be limited to the exact forms and details described and illustrated herein, but falls within the scope of the appended claims.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US3487371||Mar 3, 1967||Dec 30, 1969||Scandata Corp||Data entry system|
|US4262281||May 29, 1979||Apr 14, 1981||Quest Automation Limited||Processing of information|
|US5455901||Sep 12, 1994||Oct 3, 1995||Compaq Computer Corporation||Input device with deferred translation|
|US5546538||Dec 14, 1993||Aug 13, 1996||Intel Corporation||System for processing handwriting written by user of portable computer by server or processing by the computer when the computer no longer communicate with server|
|US5682439||Aug 7, 1995||Oct 28, 1997||Apple Computer, Inc.||Boxed input correction system and method for pen based computer systems|
|US5717939||Nov 17, 1995||Feb 10, 1998||Compaq Computer Corporation||Method and apparatus for entering and manipulating spreadsheet cell data|
|US5732152||Jan 16, 1997||Mar 24, 1998||Fujitsu Limited||Handwritten character entry method and device having two display areas|
|US5862256||Jun 14, 1996||Jan 19, 1999||International Business Machines Corporation||Distinguishing gestures from handwriting in a pen based computer by size discrimination|
|US5903666||Aug 28, 1997||May 11, 1999||Motorola, Inc.||Methods of splitting and joining handwritten input|
|US6002799 *||Jan 23, 1995||Dec 14, 1999||Ast Research, Inc.||Handwritten keyboardless entry computer system|
|US6005973||Jul 22, 1997||Dec 21, 1999||Motorola, Inc.||Combined dictionary based and likely character string method of handwriting recognition|
|US6167411||Jun 22, 1998||Dec 26, 2000||Lucent Technologies Inc.||User interface for entering and editing data in data entry fields|
|US6661409||Aug 22, 2001||Dec 9, 2003||Motorola, Inc.||Automatically scrolling handwritten input user interface for personal digital assistants and the like|
|US7062089||Nov 20, 2002||Jun 13, 2006||Fujitsu Limited||Handwritten character input assistant apparatus and method|
|US7137076||Jul 30, 2002||Nov 14, 2006||Microsoft Corporation||Correcting recognition results associated with user input|
|US20040161151||Jan 28, 2003||Aug 19, 2004||Fujitsu Limited||On-line handwritten character pattern recognizing and editing apparatus and method, and recording medium storing computer-executable program for realizing the method|
|US20050152602||Jan 14, 2004||Jul 14, 2005||International Business Machines Corporation||Method and apparatus for scaling handwritten character input for handwriting recognition|
|US20050175242||Apr 7, 2005||Aug 11, 2005||Fujitsu Limited||Online handwritten character input device and method|
|US20060033719||Oct 14, 2005||Feb 16, 2006||Leung Paul C P||System and method of pen-based data input into a computing device|
|US20070154093||Dec 30, 2005||Jul 5, 2007||Dunton Randy R||Techniques for generating information using a remote control|
|JPS59163685A||Title not available|
|1||http://www.penandinternet.com/piweb/news/pressroom/Imagesfs.asp , "Pen&Internet's Next-Generation Handwriting Recogniation Solutions Enhance Productivity of New Computing Paradigm".|
|2||John F. Pitrelli, et al., "Confidence modeling for handwriting recognition: algorithms and applications", International Journal of Document Analysis (2006) 8(1): 35-36, pp. 35-46.|
|3||John F. Pitrelli, et al., "Creating word-level language models for large-vocabulary handwriting recognition", International Journal on Document Analysis and Recognition (revised version: Jul. 22, 2002), 5:pp. 126-137.|
|4||Kai-Tai Tang, et al., "Teaching Chinese Handwriting by Automatic Feedback and Analysis for Incorrect Stroke Sequence and Stroke Production Errors", Learning by Effective Utilization of Technologies: Facilitating Intercultural Understanding, R. Mizoguchi et al. (Eds.), IOS Press, 2006; Department of Computer Science, City University of Hong Kong, Hong Kong SAR, pp. 107-114.|
|5||U.-V. Marti, et al., "Text Line Segmentation and Word Recognition in a System for General Writer Independent Handwriting Recognition", Institu für Informatik und angewandte Mathematick Universität Bern, Neubrükstrasse 10, CH-3012 Bern, Switzerland, pp. 159-163.|
|6||Wolfgang Hürst, et al., "Error Repair in Human Handwriting-An Intelligent User Interface for Automatic On-Line Handwriting Recognition", Interactive System Laboratories, Carnegie Mellon University, Pittsburgh, PA, U.S.A.; University of Karlsruhe, Germany.|
|7||Xugang Wang, et al., "Multimodal Error Correction for Continuous Handwriting Recognition in Pen-based User Interfaces", IUI'06, Jan. 29-Feb. 1, 2006, Syndney, Australia, pp. 324-326.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US8310461 *||May 13, 2010||Nov 13, 2012||Nuance Communications Inc.||Method and apparatus for on-top writing|
|US8378980 *||Feb 12, 2009||Feb 19, 2013||Acer Incorporated||Input method using a touchscreen of an electronic device|
|US8610672 *||Apr 10, 2008||Dec 17, 2013||Nokia Corporation||Device and method for stroke based graphic input|
|US8884905||Oct 12, 2012||Nov 11, 2014||Nuance Communications Inc.||Method and apparatus for on-top writing|
|US9110587 *||Jul 12, 2013||Aug 18, 2015||Samsung Electronics Co., Ltd.||Method for transmitting and receiving data between memo layer and application and electronic device using the same|
|US9111139||Jun 9, 2014||Aug 18, 2015||Nuance Communications Inc.||Method and apparatus for on-top writing|
|US9201507 *||Nov 15, 2005||Dec 1, 2015||Carefusion 303, Inc.||System and method for rapid input of data|
|US9323456 *||Aug 3, 2011||Apr 26, 2016||Intsig Information Co., Ltd||Multi-character continuous handwriting input method|
|US9329778 *||Sep 7, 2012||May 3, 2016||International Business Machines Corporation||Supplementing a virtual input keyboard|
|US20070109325 *||Nov 15, 2005||May 17, 2007||Eveleigh Simon F||System and method for rapid input of data|
|US20090256808 *||Apr 10, 2008||Oct 15, 2009||Nokia Corporation||Device and method for stroke based graphic input|
|US20100007617 *||Jan 14, 2010||Chieh-Chih Tsai||Input method using a touchscreen of an electronic device|
|US20110279379 *||May 13, 2010||Nov 17, 2011||Jonas Morwing||Method and apparatus for on-top writing|
|US20130314363 *||Aug 3, 2011||Nov 28, 2013||Intsig Information Co., Ltd.||Multi-character continuous handwriting input method|
|US20140015782 *||Jul 12, 2013||Jan 16, 2014||Samsung Electronics Co. Ltd.||Method for transmitting and receiving data between memo layer and application and electronic device using the same|
|US20140075367 *||Sep 7, 2012||Mar 13, 2014||International Business Machines Corporation||Supplementing a Virtual Input Keyboard|
|US20150193141 *||Sep 30, 2014||Jul 9, 2015||Apple Inc.||Continuous Handwriting UI|
|CN102243570A *||May 13, 2011||Nov 16, 2011||微差通信公司||Method and apparatus for on-top writing|
|CN102243570B||May 13, 2011||Aug 6, 2014||微差通信公司||Method and apparatus for on-top writing|
|U.S. Classification||345/173, 715/700|
|Cooperative Classification||G06F3/04883, G06K9/00436|
|European Classification||G06F3/0488G, G06K9/00K3U|
|Nov 21, 2007||AS||Assignment|
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:RATZLAFF, EUGENE H.;CHEE, YI-MIN;REEL/FRAME:020142/0928
Effective date: 20071115
|Oct 22, 2012||REMI||Maintenance fee reminder mailed|
|Mar 10, 2013||LAPS||Lapse for failure to pay maintenance fees|
|Apr 30, 2013||FP||Expired due to failure to pay maintenance fee|
Effective date: 20130310