US 20070094169 A1
An adapter for a text to text training. A main corpus is used for training, and a domain specific corpus is used to adapt the main corpus according to the training information in the domain specific corpus. The adaptation is carried out using a technique that may be faster than the main training. The parameter set from the main training is adapted using the domain specific part.
1. A method, comprising:
First carrying out a first training using at least one corpus of language information, using a first training operation to obtain a first parameter set;
second carrying out a second training using a domain specific corpus, using a second training operation which operates faster than said first training operation, and which is less accurate than said first training operation, to obtain a second parameter set; and
using said second parameter set to adapt said first parameter set to carry out a text-to-text operation.
2. A method as in
3. A method as in
4. A method as in
5. A method as in
6. A method as in
7. A method as in
8. A method as in
9. A method as in
10. A method as in
11. A method as in
12. A method as in
13. A method as in
14. A method as in
15. A method as in
16. A method as in
17. A method as in
18. A method as in
19. A method as in
20. A method as in
21. An apparatus, comprising:
a first training computer at a first location, carrying out a first training using at least one corpus of information, using a first training operation, to obtain a first parameter set;
a second training computer, at a second location, different than the first location, carrying out a second training using a domain specific corpus that has different information than said at least one corpus, using a second training operation which operates faster than said first training operation, and which is less accurate than said first training operation, to obtain a second parameter set, and using said first and second parameter sets together for a text-to-text operation.
22. An apparatus as in
23. An apparatus as in
24. An Apparatus as in
25. An Apparatus as in
26. An Apparatus as in
27. An Apparatus as in
28. An Apparatus as in
29. An Apparatus as in
30. An Apparatus as in
31. An apparatus, comprising:
a training part, including at least one computer, which carries out a first training for a text-to-text application using at least one corpus of training information, to obtain a first parameter set, and at a different time than said first training, carrying out a second training using a domain specific corpus that has different information than said at least one corpus, using a second training operation which operates faster than said first training operation, and which is less accurate than said first training operation, to obtain a second parameter set, and using said second parameter set to adapt said first parameter set to create an adapted parameter set, and to use the adapted parameter set for a text-to-text operation.
32. An apparatus as in
33. An apparatus as in
34. An Apparatus as in
35. An Apparatus as in
36. An Apparatus as in
Text-to-text applications, such as machine translation systems, often operate based on training data. A machine translation system may automatically learn from translated documents. The quality of the actual translation is based on the amount and quality of the data, and the precision of the training process. The processing uses a tradeoff between the data quality and the speed of processing.
Machine translation systems learn from language pairs, and thereafter may translate documents from one language in the pair to the other language in the pair. Translation quality may be greatly improved by providing field specific translation information, that is, translation information that is specific to the field of the information that is being translated.
The present application teaches a system which allows a generic training to be done by the translation developer. The translation developer can use a slow but accurate training technique for that generic training. A text to text application system may be done in two parts, generic training done first, with emphasis on accuracy, followed by specific training done with emphasis on speed. The data from the specific training is used to “adapt” the data created by the generic training. The system may be provided to the customer with the generic training already completed. The field—specific training can be done by the customer, as part of a customization process for the translation. The field specific training uses a technique that is faster but less accurate than the generic training.
Techniques are disclosed for the faster training techniques, and for merging the two training parts without completely re-training.
The general structure and techniques, and more specific embodiments which can be used to effect different ways of carrying out the more general goals are described herein.
Under the current system, a generic training is first carried out, and customers may also provide their specific training material. The software build is then customized to the training information. This may customize the software to work better than the generic translation software. However, it was noticed that customers often have data that they do not want to disclose. The data may be proprietary data, or may be classified data. For example, a military customer may have classified data which is available in both languages and could be used as very powerful specific training information. However, security restrictions may not allow that data to leave the customer's control.
According to an embodiment, a generic translation system is provided, which allows system adaptation on the customer's machine which is already running the pre-trained generic translation software. Alternately, the adaptation can be done at the same location as the generic training, to create an adapted parameter set that can be used for a text-to-text operation.
An embodiment is shown in
The data information is also supplemented with user training data shown as 125. The user training data is optimized for use in the specific translation field, and hence is domain specific. The fast training module 130 processes this data. The fast training module 130 may use a different training technique then the in-house training system, and one which is optimized for speed as compared with accuracy. The user training data may include fewer words than the generic training data. The user training data may be between ½ million and 5 million words. The user training system may train 2-10 times faster than the in-house training system 115.
The user training creates the user domain specific parameter base 135.
The parameters are merged using a merge module 140 which creates a merged parameter base 145 that includes information from both generic training data 110 and the user training data 125. The merged parameter base is then used by the text to text application 100, which may be a general purpose computer or processor that is programmed for a text-to-text application, such as translation. A foreign language sentence 150 is input into the translator 100 and converted to an English-language translation 155 or vice versa. The translation is based on both sets of parameter databases 120 and 135.
The system used herein is called an adapter, and relies on two different kinds of adaptation, the so-called off-line adaptation at 115 which uses generic training data, and the online adaptation at 130 which adapts the generic data to the specific environment. The online adaptation uses a fast and light weight word alignment model, similar to the models that are used in real-time speech recognition systems. The online adaptation is then merged with the generic system parameters using a set of mathematical formula that allow for effective parameter merge. The formula calculates approximated parameter values as if they would have been trained if all training with both databases had been trained using the off-line adaptation scheme. This avoids complete retraining of the parameters from generic data, and may speed up the adaptation process.
A previous solution to the issue of a general corpus and a domain corpus, combined the two corpora and completely retrained the system. This was typically done “offline”, since it was a relatively time consuming process. Moreover, this requires disclosure of the user's data, which, as described above, could not always be done. The table merge described herein makes this system more able to operate in the off-line/online model.
The merged parameters 145 are used for the translator 100.
The Parameter Merge Module 140 combines parameters from the generic model 115 and the domain model 130. Each model is trained separately. A generic parameter is trained only from the generic data. This training is done at the developer, and may takes more than two weeks, since the training data is large (over 100 million words). User domain parameter is trained only from user data, which is typically small (less than 1 million words). As two models are trained separately, the adaptation process is quite fast, since only the user parameters need to be trained.
If more computational resources are spent the adaptation, then one way of processing is to combine the generic data and user data, and to run the full training from scratch. It may also-be important to duplicate the user data multiple times so that the user data makes a significant effect on the model. As the data size is quite different (typically, generic data is about 100 million words, and user data is less than 1 million words), such user data duplication may be important. This method (combining the generic and the user data, and training an adaptor from scratch) is called offline adaptation.
The Basic Merge Formula
The following formulas are used for online adaptation.
For a conditional model P(e|f), two models Pg(e|f) and Pd(e|f) are merged as follows:
The formula combines a generic model Pg(e|f) and a domain model Pd(e|f) by a linear mixture weight λf K. However, the different mixture weight is assigned for different model conditioning variable f. The weight is defined by the count of f in the general corpus Cg(f) and in the domain corpus Cd(f). The domain weight factor K is a global constant for all f.
Intuitively, the mixture weight is adjusted according to the word frequencies. If a word f in a generic data, but only observed infrequently in the domain data, the probability P(e|f) should stay the same as or similar to Pg(e|f). On the other hand, if a word f is observed very frequently in domain data, then P(e|f) should be close to Pd(e|f).
This section shows that the formula actually emulates the domain duplication effect. A bigram language model is used as an example here. Notice that the adapter formula is applicable for any conditional model, such as a model-1 translation model and an ngram language model or any other model.
Consider, for example, two bigram language models, Pg(w2|w1) and Pd(w2|w1), that are separately trained from general data and from domain data.
When the domain data is duplicated by K times and concatenated with the general data, and a single language model is trained with this data, such language model Pg+Kd(w2|w1) is:
The last line shows that combining two separate language models Pg(w2|w1) and Pd(w2|w1) with duplicated term weights λw
The model parameter for this translation system may be formed of several different types of parameters. The adaptation formula above is modified differently according to the different types of parameters.
Word Probability Parameters
The formula above is directly applicable to parameters that are in the form of probabilities conditioned on a single word.
As seen in the formula (1), a merged entry P(e|f) is independent from the domain data if its probability Pd(e|f) is zero, that is
This happens when the words e and f do no co-occur in any sentence in the domain corpus.
Therefore, an efficient implementation is the following:
And Cd(e|f) is the number of sentences (in the domain corpus) in which both e and f appear. The size of the override table Pov(e|f) is identical to the word probability table from the domain corpus.
The override table is generated by the formula (12) at the end of the adaptation process. In the adapted version of the decoder, the override table is used in conjunction with the original generic word probability table Pg(e|f) as specified in the formula (11).
The following is a sample implementation of formula (11) and (12):
The OverrideWPTable(e,f) is maybe a fixed-sized array. Each element stores the precomputed value of formula (12). The size of the OverrideWPTable(e,f) is identical to the size of the word probability table from the domain corpus. The weight term W=GenCnt(f)/(GenCnt(f)+K*ComCnt(f)) implements the formula (2).
Phrase Probability Parameters
A simple application is possible to the parameters that in the form of probabilities conditioned on a sequence of words, or a phrase:
This turns out to be a linear model mixture in which the weight is determined by the domain corpus duplication factor K.
A similar approach for word probability parameters can be implemented as:
The following is a exemplary sample implementation of formula (17) and (18):
The OverridePHTable is identical to the one from the domain corpus, except that each entry is replaced with the override value.
2.3.3 Count-Based Parameters
Some model parameters are not in the form of probabilities, but rather are in the form of simple counts. For this type of parameter, the basic formula (1) cannot be applied. Instead, the following formula is applied to have the same effect.
The override table Cov(e,f) is generated by the formula (21) at the end of the adaptation process. In the adapted version of the decoder, the override lexicon table is used in conjunction with the original generic table Cg(e,f) as specified in formula (20).
The following is an exemplary implementation of formula (20) and (21):
The OverrideCTable(e,f) is a fixed-sized array. Each element stores the precomputed value of formula (21). The size of the OverrideCTable(e,f) may be identical to the size of the parameter table from the domain corpus.
Fast Train Module
The Fast Train Module uses a simpler version of the in-house Training System. It may use simple models such as bag-of-words translation models, and/or Hidden Markov models.
Simple Fast Training
The simplest form of the Fast Train Module is just to use such simple models without any additional data from the generic parameters. If the user data is sufficiently large, this scheme works well in practice.
Adaptive Fast Training
To improve the adaptation performance, especially for small user data, the generic parameters are additionally used to train the user data. In general, the training method builds a word-based conditional probability table based on the following formula:
Where cn(e,f) is the alignment count from the parameters at the n-th iteration.
In non-adapter systems, the initial parameter P0(e|f) is usually a uniform parameter. In the adapter environment, the generic parameter Pg(e|f) can be used instead.
Use Alignment Counts from General Corpus
If the generic parameter provides the final alignment counts cg N(e,f) from the general corpus, it can be used in each iteration of the user data training in the following way:
Instead of using cg N(e,f), the generic model table Pg(e|f) can be used in each iteration.
Term Weighted Adaptation
The adaptation formula (1) can alternatively be used as:
The alignment count cg N(e,f), from the general corpus can be approximated using the term frequency count Cg(e).
The operations described herein can be carried out on any computer, e.g., an Intel processor, running Windows and/or Java. Programming may be done in any suitable programming language, including C and Java.
Although only a few embodiments have been disclosed in detail above, other embodiments are possible and are intended to be encompassed within this specification. The specification describes specific examples to accomplish a more general goal that may be accomplished in other way. This disclosure is intended to be exemplary, and the claims are intended to cover any modification or alternative which might be predictable to a person having ordinary skill in the art. For example, other merging and training techniques may be used.
Also, only those claims which use the words “means for” are intended to be interpreted under 35 USC 112, sixth paragraph. Moreover, no limitations from the specification are intended to be read into any claims, unless those limitations are expressly included in the claims.