Claims
The invention claimed is:
1. A noise reduction method comprising:
a step of generating a confidence index for each band based on a spectrum of observation speech by generating a first confidence index and a second confidence index;
a step of estimating a clean speech estimated value by executing MMSE estimation on a probability model of clean speech generated on the basis of the observation speech, and, for each band, weighting posterior probability of the MMSE estimation using the confidence index as a weight, by executing MMSE estimation on a probability model of clean speech generated on the basis of the observation speech, and, for each band, weighting posterior probability of the MMSE estimation using the first confidence index as a weight, and
a step of deriving an output value by executing, for each band, linear interpolation between an observed value of the observation speech and the clean speech estimated value, with the second confidence index serving as a weight, and
wherein the step of generating a first confidence index and a second confidence index includes:
a step of extracting a harmonic structure from the spectrum of the observation speech and normalizing the harmonic structure;
a step of smoothing normalized values on a mel scale;
a step of generating the first confidence index by normalizing the smoothed values so that a mean of the smoothed values becomes 1; and
a step of generating the second confidence index by normalizing the smoothed values from 0 to 1.
2. The method according to claim 1 , wherein the confidence index is an index representing remarkableness of a harmonic structure of the observation speech.
3. A noise reduction apparatus comprising:
a confidence index generating unit configured to generate a confidence index for each band on the basis of a spectrum of observation speech by generating a first confidence index and a second confidence index; and
a noise reduction unit configured to estimate a clean speech estimated value by executing MMSE estimation on a probability model of clean speech generated on the basis of the observation speech, and, for each band, to weight posterior probability of the MMSE estimation using the confidence index as a weight by executing MMSE estimation on a probability model of clean speech generated on the basis of the observation speech, and, for each band, weighting posterior probability of the MMSE estimation using the first confidence index as a weight, and
a derivation unit configured to derive an output value by executing, for each band, linear interpolation between an observed value of the observation speech and the clean speech estimated value, with the second confidence index serving as a weight, and
wherein the confidence index generating unit generates a first confidence index and a second confidence index by:
extracting a harmonic structure from the spectrum of the observation speech and normalizing the harmonic structure;
smoothing normalized values on a mel scale;
generating the first confidence index by normalizing the smoothed values so that a mean of the smoothed values becomes 1; and
generating the second confidence index by normalizing the smoothed values from 0 to 1.
4. The apparatus according to claim 3 , wherein the confidence index is an index representing remarkableness of a harmonic structure of the observation speech.
FIELD OF THE INVENTION
The present invention relates to speech recognition technology, and more specifically, it relates to noise reduction technology.
BACKGROUND OF THE INVENTION
In speech recognition technology, removal of effects of background noise is a matter of concern that is important in improving the accuracy of utterance recognition. Filtering techniques of the related art (such as a spectral subtraction method and Wiener filtering) have a measure of success when background noise is relatively small, but fail to achieve desired results over large background noise since targeted speech is drowned out by the noise.
To this end, attention has been paid in recent years to approaches that use a probability model of clean speech (utterance speech on which no noise whatsoever is superimposed). These modelbased noise reduction methods have been reported to show high performance even over loud background noise. In this regard, Japanese Patent Application Publication No. 2008298844 discloses a speech recognition system adopting modelbased noise reduction.
SUMMARY OF INVENTION
An object of the present invention is to provide a novel technique of noise reduction applied to a speech recognition frontend.
The inventors have diligently examined a noise reduction method applied to a speech recognition frontend, and as a result, conceived a configuration in which a confidence index representing the remarkableness of the harmonic structure of observation speech is given as a weight to an output for each band, thereby accomplishing the present invention.
That is, according to the present invention, there is provided a first method, which is a noise reduction method, including: the step of generating a confidence index for each band on the basis of a spectrum of observation speech; and the step of estimating a clean speech estimated value by executing MMSE estimation on a probability model of clean speech generated on the basis of the observation speech, and, for each band, weighting posterior probability of the MMSE estimation using the confidence index as a weight.
Also, according to the present invention, there is provided a second method, which is a noise reduction method, including: the step of generating a confidence index for each band on the basis of a spectrum of observation speech; the step of estimating a clean speech estimated value from the observation speech; and the step of deriving an output value by executing, for each band, linear interpolation between an observed value of the observation speech and the clean speech estimated value, with the confidence index serving as a weight.
Further, according to the present invention, there is provided a third method combining the first method and the second method described above, a computer program product for causing a computer to execute each of the methods, and a noise reduction apparatus implementing functions for realizing each of the methods.
As described above, according to the present invention, the output of the frontend is optimized by giving, as a weight, the confidence index representing the remarkableness of the harmonic structure of observation speech to the output for each band.
BRIEF DESCRIPTION OF DRAWINGS
FIG. 1 is a conceptual diagram illustrating the configuration of a frontend in a speech recognition system of the related art.
FIG. 2 includes conceptual diagrams illustrating the steps of generating a mel LPW from the frequency spectrum of observation speech.
FIG. 3 includes conceptual diagrams illustrating the step of generating a confidence index α from a mel LPW.
FIG. 4 is a conceptual diagram illustrating the configuration of a frontend in a speech recognition system of a first embodiment.
FIG. 5 is a flowchart illustrating a process executed by the frontend of the first embodiment.
FIG. 6 includes conceptual diagrams illustrating the step of generating a confidence index β from a mel LPW.
FIG. 7 is a conceptual diagram illustrating the configuration of a frontend in a speech recognition system of a second embodiment.
FIG. 8 is a flowchart illustrating a process executed by the frontend of the second embodiment.
FIG. 9 is a conceptual diagram illustrating the configuration of a frontend in a speech recognition system of a third embodiment.
FIG. 10 is a flowchart illustrating a process executed by the frontend of the third embodiment.
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, the present invention will be described with reference to embodiments illustrated in the drawings. However, the present invention is not limited to the embodiments illustrated in the drawings. Note that, in the drawings to be referred to below, common elements are given the same reference numerals, and descriptions thereof are appropriately omitted.
In general, a speech recognition system includes a frontend that applies certain conversion to an acoustic signal of uttered speech to extract a feature amount, and a backend that searches a database on the basis of the feature amount, which is extracted in the frontend, to specify the contents of the utterance. The present invention is an improvement of noise reduction that has been applied so far to the frontend and aims to optimize the output of the frontend by utilizing confidence for each band of observation speech.
Hereinafter, the present invention will be described on the basis of three embodiments. A first embodiment discloses a technique of improving the estimation of an MMSE (Minimum Mean Square error) used in modelbased noise compensation. A second embodiment discloses a technique of optimizing an output value by implementing linear interpolation between an estimated value of clean speech and an observed value. Further, a third embodiment discloses a technique combining the abovedescribed two techniques.
ModelBased Noise Compensation
Before starting the description of the embodiments of the present invention, modelbased noise compensation applied to the frontend will be described on the basis of FIG. 1 . Note that, in the following description, “observation speech” refers to the actual observed sound in which background noise is superimposed on utterance speech; “clean speech” refers to utterance speech on which no noise whatsoever is superimposed; and a “clean speech estimated value” refers to an estimated value of clean speech estimated from the observation speech. Also, a “frequency spectrum” refers to a power spectrum or an amplitude spectrum.
Modelbased noise compensation is, in short, technology of modeling clean speech x as a probability distribution p(xy) of x when an observed value y is given, and estimating an estimated value of the clean speech x from the probability model p(xy). MMSE estimation is elemental technology used in estimation at a subsequent stage.
FIG. 1 conceptually illustrates the configuration of a frontend 500 in a speech recognition system of the related art to which modelbased noise compensation is applied.
Observation speech collected and recorded with a microphone is converted by an A/D converter to a digital signal (acoustic signal). Thereafter, the digital signal is subjected to framing and discrete Fourier transform (DFT) using an appropriate window function, which is then converted to a frequency spectrum for each frame. Next, the frequency spectrum is subjected to a mel filter bank (a filter bank in which bandpass filters are arranged at equal intervals on the mel scale), and the logs thereof are taken, which are then converted to a mel log spectrum and input to a noise reduction unit 502 .
The noise reduction unit 502 includes a modelbased noise compensation part 512 , an MMSE estimation part 514 , and a database 516 that stores a Gaussian mixture model (hereinafter referred to as a GMM) of clean speech. On the basis of the input mel log spectrum, the noise reduction unit 502 generates a clean speech estimated value for each frame, and outputs the clean speech estimated value to a feature amount extracting unit 504 .
The modelbased noise compensation part 512 is a function part that represents clean speech included in observation speech as a probability distribution, and is implemented by an existing modeling algorithm, such as VTS, SPLICE, DNA, or Segura. The modelbased noise compensation part 512 executes a certain arithmetic operation by using the mel log spectrum for each frame, which is input to the noise reduction unit 502 , and the clean speech model (GMM) stored in the database 516 , and outputs the clean speech included in the observation speech as a mixture multidimensional normal distribution with a dimension for each band.
The MMSE estimation part 514 executes MMSE estimation based on the probability distribution output from the modelbased noise compensation part 512 , and generates a clean speech estimated value. Note that the clean speech model stored in the database 516 is a GMM in the mel log spectrum domain, which is generated for each phoneme on the basis of prior leaning. The modelbased noise compensation part 512 outputs clean speech as a GMM in the mel log spectrum domain. The MMSE estimation part 514 generates a clean speech estimated value as a vector in the mel log spectrum domain.
The feature amount extracting unit 504 extracts a certain feature amount such as a mel frequency cepstrum coefficient (MFCC) from the clean speech estimated value output from the MMSE estimation part 514 , and sends the feature amount to the backend. In the backend, the contents of the utterance are specified on the basis of the feature amount received from the frontend, by using an existing configuration such as an HMM, an acoustic model, or an Ngram language model.
The configuration of the frontend in the speech recognition system of the related art has been briefly described above. Next, the existing MMSE estimation will be described in detail. Modelbased noise compensation algorithms of the related art include a first method of directly estimating a clean speech estimated value from observation speech, which has been described above, and a second method of estimating a compensation amount from observation speech and deriving a clean speech estimated value from the difference between the observation speech and an estimated value of the compensation amount. In the second method, MMSE estimation is used in estimating a compensation amount. The following description will be based on the second method, which is adopted by Segura, in order to simplify the description.
Existing MMSE Estimation
An observed value y d (t) in the mel log spectrum domain of a band d (frequency band on the melscale) in a frame t of observation speech y can be expressed as a function of a clean speech value x d (t) and a noise value n d (t) in equation (1) below:
[Eq. 1]
y d ( t )= x d ( t )+log(1+exp( n d ( t )− x d ( t ))) (1)
When t is omitted from equation (1) above and equation (1) is represented as a vector, equation (2) below is obtained:
[Eq. 2]
y=x+g (2)
A mismatch vector g for each band d is given by a mismatch function G indicated in equation (3) below:
[Eq. 3]
g d =G d ( x,n )=log(1+exp( n d −x d )) (3)
Here, clean speech x is modeled as a Kmixture GMM indicated in equation (4) below:
[
Eq
.
4
]
p
(
x
)
=
∑
k
K
γ
k
·
N
(
x
;
μ
x
,
k
,
Σ
x
,
k
)
(
4
)
In equation (4) above, γ k , μ x,k , and Σ x,k indicate a prior probability, a mean vector, and a covariance matrix, respectively, of a kth normal distribution.
By using the linear Taylor expansion on the basis of equations (1) to (4) above, the mismatch vector g is modeled as a Kmixture GMM indicated in equation (5) below:
[
Eq
.
5
]
p
(
g
)
=
∑
k
K
γ
k
·
N
(
g
;
μ
g
,
k
,
Σ
g
,
k
)
(
5
)
Note that the mean vector μ g,k in equation (5) above is represented as equation (6) below, and the covariance matrix Σ g,k is represented as equation (7) below:
[Eq. 6 and 7]
μ g,k ≅log(1+exp(μ n −μ x,k ))= G (μ x,k ,μ n ) (6)
Σ g,k ≅F (μ x,k ,μ n ) 2 ·(Σ x,k +Σ n ) (7)
An auxiliary function F in equation (7) above is defined as equation (8) below:
[Eq. 8]
F d ( x,n )=(1+exp( x d −n d )) −1 (8)
As a result, the clean speech estimated value x^ is given by equation (91) below:
[
Eq
.
9

1
]
x
^
=
y

∫
g
·
p
(
g

y
)
ⅆ
g
≅
y

∑
k
K
ρ
k
(
y
)
·
μ
g
,
k
(
9

1
)
In contrast, in the first method of directly estimating the clean speech estimated value x^ from the observation speech y, the clean speech estimated value x^ is given by equation (92) below instead of equation (91) above:
[
Eq
.
9

2
]
x
^
=
∫
x
·
p
(
x

y
)
ⅆ
x
≅
∑
k
K
ρ
k
(
y
)
·
μ
x
,
k
(
9

2
)
Here, posterior probabilities ρ k in equations (91) and (92) above are both given by equation (10) below:
[
Eq
.
10
]
ρ
k
(
y
)
=
γ
k
·
N
(
y
;
μ
y
,
k
,
Σ
y
,
k
)
/
∑
k
′
γ
k
′
·
N
(
y
;
μ
y
,
k
′
,
Σ
y
,
k
′
)
(
10
)
The mean vector μ y,k in equation (10) above is represented by equation (11) below, and the covariance matrix Σ y,k is represented by equation (12) below:
[Eq. 11 and 12]
μ y,k ≅μ x,k +G (μ x,k ,μ n ) (11)
Σ y,k ≅{1 −F (μ x,k ,μ n ) 2 }·Σ x,k +F (μ x,k ,μ n ) 2 ·Σ n (12)
Note that, in equations (11) and (12) above, speech model parameters [μ x,k , Σ x,k ] are given by prior learning data, and noise model parameters [μ n , Σ n ] are set by the modelbased noise compensation part 512 on the basis of an observed value in a nonspeech segment and given to the MMSE estimation part 514 .
As has been described above, MMSE estimation is, in short, a process of approximating the clean speech estimated value x^ as the sum of mean vectors μ x,k of k probability distributions weighted using the posterior probability ρ k (y) as a weight. The first embodiment of the present invention pays attention to this point and optimizes the output of the frontend by appropriately modifying the posterior probability in MMSE estimation.
CWMMSE
In this section, a technique of weighting the posterior probability in MMSE estimation by using a confidence index as a weight will be described. In the following description, this technique is referred to as CWMMSE (Confidence Weighted MMSE).
In existing MMSE, the clean speech estimated value x^ is given by equation (91) or (92) below, and the posterior probability ρ k (y) in each equation is given by equation (10) below, as has been described above.
x
^
=
y

∫
g
·
p
(
g

y
)
ⅆ
g
≅
y

∑
k
K
ρ
k
(
y
)
·
μ
g
,
k
(
9

1
)
x
^
=
∫
x
·
p
(
x

y
)
ⅆ
x
≅
∑
k
K
ρ
k
(
y
)
·
μ
x
,
k
(
9

2
)
ρ
k
(
y
)
=
γ
k
·
N
(
y
;
μ
y
,
k
,
Σ
y
,
k
)
/
∑
k
′
γ
k
′
·
N
(
y
;
μ
y
,
k
′
,
Σ
y
,
k
′
)
(
9

3
)
Here, CWMMSE uses, in equation (91) or (92) above, which gives the clean speech estimated value x^ a posterior probability ρ′ k (y) weighted using a confidence index α d , instead of the posterior probability ρ k (y), as a weight. Equation (13) below indicates the posterior probability ρ′ k (y) used by CWMMSE:
[
Eq
.
13
]
ρ
k
′
(
y
)
=
γ
k
·
N
′
(
y
;
μ
y
,
k
,
Σ
y
,
k
)
/
∑
k
′
γ
k
′
·
N
′
(
y
;
μ
y
,
k
′
,
Σ
y
,
k
′
)
(
13
)
Here, the normal distribution in equation (13) above is given by equation (14) below, which uses a diagonal covariance assumption. Note that, in equation (14) below, D indicates the number of dimensions of the normal distribution:
[Eq. 14]
N ′( y;μ y,k ,Σ y,k )=Π d=1 D {(2π)−½·Σ y,k,d −½·exp(−( y d −μ y,k,d ) 2 /2Σ y,k,d )} α d (14)
With reference to equation (14) above, it can be understood that a normal distribution N′, which is a term in an equation for calculating the posterior probability ρ′ k (y), is multiplied by an index using the confidence index α d as a weight. Here, the confidence index is an index that represents the confidence of a band. In general, the confidence of a band is evaluated from the viewpoint of signal deterioration caused by background noise. In the present invention, the confidence index is defined as follows.
Confidence Index
It is known that the frequency spectrum of vowels included in human utterance has a harmonic structure. In an environment with no background noise, the harmonic structure of vowels is maintained in the entire band of the frequency spectrum of observation speech. In contrast, over broadband noise such as a vehicle's driving sound or air conditioning sound, the harmonic structure of vowels is lost in many bands, and the harmonic structure is maintained only in a band such as a formant where the speech power is concentrated.
From this point, the present invention assumes that deterioration caused by background noise occurs infrequently in a band with a remarkable harmonic structure, and defines the remarkableness of the harmonic structure as a confidence index for that band. Hereinafter, a method of generating a confidence index of the present invention will be described.
A confidence index of the present invention is generated using an LPW (Local Peak Weight). Here, an LPW is such that a great change such as that including formant information is removed from the spectral energy distribution of observation speech, only regular peaks and troughs corresponding to the harmonic structure are extracted, and their values are normalized. In the present invention, an LPW is generated for each frame by performing the following procedure.
Firstly, the logarithm of the frequency spectrum of a frame t of observation speech is taken, and its log spectrum is subjected to discrete cosine transform to obtain a cepstrum. Next, among terms of the obtained cepstrum, only terms in a domain(s) corresponding to the harmonic structure of LPW vowels are left, and the other terms are cut. Thereafter, the processed cepstrum is subjected to inverse discrete cosine transform to convert the cepstrum back to the log spectrum domain, which is further converted back to the frequency spectrum domain. Finally, the converted frequency spectrum is normalized so that the mean of the frequency spectrum becomes 1, thereby obtaining an LPW. FIG. 2( a ) illustrates the frequency spectrum of a frame t of observation speech. FIG. 2( b ) illustrates an LPW generated on the basis of the frequency spectrum illustrated in FIG. 2( a ).
Next, a mel LPW illustrated in FIG. 2( c ) is obtained by smoothing the LPW illustrated in FIG. 2( b ) on the mel scale. Specifically, values are smoothed by subjecting the LPW to a mel filter bank (a filter bank in which bandpass filters are arranged at equal intervals on the mel scale), and one value is obtained for each mel band. As illustrated in FIG. 2( c ), the mel LPW value is given in each mel band. The magnitude of the mel LPW value corresponds to the remarkableness of the harmonic structure of a frequency spectrum band at high resolution, which corresponds to each mel band.
In the present invention, the mel LPW value that is normalized by an appropriate method is used as a confidence index. In the case of CWMMSE, the confidence index α d in equation (14) above can be generated by the following procedure.
Firstly, in order that the confidence index α d does not take an extreme value, the dynamic range of a mel LPW illustrated in FIG. 3( a ) is compressed by using an appropriate scaling function. As a result, the mel LPW value w d of each band is converted to α′ d . An example of the scaling function includes a sigmoid function. Equation (15) below indicates an equation for converting the mel LPW value w d to α′ d by using a sigmoid function:
[Eq. 15]
α′ d =1.0/(1.0+exp(−α·( w d −1.0))) (15)
Note that, in equation (15) above, a is a tuning parameter, and an appropriate constant value is set.
Next, the compressed value α′ d is normalized so that the mean becomes 1. Normalization such that the mean becomes 1 is done because the confidence index α d is used as an index weight in equation (14) above. Equation (16) below indicates an equation for normalizing α′ d and obtaining the confidence index α d :
[
Eq
.
16
]
α
d
=
α
d
′
/
(
1
D
∑
d
′
α
d
′
′
)
(
16
)
FIG. 3( b ) illustrates the confidence index α d generated by the abovedescribed procedure. As illustrated in FIG. 3( b ), the confidence index α d is given in each mel band.
In the case of a voiceless section or an unvoiced section, the confidence index α d of a band d in a frame t will uniformly approach 1. In this case, the normal distribution N′ in equation (14) above becomes a general normal distribution. Thus, the posterior probability ρ′ k (y) is not weighted.
When there is a spectral band where the harmonic structure of vowels is remarkable in a frame t of a voiced section, the confidence index α d of a corresponding band d will become greater than 1. As a result, regarding the band d, the normal distribution N′ in equation (14) above becomes great, and the posterior probability ρ′ k (y) of the band d becomes great. This means that, in MMSE estimation executed by CWMMSE, the contribution of a mel band corresponding to the spectral band where the harmonic structure of vowels is remarkable becomes great.
In contrast, when there is a spectral band where the harmonic structure of vowels is lost in the frame t of the voiced section, the confidence index α d of the corresponding band d will become smaller than 1. As a result, regarding the band d, the normal distribution N′ in equation (14) above becomes small, and the posterior probability ρ′ k (y) of the band d becomes small. This means that, in MMSE estimation executed by CWMMSE, the contribution of a mel band corresponding to the spectral band where the harmonic structure of vowels is lost becomes small.
FIG. 4 conceptually illustrates the configuration of a frontend 1000 in the speech recognition system according to the first embodiment. A noise reduction unit 100 of the frontend 1000 includes a CWMMSE estimation part 102 implementing CWMMSE instead of existing MMSE. The frontend 1000 further includes a confidence index generating unit 104 . The confidence index generating unit 104 includes an LPW generating part 105 , a mel LPW generating part 106 , and a normalizing part 107 . Hereinafter, a process executed by the frontend 1000 will be described on the basis of a flowchart illustrated in FIG. 5 . Note that, in the following description, reference will be appropriately made to FIG. 4 .
Firstly, when the frequency spectrum Y corresponding to one frame of observation speech is obtained (step S 101 ), the LPW generating part 105 extracts the harmonic structure of the frequency spectrum Y and generates an LPW (step S 102 ). Next, the mel LPW generating part 106 generates a mel LPW from the LPW (step S 103 ). The normalizing part 107 normalizes the mel LPW with an appropriate method, and generates a confidence index α (the mean of values is 1) for each band (step S 104 ). The confidence index generating unit 104 provides the generated confidence index α to the CWMMSE estimation part 102 of the noise reduction unit 100 .
Thereafter, the frequency spectrum Y corresponding to one frame is converted to a mel log spectrum y (step S 105 ) and is input to the noise reduction unit 100 . The noise reduction unit 100 estimates clean speech by using the input mel log spectrum y and the confidence index α provided from the confidence index generating unit 104 . At this time, the CWMMSE estimation part 102 weights the posterior probability of MMSE estimation using the confidence index α as a weight (step S 106 ).
The noise reduction unit 100 outputs a clean speech estimated value x^ estimated by the CWMMSE estimation part 102 to the feature amount extracting unit 504 (step S 107 ). The feature amount extracting unit 504 extracts a certain feature amount from the clean speech estimated value x^ and sends the feature amount to the backend. The abovedescribed series of procedures is repeatedly executed for each frame of the observation speech (No in step 108 ). The process ends when the last frame is reached (Yes in step 108 ).
As has been described above, with CWMMSE, each band influences the result of MMSE estimation, with a degree of contribution in accordance with the level of its confidence. As a result, the output of the frontend is optimized.
The first embodiment has been described above. Next, a second embodiment in which the output value is optimized by implementing linear interpolation between a speech modal and observation speech will be described.
CWINT
If observation speech is equivalent to clean speech, the best thing to do is to output the observation speech without processing it. However, with a modelbased approach, even in such a case, a model of clean speech is estimated, and consequently, the output becomes more deteriorated than the clean speech. The second embodiment discloses a technique of solving this problem.
In this section, a technique of implementing linear interpolation between a speech model and observation speech, with a confidence index as a weight, will be described. In the following description, this technique is referred to as CWINT (Confidence Weighted Interpolation).
In CWINT, an output value x{tilde over ( )} d in a band d is given by a linear interpolation function indicated in equation (17) below:
[Eq. 17]
{tilde over (x)} d =(1.0−β d )· {circumflex over (x)} d +β d ·y d (17)
In equation (17) above, x^ d indicates a clean speech estimated value in the band d, β d indicates a confidence index given to the band d, y d indicates an observed value in the band d, and x{tilde over ( )} d indicates the output value in the band d. The linear interpolation function indicated in equation (17) above is weighted using the confidence index β d as a weight, which takes a value from 0 to 1. The linear interpolation function indicates that, as β d approaches 1, the output value x{tilde over ( )} d approaches the observed value y d , and, as β d approaches 0, the output value x{tilde over ( )} d approaches the clean speech estimated value x^ d .
Here, the confidence index of the present invention is generated by normalizing the mel LPW value by an appropriate method, as has been described above. In the case of CWINT, the confidence index β d in equation (17) above can be generated by the following procedure.
FIG. 6( a ) illustrates a mel LPW obtained for a frame t. In the case of CWINT, the mel MPW value w d is normalized by using an appropriate scaling function so that the value w d takes a value from 0 to 1 where 1 is the maximum value. An example of the scaling function includes a sigmoid function. Equation (18) below indicates an equation for normalizing the mel LPW value w d by using a sigmoid function and obtaining the confidence index β d :
[Eq. 18]
β d =1.0/(1.0+exp(−α·( w d −1.0 −b ))) (18)
Note that, in equation (18) above, a and b are tuning parameters, and appropriate constant values are set.
FIG. 6( b ) indicates the confidence index β d generated by the abovedescribed procedure. As illustrated in FIG. 6( b ), the confidence index β d is given in each mel band.
When there is a spectral band where the harmonic structure of vowels is remarkable in a frame t of a voiced section, the confidence index β d of a corresponding band d approaches 1. As a result, the output value x{tilde over ( )} d in the band d, which is the result of linear interpolation indicated in equation (17) above, will become closer to the observed value y d than to the clean speech estimated value x^ d .
In contrast, when there is a spectral band where the harmonic structure of vowels is lost in the frame t of the voiced section, the confidence index β d of the corresponding band d approaches 0. As a result, the output value x{tilde over ( )} d in the band d, which is the result of linear interpolation indicated in equation (17) above, will become closer to the clean speech estimated value x^ d than to the observed value y d .
FIG. 7 conceptually illustrates the configuration of a frontend 2000 in a speech recognition system according to the second embodiment. A noise reduction unit 503 included in the frontend 2000 is a function unit that generates and outputs a clean speech estimated value (vector in the mel log spectrum domain) on the basis of an input of the mel log spectrum for each frame of observation speech. In the case of the second embodiment, the noise reduction unit 503 may adopt modelbased noise compensation, as in the noise reduction unit 502 illustrated in FIG. 1 , or may adopt a filtering technique, such as a spectral subtraction technique. The frontend 2000 further includes the confidence index generating unit 104 and a linear interpolation unit 200 . The confidence index generating unit 104 includes the LPW generating part 105 , the mel LPW generating part 106 , and a normalizing part 108 . Hereinafter, a process executed by the frontend 2000 will be described on the basis of a flowchart illustrated in FIG. 8 . Note that, in the following description, reference will be appropriately made to FIG. 7 .
Firstly, when the frequency spectrum Y corresponding to one frame of observation speech is obtained (step S 201 ), the LPW generating part 105 extracts the harmonic structure of the frequency spectrum Y and generates an LPW (step S 202 ). Next, the mel LPW generating part 106 generates a mel LPW from the LPW (step S 203 ). The normalizing part 108 normalizes the mel LPW with an appropriate method, and generates a confidence index β (value is from 0 to 1) for each band (step S 204 ).
Thereafter, the frequency spectrum Y corresponding to one frame is converted to a mel log spectrum y (step S 205 ). The converted mel log spectrum y is input to both the linear interpolation unit 200 and the noise reduction unit 503 . The noise reduction unit 503 estimates clean speech from the mel log spectrum y by using an appropriate method, and outputs a clean speech estimated value x^ to the linear interpolation unit 200 (step S 206 ).
The linear interpolation unit 200 executes, for each band, linear interpolation between the vector of the mel log spectrum y and the clean speech estimated value x^ (vector in the mel log spectrum domain) input from the noise reduction unit 503 . At this point, the linear interpolation unit 200 implements linear interpolation with the confidence index β as a weight and derives an output value x{tilde over ( )} (step S 207 ).
The linear interpolation unit 200 outputs the derived output value x{tilde over ( )} to the feature amount extracting unit 504 (step S 208 ). The feature amount extracting unit 504 extracts a certain feature amount from the output value x{tilde over ( )} and sends the feature amount to the backend. The abovedescribed series of procedures is repeatedly executed for each frame of the observation speech (No in step 209 ). The process ends when the last frame is reached (Yes in step 209 ).
As has been described above, with CWINT, the observed value in each band is reflected in the output value in accordance with the level of its confidence. As a result, the output of the frontend is optimized.
The second embodiment has been described above. At last, a third embodiment combining the first embodiment and the second embodiment will be described.
CWMMSE+CWINT
In this section, a technique combining CWMMSE and CWINT, which have been described so far, will be described. In the following description, this technique is referred to as CWMMSE+CWINT.
FIG. 9 conceptually illustrates the configuration of a frontend 3000 in a speech recognition system according to the third embodiment. The frontend 3000 includes the noise reduction unit 100 including the CWMMSE estimation part 102 , the linear interpolation unit 200 (CWINT), the LPW generating part 105 , the confidence index generating unit 104 , and the mel LPW generating part 106 . Hereinafter, a process executed by the frontend 3000 will be described on the basis of a flowchart illustrated in FIG. 10 . Note that, in the following description, reference will be appropriately made to FIG. 9 .
Firstly, when the frequency spectrum Y corresponding to one frame of observation speech is obtained (step S 301 ), the LPW generating part 105 extracts the harmonic structure of the frequency spectrum Y and generates an LPW (step S 302 ). Next, the mel LPW generating part 106 generates a mel LPW from the LPW (step S 303 ). The normalizing part 107 normalizes the mel LPW with an appropriate method, and generates a confidence index α (the mean of values is 1) for each band. At the same time, the normalizing part 108 normalizes the mel LPW, and generates a confidence index β (value is from 0 to 1) for each band (step S 304 ). The confidence index generating unit 104 provides the generated confidence index α and confidence index β to the CWMMSE estimation part 102 in the noise reduction unit 100 and the linear interpolation unit 200 (CWINT), respectively.
Thereafter, the frequency spectrum Y corresponding to one frame is converted to a mel log spectrum y (step S 305 ). The converted mel log spectrum y is input to both the linear interpolation unit 200 and the noise reduction unit 100 . The noise reduction unit 100 estimates clean speech by using the input mel log spectrum y and the confidence index α provided from the confidence index generating unit 104 . At this time, the CWMMSE estimation part 102 estimates clean speech by weighting the posterior probability of MMSE estimation using the confidence index α as a weight (step S 306 ), and outputs a clean speech estimated value x^ to the linear interpolation unit 200 .
The linear interpolation unit 200 executes, for each band, linear interpolation between the vector of the mel log spectrum y and the clean speech estimated value x^ (vector in the mel log spectrum domain) input from the noise reduction unit 100 (step S 307 ). At this point, the linear interpolation unit 200 implements linear interpolation with the confidence index β as a weight and derives an output value x{tilde over ( )}.
The linear interpolation unit 200 outputs the derived output value x{tilde over ( )} to the feature amount extracting unit 504 (step S 308 ). The feature amount extracting unit 504 extracts a certain feature amount from the output value x{tilde over ( )} and sends the feature amount to the backend. The abovedescribed series of procedures is repeatedly executed for each frame of the observation speech (No in step 309 ). The process ends when the last frame is reached (Yes in step 309 ).
As has been described above, with CWMMSE+CWINT, the estimated value of MMSE is optimized, and further, optimal interpolation is implemented between the estimated value and the observed value. As a result, the output of the frontend is optimized.
The present invention has been described with reference to the embodiments. However, the present invention is not limited to the abovedescribed embodiments, and changes are included within the scope of the present invention as long as they have operations and advantageous effects of the present invention within the scope of an embodiment conceivable by those skilled in the art.
The functions of the abovedescribed embodiments can be realized with a program product that is written in an objectoriented programming language such as C, C++, C#, or Java (registered trademark) and that is executable on an apparatus. The program product of the embodiments can be stored in an apparatusreadable recording medium, such as a hard disk drive, a CDROM, an MO, a DVD, a flexible disk, an EEPROM, or an EPROM, for distribution. Alternatively, the program product can be transferred via a network in an apparatusreadable format.
Hereinafter, the present invention will be more specifically described using an example. However, the present invention is not limited to the example described below.
Three types of computer program products (CWMMSE, CWINT, and CWMMSE+CWINT) were generated to cause a computer to execute methods of the present invention disclosed above, and the performance of each computer program product was evaluated using an invehicle speech recognition evaluation framework provided by Information Processing Society of Japan (IPSJ).
AN experiment was conducted using the invehicle speech recognition evaluation database CENSREC3. An evaluation condition was Condition 3 that uses a remote microphone for both learning data and test data. A frontend that outputs various feature amounts necessary for the experiment was prepared and was applied to both the learning data and the test data. The feature amounts were in 39 dimensions including MFCC 12 dimensions+ΔMFCC 12 dimensions+ΔΔMFCC 12 dimensions+C0+ΔC0+ΔΔC0, and CMN in units of utterances was applied. The configuration of a backend, such as the method of generating an acoustic model, was unchanged (Category 0).
A GMM of clean speech was learned using data collected and recorded with a closetalking microphone when the vehicle was idle. Feature amounts of the mel log spectrum in 24 dimensions were used, and the number of mixtures was 256.
Table 1 below collectively indicates the word correct accuracy (%) of the three example program products (CWMMSE, CWINT, and CWMMSE+CWINT) and a comparative example program product (standard MMSE: MMSE of Segra). Note that “CENSREC3 Baseline” is a baseline where no speech enhancement or noise reduction is performed.
On the basis of the allenvironment mean of the word correct accuracy (%) indicated in Table 1 above, the error reduction rate (%) with reference to the standard MMSE, which is a comparative example, was calculated using equation (19) below:
[
Eq
.
19
]
Error
reduction
rate
(
%
)
=
Correct
accuracy
rate
(
%
)
of
example

Correct
accuracy
rate
(
%
)
of
standard
MMSE
100

Correct
accuracy
rate
(
%
)
of
standard
MMSE
×
100
(
19
)
TABLE 1
Recording Environment
of Evaluation Data
Word Correct Accuracy (%)
Driving
Invehicle
CENSREC3
Standard
CWMMSE +
Velocity
Environment
Baseline
MMSE
CWMMSE
CWINT
CWINT
Idling
Normal
100.0
100.0
99.9
99.8
99.8
Hazard On
99.4
97.9
98.0
98.2
98.3
Air Conditioning
98.0
98.8
98.8
99.2
99.4
(Low)
Air Conditioning
63.1
81.2
85.2
86.7
88.1
(High)
Open Window
93.1
96.6
96.9
97.6
97.3
Mean
90.7
94.9
95.8
96.3
96.6
Lowspeed
Normal
99.8
98.7
98.9
98.8
99.1
Driving
Air Conditioning
96.8
97.8
98.0
98.5
97.8
(Low)
Air Conditioning
69.3
84.5
87.4
89.8
89.7
(High)
Open Window
80.8
82.5
85.1
86.7
86.7
Mean
87.5
91.7
93.0
94.0
94.0
Highspeed
Normal
98.1
97.3
97.8
98.3
98.7
Driving
Air Conditioning
94.8
96.2
96.9
97.2
97.8
(Low)
Air Conditioning
64.8
83.8
85.4
88.7
87.4
(High)
Open Window
49.0
61.5
66.2
67.3
68.8
Mean
78.8
86.1
87.9
89.1
89.3
Allenvironment Mean
85.2
90.5
91.9
92.8
93.0
As a result, the error reduction rates (%) with reference to the standard MMSE were 14.4%, 24.4%, and 26.2% in CWMMSE, CWINT, and CWMMSE+CWINT, respectively. The results indicate that the methods of the present invention greatly improve the speech recognition accuracy.
Although exemplary embodiments of the present invention have been described herein with reference to the accompanying figures, it is to be understood that the invention is not limited to those precise embodiments, and that various other changes and modifications may be made therein by one skilled in the art without departing from the scope of the appended claims.