We train machines to process textual data (automated text processing)

We train machines to read through sentences of a given text document, and locate and tag terms that are important for the project at hand, also called "entity recognition" "entity extraction" and "tagging".

For exmaple, given the following sentence, which is title of one of the 30 million text documents served by pubmed.gov
"inhibition of prostatic cancer growth by ginsenoside rh2"

Our trained machine read through the sentence above, recognized the terms within, and added "tags" to the sentence (also called "inline tagging"):
"inhibition U00ZqN U00Zr9 U2zS4Q T7W T7p T9m of prostatic U01y4s U18hqj T4d TeG cancer U0hmZc TxT U00jxc U0K4DF U0Yizf U0c1ux U19piI U20X5d T5Z TAk Tgh Ty1 TzN growth U00QhA U0abY8 U15hVs U1cRqi U2ao0Y T5Z T73 T7A T7e T9m by ginsenoside U02Pm2 U0RMwH Tjt TlB U02Pln Tjl rh2 U13vHj T4Y"

In the above sentence, the words with yellow background are original sentence, and the tags starting with 'U' and 'T' are added by the machine.

Each tag points to a particular meaning, also called a 'concept'. For exmaple, the tag 'U0hmZc' highlighted orange below:
"inhibition U00ZqN U00Zr9 U2zS4Q T7W T7p T9m of prostatic U01y4s U18hqj T4d TeG cancer U0hmZc TxT U00jxc U0K4DF U0Yizf U0c1ux U19piI U20X5d T5Z TAk Tgh Ty1 TzN growth U00QhA U0abY8 U15hVs U1cRqi U2ao0Y T5Z T73 T7A T7e T9m by ginsenoside U02Pm2 U0RMwH Tjt TlB U02Pln Tjl rh2 U13vHj T4Y"
The tag 'U0hmZc' points to the disease expressed by the term 'prostate cancer'. The machine understand over 80 terms that express the same concept, the 'prostate cancer'; see the following list from machine's knowledge-base:


MariaDB [bmlrnd]> select strcv46, LAT, CuiB62IrSpecCompact from U17ABMrconsoStrsCols14MbgLxvrN10Cols where binary CuiB62IrSpecCompact='U0hmZc' order by LAT;
+-----------------------------------------------------------+------+---------------------+
| strcv46                                                   | LAT  | CuiB62IrSpecCompact |
+-----------------------------------------------------------+------+---------------------+
| maligní novotvar prostaty                                 | CZE  | U0hmZc              |
| maligní nádor prostaty                                    | CZE  | U0hmZc              |
| maligní nádor prostaty nos                                | CZE  | U0hmZc              |
| maligní nádorové onemocnění prostaty                      | CZE  | U0hmZc              |
| novotvary prostaty maligní                                | CZE  | U0hmZc              |
| rakovina prostaty                                         | CZE  | U0hmZc              |
| ondartet svulst i prostata                                | DAN  | U0hmZc              |
| kanker van de prostaat                                    | DUT  | U0hmZc              |
| maligne neoplasma prostaat                                | DUT  | U0hmZc              |
| maligne neoplasma van de prostaat                         | DUT  | U0hmZc              |
| maligne neoplasma van prostaat                            | DUT  | U0hmZc              |
| maligniteit prostaat                                      | DUT  | U0hmZc              |
| prostaatkanker                                            | DUT  | U0hmZc              |
| prostaatkanker nao                                        | DUT  | U0hmZc              |
| prostaatneoplasmata maligne                               | DUT  | U0hmZc              |
| ca - cancer of prostate                                   | ENG  | U0hmZc              |
| malignant neoplasm of prostate                            | ENG  | U0hmZc              |
| malignant neoplasm of prostate gland                      | ENG  | U0hmZc              |
| malignant neoplasm of prostate gland (diagnosis)          | ENG  | U0hmZc              |
| malignant neoplasm of the prostate                        | ENG  | U0hmZc              |
| malignant neoplasm prostate                               | ENG  | U0hmZc              |
| malignant neosplasm of the prostate                       | ENG  | U0hmZc              |
| malignant prostate neoplasm                               | ENG  | U0hmZc              |
| malignant prostatic tumor                                 | ENG  | U0hmZc              |
| malignant tumor of prostate                               | ENG  | U0hmZc              |
| malignant tumor of prostate (disorder)                    | ENG  | U0hmZc              |
| malignant tumor of prostate gland                         | ENG  | U0hmZc              |
| neoplasm malig;prostate                                   | ENG  | U0hmZc              |
| prostate cancer (diagnosis)                               | ENG  | U0hmZc              |
| prostatic cancer                                          | ENG  | U0hmZc              |
| prostatic neoplasms malignant                             | ENG  | U0hmZc              |
| eturauhasen pahanlaatuinen kasvain                        | FIN  | U0hmZc              |
| cancer de la prostate                                     | FRE  | U0hmZc              |
| cancer de la prostate sai                                 | FRE  | U0hmZc              |
| cancer prostate                                           | FRE  | U0hmZc              |
| cancer prostatique                                        | FRE  | U0hmZc              |
| tumeur maligne de la prostate                             | FRE  | U0hmZc              |
| tumeurs malignes prostatiques                             | FRE  | U0hmZc              |
| boesartige neubildung der prostata                        | GER  | U0hmZc              |
| krebs der prostata                                        | GER  | U0hmZc              |
| neubildungen der prostata boesartig                       | GER  | U0hmZc              |
| prostatakarzinom nnb                                      | GER  | U0hmZc              |
| prostatakrebs                                             | GER  | U0hmZc              |
| sartan haarmonit                                          | HEB  | U0hmZc              |
| malignus prostata tumor                                   | HUN  | U0hmZc              |
| prostata malignus neoplasiája                             | HUN  | U0hmZc              |
| prostata neoplasmák, rosszindulatú                        | HUN  | U0hmZc              |
| prostata rák                                              | HUN  | U0hmZc              |
| prostatarák k.m.n.                                        | HUN  | U0hmZc              |
| prosztata rákja                                           | HUN  | U0hmZc              |
| prosztatarák                                              | HUN  | U0hmZc              |
| cancro della prostata                                     | ITA  | U0hmZc              |
| cancro della prostata nas                                 | ITA  | U0hmZc              |
| cancro prostatico                                         | ITA  | U0hmZc              |
| neoplasie maligne prostata                                | ITA  | U0hmZc              |
| tumore maligno della prostata                             | ITA  | U0hmZc              |
| tumori maligni della prostata                             | ITA  | U0hmZc              |
| 前立腺の悪性新生物                                        | JPN  | U0hmZc              |
| 前立腺癌nos                                            | JPN  | U0hmZc              |
| 悪性前立腺新生物                                          | JPN  | U0hmZc              |
| アクセイゼンリツセンシンセイブツ                                        | JPN  | U0hmZc              |
| ゼンリツセンガンnos                                             | JPN  | U0hmZc              |
| ゼンリツセンノアクセイシンセイブツ                                       | JPN  | U0hmZc              |
| 전립샘의 악성신생물                                       | KOR  | U0hmZc              |
| kreft i blærehalskjertel                                  | NOR  | U0hmZc              |
| ondartet svulst i blærehalskjertel                        | NOR  | U0hmZc              |
| prostatacancer                                            | NOR  | U0hmZc              |
| prostatakreft                                             | NOR  | U0hmZc              |
| cancro da próstata                                        | POR  | U0hmZc              |
| cancro da próstata ne                                     | POR  | U0hmZc              |
| câncer da próstata                                        | POR  | U0hmZc              |
| câncer de próstata                                        | POR  | U0hmZc              |
| neoplasia maligna da prostata                             | POR  | U0hmZc              |
| neoplasia maligna da próstata                             | POR  | U0hmZc              |
| neoplasias malignas prostáticas                           | POR  | U0hmZc              |
| cáncer de la próstata                                     | SPA  | U0hmZc              |
| cáncer de próstata                                        | SPA  | U0hmZc              |
| cáncer de próstata neom                                   | SPA  | U0hmZc              |
| neoplasia maligna de próstata                             | SPA  | U0hmZc              |
| neoplasias malignas de prostata                           | SPA  | U0hmZc              |
| neoplasias prostáticas malignas                           | SPA  | U0hmZc              |
| tumor maligno de próstata                                 | SPA  | U0hmZc              |
| tumor maligno de próstata (trastorno)                     | SPA  | U0hmZc              |
| malign tumor i prostata                                   | SWE  | U0hmZc              |
+-----------------------------------------------------------+------+---------------------+
84 rows in set (20.66 sec)

Above list of 84 terms is called a 'synonym set' or a 'synset' for the tag 'U0hmZc', where all 84 of them express the same concept of this particular disease.


How many terms can the machine detect?
Currently the machine is trained to recognize ~21 million English terms, plus select terms from 24 other languages, for a total of 28.7 million terms:

MariaDB [bmlrnd]> select LAT, count(*) from U17ABMrconsoStrsCols14MbgLxvrN10Cols group by LAT order by count(*) desc;
+------+----------+
| LAT  | count(*) |
+------+----------+
| ENG  | 20903344 |
| SPA  |  3009972 |
| FRE  |   822982 |
| POR  |   562910 |
| DUT  |   554195 |
| ITA  |   413779 |
| RUS  |   356515 |
| GER  |   329935 |
| JPN  |   274536 |
| CZE  |   229695 |
| CHI  |   221338 |
| TUR  |   149464 |
| HUN  |   146143 |
| KOR  |   102353 |
| EST  |    90990 |
| NOR  |    74190 |
| POL  |    70787 |
| FIN  |    32443 |
| SWE  |    31721 |
| SCR  |    16215 |
| GRE  |     6456 |
| LAV  |     2306 |
| BAQ  |     1571 |
| DAN  |     1558 |
| HEB  |      917 |
+------+----------+
26 rows in set (45.62 sec)

MariaDB [bmlrnd]> select count(*) from U17ABMrconsoStrsCols14MbgLxvrN10Cols;
+----------+
| count(*) |
+----------+
| 28675919 |
+----------+
1 row in set (0.00 sec)