Sakurai/pdf.js
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19f7a8b899
pdf.js/src/core/unicode.js

274 lines
12 KiB
JavaScript
Raw Normal View History

Lazify NormalizedUnicodes
2016-01-22 07:10:42 +09:00
/* Copyright 2016 Mozilla Foundation
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
[api-minor] Don't normalize the text used in the text layer. Some arabic chars like \ufe94 could be searched in a pdf, hence it must be normalized when creating the search query. So to avoid to duplicate the normalization code, everything is moved in the find controller. The previous code to normalize text was using NFKC but with a hardcoded map, hence it has been replaced by the use of normalize("NFKC") (it helps to reduce the bundle size by 30kb). In playing with this \ufe94 char, I noticed that the bidi algorithm wasn't taking into account some RTL unicode ranges, the generated font wasn't embedding the mapping this char and the unicode ranges in the OS/2 table weren't up-to-date. When normalized some chars can be replaced by several ones and it induced to have some extra chars in the text layer. To avoid any regression, when copying some text from the text layer, a copied string is normalized (NFKC) before being put in the clipboard (it works like this in either Acrobat or Chrome).
2023-03-23 18:15:14 +09:00
import { getLookupTableFactory } from "./core_utils.js";
Lazify NormalizedUnicodes
2016-01-22 07:10:42 +09:00
Convert the files in the `/src/core` folder to ES6 modules Please note that the `glyphlist.js` and `unicode.js` files are converted to CommonJS modules instead, since Babel cannot handle files that large and they are thus excluded from transpilation.
2017-04-02 23:14:30 +09:00
// Some characters, e.g. copyrightserif, are mapped to the private use area
// and might not be displayed using standard fonts. Mapping/hacking well-known
// chars to the similar equivalents in the normal characters range.
Enable the `no-var` linting rule in `src/core/unicode.js`
2021-02-27 20:42:55 +09:00
const getSpecialPUASymbols = getLookupTableFactory(function (t) {
Enable auto-formatting of the entire code-base using Prettier (issue 11444) Note that Prettier, purposely, has only limited [configuration options](https://prettier.io/docs/en/options.html). The configuration file is based on [the one in `mozilla central`](https://searchfox.org/mozilla-central/source/.prettierrc) with just a few additions (to avoid future breakage if the defaults ever changes). Prettier is being used for a couple of reasons: - To be consistent with `mozilla-central`, where Prettier is already in use across the tree. - To ensure a *consistent* coding style everywhere, which is automatically enforced during linting (since Prettier is used as an ESLint plugin). This thus ends "all" formatting disussions once and for all, removing the need for review comments on most stylistic matters. Many ESLint options are now redundant, and I've tried my best to remove all the now unnecessary options (but I may have missed some). Note also that since Prettier considers the `printWidth` option as a guide, rather than a hard rule, this patch resorts to a small hack in the ESLint config to ensure that *comments* won't become too long. *Please note:* This patch is generated automatically, by appending the `--fix` argument to the ESLint call used in the `gulp lint` task. It will thus require some additional clean-up, which will be done in a *separate* commit. (On a more personal note, I'll readily admit that some of the changes Prettier makes are *extremely* ugly. However, in the name of consistency we'll probably have to live with that.)
2019-12-25 23:59:37 +09:00
t[63721] = 0x00a9; // copyrightsans (0xF8E9) => copyright
t[63193] = 0x00a9; // copyrightserif (0xF6D9) => copyright
t[63720] = 0x00ae; // registersans (0xF8E8) => registered
t[63194] = 0x00ae; // registerserif (0xF6DA) => registered
Convert the files in the `/src/core` folder to ES6 modules Please note that the `glyphlist.js` and `unicode.js` files are converted to CommonJS modules instead, since Babel cannot handle files that large and they are thus excluded from transpilation.
2017-04-02 23:14:30 +09:00
t[63722] = 0x2122; // trademarksans (0xF8EA) => trademark
t[63195] = 0x2122; // trademarkserif (0xF6DB) => trademark
Enable auto-formatting of the entire code-base using Prettier (issue 11444) Note that Prettier, purposely, has only limited [configuration options](https://prettier.io/docs/en/options.html). The configuration file is based on [the one in `mozilla central`](https://searchfox.org/mozilla-central/source/.prettierrc) with just a few additions (to avoid future breakage if the defaults ever changes). Prettier is being used for a couple of reasons: - To be consistent with `mozilla-central`, where Prettier is already in use across the tree. - To ensure a *consistent* coding style everywhere, which is automatically enforced during linting (since Prettier is used as an ESLint plugin). This thus ends "all" formatting disussions once and for all, removing the need for review comments on most stylistic matters. Many ESLint options are now redundant, and I've tried my best to remove all the now unnecessary options (but I may have missed some). Note also that since Prettier considers the `printWidth` option as a guide, rather than a hard rule, this patch resorts to a small hack in the ESLint config to ensure that *comments* won't become too long. *Please note:* This patch is generated automatically, by appending the `--fix` argument to the ESLint call used in the `gulp lint` task. It will thus require some additional clean-up, which will be done in a *separate* commit. (On a more personal note, I'll readily admit that some of the changes Prettier makes are *extremely* ugly. However, in the name of consistency we'll probably have to live with that.)
2019-12-25 23:59:37 +09:00
t[63729] = 0x23a7; // bracelefttp (0xF8F1)
t[63730] = 0x23a8; // braceleftmid (0xF8F2)
t[63731] = 0x23a9; // braceleftbt (0xF8F3)
t[63740] = 0x23ab; // bracerighttp (0xF8FC)
t[63741] = 0x23ac; // bracerightmid (0xF8FD)
t[63742] = 0x23ad; // bracerightbt (0xF8FE)
t[63726] = 0x23a1; // bracketlefttp (0xF8EE)
t[63727] = 0x23a2; // bracketleftex (0xF8EF)
t[63728] = 0x23a3; // bracketleftbt (0xF8F0)
t[63737] = 0x23a4; // bracketrighttp (0xF8F9)
t[63738] = 0x23a5; // bracketrightex (0xF8FA)
t[63739] = 0x23a6; // bracketrightbt (0xF8FB)
t[63723] = 0x239b; // parenlefttp (0xF8EB)
t[63724] = 0x239c; // parenleftex (0xF8EC)
t[63725] = 0x239d; // parenleftbt (0xF8ED)
t[63734] = 0x239e; // parenrighttp (0xF8F6)
t[63735] = 0x239f; // parenrightex (0xF8F7)
t[63736] = 0x23a0; // parenrightbt (0xF8F8)
Convert the files in the `/src/core` folder to ES6 modules Please note that the `glyphlist.js` and `unicode.js` files are converted to CommonJS modules instead, since Babel cannot handle files that large and they are thus excluded from transpilation.
2017-04-02 23:14:30 +09:00
});
Lazify NormalizedUnicodes
2016-01-22 07:10:42 +09:00
Convert the files in the `/src/core` folder to ES6 modules Please note that the `glyphlist.js` and `unicode.js` files are converted to CommonJS modules instead, since Babel cannot handle files that large and they are thus excluded from transpilation.
2017-04-02 23:14:30 +09:00
function mapSpecialUnicodeValues(code) {
Enable auto-formatting of the entire code-base using Prettier (issue 11444) Note that Prettier, purposely, has only limited [configuration options](https://prettier.io/docs/en/options.html). The configuration file is based on [the one in `mozilla central`](https://searchfox.org/mozilla-central/source/.prettierrc) with just a few additions (to avoid future breakage if the defaults ever changes). Prettier is being used for a couple of reasons: - To be consistent with `mozilla-central`, where Prettier is already in use across the tree. - To ensure a *consistent* coding style everywhere, which is automatically enforced during linting (since Prettier is used as an ESLint plugin). This thus ends "all" formatting disussions once and for all, removing the need for review comments on most stylistic matters. Many ESLint options are now redundant, and I've tried my best to remove all the now unnecessary options (but I may have missed some). Note also that since Prettier considers the `printWidth` option as a guide, rather than a hard rule, this patch resorts to a small hack in the ESLint config to ensure that *comments* won't become too long. *Please note:* This patch is generated automatically, by appending the `--fix` argument to the ESLint call used in the `gulp lint` task. It will thus require some additional clean-up, which will be done in a *separate* commit. (On a more personal note, I'll readily admit that some of the changes Prettier makes are *extremely* ugly. However, in the name of consistency we'll probably have to live with that.)
2019-12-25 23:59:37 +09:00
if (code >= 0xfff0 && code <= 0xffff) {
// Specials unicode block.
Convert the files in the `/src/core` folder to ES6 modules Please note that the `glyphlist.js` and `unicode.js` files are converted to CommonJS modules instead, since Babel cannot handle files that large and they are thus excluded from transpilation.
2017-04-02 23:14:30 +09:00
return 0;
Enable auto-formatting of the entire code-base using Prettier (issue 11444) Note that Prettier, purposely, has only limited [configuration options](https://prettier.io/docs/en/options.html). The configuration file is based on [the one in `mozilla central`](https://searchfox.org/mozilla-central/source/.prettierrc) with just a few additions (to avoid future breakage if the defaults ever changes). Prettier is being used for a couple of reasons: - To be consistent with `mozilla-central`, where Prettier is already in use across the tree. - To ensure a *consistent* coding style everywhere, which is automatically enforced during linting (since Prettier is used as an ESLint plugin). This thus ends "all" formatting disussions once and for all, removing the need for review comments on most stylistic matters. Many ESLint options are now redundant, and I've tried my best to remove all the now unnecessary options (but I may have missed some). Note also that since Prettier considers the `printWidth` option as a guide, rather than a hard rule, this patch resorts to a small hack in the ESLint config to ensure that *comments* won't become too long. *Please note:* This patch is generated automatically, by appending the `--fix` argument to the ESLint call used in the `gulp lint` task. It will thus require some additional clean-up, which will be done in a *separate* commit. (On a more personal note, I'll readily admit that some of the changes Prettier makes are *extremely* ugly. However, in the name of consistency we'll probably have to live with that.)
2019-12-25 23:59:37 +09:00
} else if (code >= 0xf600 && code <= 0xf8ff) {
return getSpecialPUASymbols()[code] || code;
Fix the linting errors, from the Prettier auto-formatting, that ESLint `--fix` couldn't handle This patch makes the follow changes: - Remove no longer necessary inline `// eslint-disable-...` comments. - Fix `// eslint-disable-...` comments that Prettier moved down, thus causing new linting errors. - Concatenate strings which now fit on just one line. - Fix comments that are now too long. - Finally, and most importantly, adjust comments that Prettier moved down, since the new positions often is confusing or outright wrong.
2019-12-26 04:03:46 +09:00
} else if (code === /* softhyphen = */ 0x00ad) {
Enable auto-formatting of the entire code-base using Prettier (issue 11444) Note that Prettier, purposely, has only limited [configuration options](https://prettier.io/docs/en/options.html). The configuration file is based on [the one in `mozilla central`](https://searchfox.org/mozilla-central/source/.prettierrc) with just a few additions (to avoid future breakage if the defaults ever changes). Prettier is being used for a couple of reasons: - To be consistent with `mozilla-central`, where Prettier is already in use across the tree. - To ensure a *consistent* coding style everywhere, which is automatically enforced during linting (since Prettier is used as an ESLint plugin). This thus ends "all" formatting disussions once and for all, removing the need for review comments on most stylistic matters. Many ESLint options are now redundant, and I've tried my best to remove all the now unnecessary options (but I may have missed some). Note also that since Prettier considers the `printWidth` option as a guide, rather than a hard rule, this patch resorts to a small hack in the ESLint config to ensure that *comments* won't become too long. *Please note:* This patch is generated automatically, by appending the `--fix` argument to the ESLint call used in the `gulp lint` task. It will thus require some additional clean-up, which will be done in a *separate* commit. (On a more personal note, I'll readily admit that some of the changes Prettier makes are *extremely* ugly. However, in the name of consistency we'll probably have to live with that.)
2019-12-25 23:59:37 +09:00
return 0x002d; // hyphen
Convert the files in the `/src/core` folder to ES6 modules Please note that the `glyphlist.js` and `unicode.js` files are converted to CommonJS modules instead, since Babel cannot handle files that large and they are thus excluded from transpilation.
2017-04-02 23:14:30 +09:00
}
return code;
}
Lazify NormalizedUnicodes
2016-01-22 07:10:42 +09:00
Convert the files in the `/src/core` folder to ES6 modules Please note that the `glyphlist.js` and `unicode.js` files are converted to CommonJS modules instead, since Babel cannot handle files that large and they are thus excluded from transpilation.
2017-04-02 23:14:30 +09:00
function getUnicodeForGlyph(name, glyphsUnicodeMap) {
Enable the `no-var` linting rule in `src/core/unicode.js`
2021-02-27 20:42:55 +09:00
let unicode = glyphsUnicodeMap[name];
Convert the files in the `/src/core` folder to ES6 modules Please note that the `glyphlist.js` and `unicode.js` files are converted to CommonJS modules instead, since Babel cannot handle files that large and they are thus excluded from transpilation.
2017-04-02 23:14:30 +09:00
if (unicode !== undefined) {
return unicode;
}
if (!name) {
return -1;
Lazify NormalizedUnicodes
2016-01-22 07:10:42 +09:00
}
Convert the files in the `/src/core` folder to ES6 modules Please note that the `glyphlist.js` and `unicode.js` files are converted to CommonJS modules instead, since Babel cannot handle files that large and they are thus excluded from transpilation.
2017-04-02 23:14:30 +09:00
// Try to recover valid Unicode values from 'uniXXXX'/'uXXXX{XX}' glyphs.
Enable auto-formatting of the entire code-base using Prettier (issue 11444) Note that Prettier, purposely, has only limited [configuration options](https://prettier.io/docs/en/options.html). The configuration file is based on [the one in `mozilla central`](https://searchfox.org/mozilla-central/source/.prettierrc) with just a few additions (to avoid future breakage if the defaults ever changes). Prettier is being used for a couple of reasons: - To be consistent with `mozilla-central`, where Prettier is already in use across the tree. - To ensure a *consistent* coding style everywhere, which is automatically enforced during linting (since Prettier is used as an ESLint plugin). This thus ends "all" formatting disussions once and for all, removing the need for review comments on most stylistic matters. Many ESLint options are now redundant, and I've tried my best to remove all the now unnecessary options (but I may have missed some). Note also that since Prettier considers the `printWidth` option as a guide, rather than a hard rule, this patch resorts to a small hack in the ESLint config to ensure that *comments* won't become too long. *Please note:* This patch is generated automatically, by appending the `--fix` argument to the ESLint call used in the `gulp lint` task. It will thus require some additional clean-up, which will be done in a *separate* commit. (On a more personal note, I'll readily admit that some of the changes Prettier makes are *extremely* ugly. However, in the name of consistency we'll probably have to live with that.)
2019-12-25 23:59:37 +09:00
if (name[0] === "u") {
Enable the `no-var` linting rule in `src/core/unicode.js`
2021-02-27 20:42:55 +09:00
const nameLen = name.length;
let hexStr;
Lazify NormalizedUnicodes
2016-01-22 07:10:42 +09:00
Enable auto-formatting of the entire code-base using Prettier (issue 11444) Note that Prettier, purposely, has only limited [configuration options](https://prettier.io/docs/en/options.html). The configuration file is based on [the one in `mozilla central`](https://searchfox.org/mozilla-central/source/.prettierrc) with just a few additions (to avoid future breakage if the defaults ever changes). Prettier is being used for a couple of reasons: - To be consistent with `mozilla-central`, where Prettier is already in use across the tree. - To ensure a *consistent* coding style everywhere, which is automatically enforced during linting (since Prettier is used as an ESLint plugin). This thus ends "all" formatting disussions once and for all, removing the need for review comments on most stylistic matters. Many ESLint options are now redundant, and I've tried my best to remove all the now unnecessary options (but I may have missed some). Note also that since Prettier considers the `printWidth` option as a guide, rather than a hard rule, this patch resorts to a small hack in the ESLint config to ensure that *comments* won't become too long. *Please note:* This patch is generated automatically, by appending the `--fix` argument to the ESLint call used in the `gulp lint` task. It will thus require some additional clean-up, which will be done in a *separate* commit. (On a more personal note, I'll readily admit that some of the changes Prettier makes are *extremely* ugly. However, in the name of consistency we'll probably have to live with that.)
2019-12-25 23:59:37 +09:00
if (nameLen === 7 && name[1] === "n" && name[2] === "i") {
// 'uniXXXX'
Replace `String.prototype.substr()` occurrences with `String.prototype.substring()` As outlined in https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/substr, which refers to the ECMA-262 specification, using the `substr` function is advised against. Hence this PR, which replaces all remaining `substr` occurrences with `substring` instead. Please refer to https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/substr#Syntax respectively https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/substring#Syntax for the differences between the two functions. Note that in most cases in the code-base there's only one argument passed to `substr`, and those require no other changes except replacing "substr" with "substring". For the other cases, the `substr(start, length)` calls are changed to `substring(start, start + length)` instead.
2018-09-28 18:41:07 +09:00
hexStr = name.substring(3);
Enable auto-formatting of the entire code-base using Prettier (issue 11444) Note that Prettier, purposely, has only limited [configuration options](https://prettier.io/docs/en/options.html). The configuration file is based on [the one in `mozilla central`](https://searchfox.org/mozilla-central/source/.prettierrc) with just a few additions (to avoid future breakage if the defaults ever changes). Prettier is being used for a couple of reasons: - To be consistent with `mozilla-central`, where Prettier is already in use across the tree. - To ensure a *consistent* coding style everywhere, which is automatically enforced during linting (since Prettier is used as an ESLint plugin). This thus ends "all" formatting disussions once and for all, removing the need for review comments on most stylistic matters. Many ESLint options are now redundant, and I've tried my best to remove all the now unnecessary options (but I may have missed some). Note also that since Prettier considers the `printWidth` option as a guide, rather than a hard rule, this patch resorts to a small hack in the ESLint config to ensure that *comments* won't become too long. *Please note:* This patch is generated automatically, by appending the `--fix` argument to the ESLint call used in the `gulp lint` task. It will thus require some additional clean-up, which will be done in a *separate* commit. (On a more personal note, I'll readily admit that some of the changes Prettier makes are *extremely* ugly. However, in the name of consistency we'll probably have to live with that.)
2019-12-25 23:59:37 +09:00
} else if (nameLen >= 5 && nameLen <= 7) {
// 'uXXXX{XX}'
Replace `String.prototype.substr()` occurrences with `String.prototype.substring()` As outlined in https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/substr, which refers to the ECMA-262 specification, using the `substr` function is advised against. Hence this PR, which replaces all remaining `substr` occurrences with `substring` instead. Please refer to https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/substr#Syntax respectively https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/substring#Syntax for the differences between the two functions. Note that in most cases in the code-base there's only one argument passed to `substr`, and those require no other changes except replacing "substr" with "substring". For the other cases, the `substr(start, length)` calls are changed to `substring(start, start + length)` instead.
2018-09-28 18:41:07 +09:00
hexStr = name.substring(1);
Convert the files in the `/src/core` folder to ES6 modules Please note that the `glyphlist.js` and `unicode.js` files are converted to CommonJS modules instead, since Babel cannot handle files that large and they are thus excluded from transpilation.
2017-04-02 23:14:30 +09:00
} else {
Convert `uniXXXX` glyph names to proper ones when building the `charCodeToGlyphId` map for TrueType fonts (bug 1132849, issue 6893, issue 6894) This patch adds a `getUnicodeForGlyph` helper function, which is used to recover Unicode values for non-standard glyph names. Some PDF generators, e.g. Scribus PDF, use improper `uniXXXX` glyph names which breaks the glyph mapping. We can avoid this by converting them to "standard" glyph names instead. Fixes https://bugzilla.mozilla.org/show_bug.cgi?id=1132849. Fixes 6893. Fixes 6894.
2016-03-08 04:56:15 +09:00
return -1;
}
Convert the files in the `/src/core` folder to ES6 modules Please note that the `glyphlist.js` and `unicode.js` files are converted to CommonJS modules instead, since Babel cannot handle files that large and they are thus excluded from transpilation.
2017-04-02 23:14:30 +09:00
// Check for upper-case hexadecimal characters, to avoid false positives.
if (hexStr === hexStr.toUpperCase()) {
unicode = parseInt(hexStr, 16);
if (unicode >= 0) {
return unicode;
Convert `uniXXXX` glyph names to proper ones when building the `charCodeToGlyphId` map for TrueType fonts (bug 1132849, issue 6893, issue 6894) This patch adds a `getUnicodeForGlyph` helper function, which is used to recover Unicode values for non-standard glyph names. Some PDF generators, e.g. Scribus PDF, use improper `uniXXXX` glyph names which breaks the glyph mapping. We can avoid this by converting them to "standard" glyph names instead. Fixes https://bugzilla.mozilla.org/show_bug.cgi?id=1132849. Fixes 6893. Fixes 6894.
2016-03-08 04:56:15 +09:00
}
}
}
Convert the files in the `/src/core` folder to ES6 modules Please note that the `glyphlist.js` and `unicode.js` files are converted to CommonJS modules instead, since Babel cannot handle files that large and they are thus excluded from transpilation.
2017-04-02 23:14:30 +09:00
return -1;
}
Convert `uniXXXX` glyph names to proper ones when building the `charCodeToGlyphId` map for TrueType fonts (bug 1132849, issue 6893, issue 6894) This patch adds a `getUnicodeForGlyph` helper function, which is used to recover Unicode values for non-standard glyph names. Some PDF generators, e.g. Scribus PDF, use improper `uniXXXX` glyph names which breaks the glyph mapping. We can avoid this by converting them to "standard" glyph names instead. Fixes https://bugzilla.mozilla.org/show_bug.cgi?id=1132849. Fixes 6893. Fixes 6894.
2016-03-08 04:56:15 +09:00
[api-minor] Don't normalize the text used in the text layer. Some arabic chars like \ufe94 could be searched in a pdf, hence it must be normalized when creating the search query. So to avoid to duplicate the normalization code, everything is moved in the find controller. The previous code to normalize text was using NFKC but with a hardcoded map, hence it has been replaced by the use of normalize("NFKC") (it helps to reduce the bundle size by 30kb). In playing with this \ufe94 char, I noticed that the bidi algorithm wasn't taking into account some RTL unicode ranges, the generated font wasn't embedding the mapping this char and the unicode ranges in the OS/2 table weren't up-to-date. When normalized some chars can be replaced by several ones and it induced to have some extra chars in the text layer. To avoid any regression, when copying some text from the text layer, a copied string is normalized (NFKC) before being put in the clipboard (it works like this in either Acrobat or Chrome).
2023-03-23 18:15:14 +09:00
// See https://learn.microsoft.com/en-us/typography/opentype/spec/os2#ulunicoderange1-bits-031ulunicoderange2-bits-3263ulunicoderange3-bits-6495ulunicoderange4-bits-96127
Enable the `no-var` linting rule in `src/core/unicode.js`
2021-02-27 20:42:55 +09:00
const UnicodeRanges = [
[api-minor] Don't normalize the text used in the text layer. Some arabic chars like \ufe94 could be searched in a pdf, hence it must be normalized when creating the search query. So to avoid to duplicate the normalization code, everything is moved in the find controller. The previous code to normalize text was using NFKC but with a hardcoded map, hence it has been replaced by the use of normalize("NFKC") (it helps to reduce the bundle size by 30kb). In playing with this \ufe94 char, I noticed that the bidi algorithm wasn't taking into account some RTL unicode ranges, the generated font wasn't embedding the mapping this char and the unicode ranges in the OS/2 table weren't up-to-date. When normalized some chars can be replaced by several ones and it induced to have some extra chars in the text layer. To avoid any regression, when copying some text from the text layer, a copied string is normalized (NFKC) before being put in the clipboard (it works like this in either Acrobat or Chrome).
2023-03-23 18:15:14 +09:00
[0x0000, 0x007f], // 0 - Basic Latin
[0x0080, 0x00ff], // 1 - Latin-1 Supplement
[0x0100, 0x017f], // 2 - Latin Extended-A
[0x0180, 0x024f], // 3 - Latin Extended-B
[0x0250, 0x02af, 0x1d00, 0x1d7f, 0x1d80, 0x1dbf], // 4 - IPA Extensions - Phonetic Extensions - Phonetic Extensions Supplement
[0x02b0, 0x02ff, 0xa700, 0xa71f], // 5 - Spacing Modifier Letters - Modifier Tone Letters
[0x0300, 0x036f, 0x1dc0, 0x1dff], // 6 - Combining Diacritical Marks - Combining Diacritical Marks Supplement
[0x0370, 0x03ff], // 7 - Greek and Coptic
[0x2c80, 0x2cff], // 8 - Coptic
[0x0400, 0x04ff, 0x0500, 0x052f, 0x2de0, 0x2dff, 0xa640, 0xa69f], // 9 - Cyrillic - Cyrillic Supplement - Cyrillic Extended-A - Cyrillic Extended-B
[0x0530, 0x058f], // 10 - Armenian
[0x0590, 0x05ff], // 11 - Hebrew
[0xa500, 0xa63f], // 12 - Vai
[0x0600, 0x06ff, 0x0750, 0x077f], // 13 - Arabic - Arabic Supplement
[0x07c0, 0x07ff], // 14 - NKo
[0x0900, 0x097f], // 15 - Devanagari
[0x0980, 0x09ff], // 16 - Bengali
[0x0a00, 0x0a7f], // 17 - Gurmukhi
[0x0a80, 0x0aff], // 18 - Gujarati
[0x0b00, 0x0b7f], // 19 - Oriya
[0x0b80, 0x0bff], // 20 - Tamil
[0x0c00, 0x0c7f], // 21 - Telugu
[0x0c80, 0x0cff], // 22 - Kannada
[0x0d00, 0x0d7f], // 23 - Malayalam
[0x0e00, 0x0e7f], // 24 - Thai
[0x0e80, 0x0eff], // 25 - Lao
[0x10a0, 0x10ff, 0x2d00, 0x2d2f], // 26 - Georgian - Georgian Supplement
[0x1b00, 0x1b7f], // 27 - Balinese
[0x1100, 0x11ff], // 28 - Hangul Jamo
[0x1e00, 0x1eff, 0x2c60, 0x2c7f, 0xa720, 0xa7ff], // 29 - Latin Extended Additional - Latin Extended-C - Latin Extended-D
[0x1f00, 0x1fff], // 30 - Greek Extended
[0x2000, 0x206f, 0x2e00, 0x2e7f], // 31 - General Punctuation - Supplemental Punctuation
[0x2070, 0x209f], // 32 - Superscripts And Subscripts
[0x20a0, 0x20cf], // 33 - Currency Symbol
[0x20d0, 0x20ff], // 34 - Combining Diacritical Marks
[0x2100, 0x214f], // 35 - Letterlike Symbols
[0x2150, 0x218f], // 36 - Number Forms
[0x2190, 0x21ff, 0x27f0, 0x27ff, 0x2900, 0x297f, 0x2b00, 0x2bff], // 37 - Arrows - Supplemental Arrows-A - Supplemental Arrows-B - Miscellaneous Symbols and Arrows
[0x2200, 0x22ff, 0x2a00, 0x2aff, 0x27c0, 0x27ef, 0x2980, 0x29ff], // 38 - Mathematical Operators - Supplemental Mathematical Operators - Miscellaneous Mathematical Symbols-A - Miscellaneous Mathematical Symbols-B
[0x2300, 0x23ff], // 39 - Miscellaneous Technical
[0x2400, 0x243f], // 40 - Control Pictures
[0x2440, 0x245f], // 41 - Optical Character Recognition
[0x2460, 0x24ff], // 42 - Enclosed Alphanumerics
[0x2500, 0x257f], // 43 - Box Drawing
[0x2580, 0x259f], // 44 - Block Elements
[0x25a0, 0x25ff], // 45 - Geometric Shapes
[0x2600, 0x26ff], // 46 - Miscellaneous Symbols
[0x2700, 0x27bf], // 47 - Dingbats
[0x3000, 0x303f], // 48 - CJK Symbols And Punctuation
[0x3040, 0x309f], // 49 - Hiragana
[0x30a0, 0x30ff, 0x31f0, 0x31ff], // 50 - Katakana - Katakana Phonetic Extensions
[0x3100, 0x312f, 0x31a0, 0x31bf], // 51 - Bopomofo - Bopomofo Extended
[0x3130, 0x318f], // 52 - Hangul Compatibility Jamo
[0xa840, 0xa87f], // 53 - Phags-pa
[0x3200, 0x32ff], // 54 - Enclosed CJK Letters And Months
[0x3300, 0x33ff], // 55 - CJK Compatibility
[0xac00, 0xd7af], // 56 - Hangul Syllables
[0xd800, 0xdfff], // 57 - Non-Plane 0 *
[0x10900, 0x1091f], // 58 - Phoenicia
[
0x4e00, 0x9fff, 0x2e80, 0x2eff, 0x2f00, 0x2fdf, 0x2ff0, 0x2fff, 0x3400,
0x4dbf, 0x20000, 0x2a6df, 0x3190, 0x319f,
], // 59 - CJK Unified Ideographs - CJK Radicals Supplement - Kangxi Radicals - Ideographic Description Characters - CJK Unified Ideographs Extension A - CJK Unified Ideographs Extension B - Kanbun
[0xe000, 0xf8ff], // 60 - Private Use Area (plane 0)
[0x31c0, 0x31ef, 0xf900, 0xfaff, 0x2f800, 0x2fa1f], // 61 - CJK Strokes - CJK Compatibility Ideographs - CJK Compatibility Ideographs Supplement
[0xfb00, 0xfb4f], // 62 - Alphabetic Presentation Forms
[0xfb50, 0xfdff], // 63 - Arabic Presentation Forms-A
[0xfe20, 0xfe2f], // 64 - Combining Half Marks
[0xfe10, 0xfe1f], // 65 - Vertical Forms
[0xfe50, 0xfe6f], // 66 - Small Form Variants
[0xfe70, 0xfeff], // 67 - Arabic Presentation Forms-B
[0xff00, 0xffef], // 68 - Halfwidth And Fullwidth Forms
[0xfff0, 0xffff], // 69 - Specials
[0x0f00, 0x0fff], // 70 - Tibetan
[0x0700, 0x074f], // 71 - Syriac
[0x0780, 0x07bf], // 72 - Thaana
[0x0d80, 0x0dff], // 73 - Sinhala
[0x1000, 0x109f], // 74 - Myanmar
[0x1200, 0x137f, 0x1380, 0x139f, 0x2d80, 0x2ddf], // 75 - Ethiopic - Ethiopic Supplement - Ethiopic Extended
[0x13a0, 0x13ff], // 76 - Cherokee
[0x1400, 0x167f], // 77 - Unified Canadian Aboriginal Syllabics
[0x1680, 0x169f], // 78 - Ogham
[0x16a0, 0x16ff], // 79 - Runic
[0x1780, 0x17ff], // 80 - Khmer
[0x1800, 0x18af], // 81 - Mongolian
[0x2800, 0x28ff], // 82 - Braille Patterns
[0xa000, 0xa48f], // 83 - Yi Syllables
[0x1700, 0x171f, 0x1720, 0x173f, 0x1740, 0x175f, 0x1760, 0x177f], // 84 - Tagalog - Hanunoo - Buhid - Tagbanwa
[0x10300, 0x1032f], // 85 - Old Italic
[0x10330, 0x1034f], // 86 - Gothic
[0x10400, 0x1044f], // 87 - Deseret
[0x1d000, 0x1d0ff, 0x1d100, 0x1d1ff, 0x1d200, 0x1d24f], // 88 - Byzantine Musical Symbols - Musical Symbols - Ancient Greek Musical Notation
[0x1d400, 0x1d7ff], // 89 - Mathematical Alphanumeric Symbols
[0xff000, 0xffffd], // 90 - Private Use (plane 15)
[0xfe00, 0xfe0f, 0xe0100, 0xe01ef], // 91 - Variation Selectors - Variation Selectors Supplement
[0xe0000, 0xe007f], // 92 - Tags
[0x1900, 0x194f], // 93 - Limbu
[0x1950, 0x197f], // 94 - Tai Le
[0x1980, 0x19df], // 95 - New Tai Lue
[0x1a00, 0x1a1f], // 96 - Buginese
[0x2c00, 0x2c5f], // 97 - Glagolitic
[0x2d30, 0x2d7f], // 98 - Tifinagh
[0x4dc0, 0x4dff], // 99 - Yijing Hexagram Symbols
[0xa800, 0xa82f], // 100 - Syloti Nagri
[0x10000, 0x1007f, 0x10080, 0x100ff, 0x10100, 0x1013f], // 101 - Linear B Syllabary - Linear B Ideograms - Aegean Numbers
[0x10140, 0x1018f], // 102 - Ancient Greek Numbers
[0x10380, 0x1039f], // 103 - Ugaritic
[0x103a0, 0x103df], // 104 - Old Persian
[0x10450, 0x1047f], // 105 - Shavian
[0x10480, 0x104af], // 106 - Osmanya
[0x10800, 0x1083f], // 107 - Cypriot Syllabary
[0x10a00, 0x10a5f], // 108 - Kharoshthi
[0x1d300, 0x1d35f], // 109 - Tai Xuan Jing Symbols
[0x12000, 0x123ff, 0x12400, 0x1247f], // 110 - Cuneiform - Cuneiform Numbers and Punctuation
[0x1d360, 0x1d37f], // 111 - Counting Rod Numerals
[0x1b80, 0x1bbf], // 112 - Sundanese
[0x1c00, 0x1c4f], // 113 - Lepcha
[0x1c50, 0x1c7f], // 114 - Ol Chiki
[0xa880, 0xa8df], // 115 - Saurashtra
[0xa900, 0xa92f], // 116 - Kayah Li
[0xa930, 0xa95f], // 117 - Rejang
[0xaa00, 0xaa5f], // 118 - Cham
[0x10190, 0x101cf], // 119 - Ancient Symbols
[0x101d0, 0x101ff], // 120 - Phaistos Disc
[0x102a0, 0x102df, 0x10280, 0x1029f, 0x10920, 0x1093f], // 121 - Carian - Lycian - Lydian
[0x1f030, 0x1f09f, 0x1f000, 0x1f02f], // 122 - Domino Tiles - Mahjong Tiles
Convert the files in the `/src/core` folder to ES6 modules Please note that the `glyphlist.js` and `unicode.js` files are converted to CommonJS modules instead, since Babel cannot handle files that large and they are thus excluded from transpilation.
2017-04-02 23:14:30 +09:00
];
Lazify NormalizedUnicodes
2016-01-22 07:10:42 +09:00
[api-minor] Don't normalize the text used in the text layer. Some arabic chars like \ufe94 could be searched in a pdf, hence it must be normalized when creating the search query. So to avoid to duplicate the normalization code, everything is moved in the find controller. The previous code to normalize text was using NFKC but with a hardcoded map, hence it has been replaced by the use of normalize("NFKC") (it helps to reduce the bundle size by 30kb). In playing with this \ufe94 char, I noticed that the bidi algorithm wasn't taking into account some RTL unicode ranges, the generated font wasn't embedding the mapping this char and the unicode ranges in the OS/2 table weren't up-to-date. When normalized some chars can be replaced by several ones and it induced to have some extra chars in the text layer. To avoid any regression, when copying some text from the text layer, a copied string is normalized (NFKC) before being put in the clipboard (it works like this in either Acrobat or Chrome).
2023-03-23 18:15:14 +09:00
function getUnicodeRangeFor(value, lastPosition = -1) {
// TODO: create a map range => position, sort the ranges and cache it.
// Then we can make a binary search for finding a range for a given unicode.
if (lastPosition !== -1) {
const range = UnicodeRanges[lastPosition];
for (let i = 0, ii = range.length; i < ii; i += 2) {
if (value >= range[i] && value <= range[i + 1]) {
return lastPosition;
}
}
}
Enable the `no-var` linting rule in `src/core/unicode.js`
2021-02-27 20:42:55 +09:00
for (let i = 0, ii = UnicodeRanges.length; i < ii; i++) {
const range = UnicodeRanges[i];
[api-minor] Don't normalize the text used in the text layer. Some arabic chars like \ufe94 could be searched in a pdf, hence it must be normalized when creating the search query. So to avoid to duplicate the normalization code, everything is moved in the find controller. The previous code to normalize text was using NFKC but with a hardcoded map, hence it has been replaced by the use of normalize("NFKC") (it helps to reduce the bundle size by 30kb). In playing with this \ufe94 char, I noticed that the bidi algorithm wasn't taking into account some RTL unicode ranges, the generated font wasn't embedding the mapping this char and the unicode ranges in the OS/2 table weren't up-to-date. When normalized some chars can be replaced by several ones and it induced to have some extra chars in the text layer. To avoid any regression, when copying some text from the text layer, a copied string is normalized (NFKC) before being put in the clipboard (it works like this in either Acrobat or Chrome).
2023-03-23 18:15:14 +09:00
for (let j = 0, jj = range.length; j < jj; j += 2) {
if (value >= range[j] && value <= range[j + 1]) {
return i;
}
Lazify NormalizedUnicodes
2016-01-22 07:10:42 +09:00
}
}
Convert the files in the `/src/core` folder to ES6 modules Please note that the `glyphlist.js` and `unicode.js` files are converted to CommonJS modules instead, since Babel cannot handle files that large and they are thus excluded from transpilation.
2017-04-02 23:14:30 +09:00
return -1;
}
Lazify NormalizedUnicodes
2016-01-22 07:10:42 +09:00
Remove the invisible format marks from the text chunks - it aims to fix issue #9186.
2022-01-24 07:04:18 +09:00
const SpecialCharRegExp = new RegExp("^(\\s)|(\\p{Mn})|(\\p{Cf})$", "u");
Add a (global) cache to the `getCharUnicodeCategory` function Given that the regular expression has already become more complex (after the initial patch adding it), it seems to me that it probably cannot hurt to add a global cache to reduce unnecessary re-parsing. Obviously the `Glyph`-instances are being cached *per* font, however in most documents multiple fonts are being used and in practice there's very often a fair amount of overlap between the /ToUnicode-data in different fonts[1]. Consider for example loading and rendering the entire `tracemonkey.pdf` document (from the test-suite), which isn't a particularily large document. In that case the `getCharUnicodeCategory` function is being called a total of `601` times, however there's only `106` *unique* unicode-chars being checked. *Please note:* In practice I suppose that this won't have a *huge* effect on overall performance, however given the relative simplicity of this patch I figured that it'd not hurt to submit it for review. --- [1] Consider e.g. how there's usually different fonts used for regular, bold, respectively italic text.
2022-01-25 00:16:54 +09:00
const CategoryCache = new Map();
Handle all the whitespaces the same way when creating text chunks
2022-01-08 05:20:53 +09:00
function getCharUnicodeCategory(char) {
Add a (global) cache to the `getCharUnicodeCategory` function Given that the regular expression has already become more complex (after the initial patch adding it), it seems to me that it probably cannot hurt to add a global cache to reduce unnecessary re-parsing. Obviously the `Glyph`-instances are being cached *per* font, however in most documents multiple fonts are being used and in practice there's very often a fair amount of overlap between the /ToUnicode-data in different fonts[1]. Consider for example loading and rendering the entire `tracemonkey.pdf` document (from the test-suite), which isn't a particularily large document. In that case the `getCharUnicodeCategory` function is being called a total of `601` times, however there's only `106` *unique* unicode-chars being checked. *Please note:* In practice I suppose that this won't have a *huge* effect on overall performance, however given the relative simplicity of this patch I figured that it'd not hurt to submit it for review. --- [1] Consider e.g. how there's usually different fonts used for regular, bold, respectively italic text.
2022-01-25 00:16:54 +09:00
const cachedCategory = CategoryCache.get(char);
if (cachedCategory) {
return cachedCategory;
}
Handle all the whitespaces the same way when creating text chunks
2022-01-08 05:20:53 +09:00
const groups = char.match(SpecialCharRegExp);
Add a (global) cache to the `getCharUnicodeCategory` function Given that the regular expression has already become more complex (after the initial patch adding it), it seems to me that it probably cannot hurt to add a global cache to reduce unnecessary re-parsing. Obviously the `Glyph`-instances are being cached *per* font, however in most documents multiple fonts are being used and in practice there's very often a fair amount of overlap between the /ToUnicode-data in different fonts[1]. Consider for example loading and rendering the entire `tracemonkey.pdf` document (from the test-suite), which isn't a particularily large document. In that case the `getCharUnicodeCategory` function is being called a total of `601` times, however there's only `106` *unique* unicode-chars being checked. *Please note:* In practice I suppose that this won't have a *huge* effect on overall performance, however given the relative simplicity of this patch I figured that it'd not hurt to submit it for review. --- [1] Consider e.g. how there's usually different fonts used for regular, bold, respectively italic text.
2022-01-25 00:16:54 +09:00
const category = {
Introduce more optional chaining in the `src/core/` folder After PR 12563 we're now free to use optional chaining in the worker-thread as well. (This patch also fixes one previously "missed" case in the `web/` folder.) For the MOZCENTRAL build-target this patch reduces the total bundle-size by `1.6` kilobytes.
2023-05-15 19:38:28 +09:00
isWhitespace: !!groups?.[1],
isZeroWidthDiacritic: !!groups?.[2],
isInvisibleFormatMark: !!groups?.[3],
Handle all the whitespaces the same way when creating text chunks
2022-01-08 05:20:53 +09:00
};
Add a (global) cache to the `getCharUnicodeCategory` function Given that the regular expression has already become more complex (after the initial patch adding it), it seems to me that it probably cannot hurt to add a global cache to reduce unnecessary re-parsing. Obviously the `Glyph`-instances are being cached *per* font, however in most documents multiple fonts are being used and in practice there's very often a fair amount of overlap between the /ToUnicode-data in different fonts[1]. Consider for example loading and rendering the entire `tracemonkey.pdf` document (from the test-suite), which isn't a particularily large document. In that case the `getCharUnicodeCategory` function is being called a total of `601` times, however there's only `106` *unique* unicode-chars being checked. *Please note:* In practice I suppose that this won't have a *huge* effect on overall performance, however given the relative simplicity of this patch I figured that it'd not hurt to submit it for review. --- [1] Consider e.g. how there's usually different fonts used for regular, bold, respectively italic text.
2022-01-25 00:16:54 +09:00
CategoryCache.set(char, category);
return category;
}
function clearUnicodeCaches() {
CategoryCache.clear();
Handle all the whitespaces the same way when creating text chunks
2022-01-08 05:20:53 +09:00
}
Change `src/core/{glyphlist, unicode}.js` to use standard `import`/`export` statements While the *built* `pdf.worker.js` file still works correctly with these changes, despite these two files being excluded by Babel[1], the development viewer does not because of issues with SystemJS[2] and/or its Babel-plugin (both of which are old). Furthermore, note also that excluding these two files from Babel-processing isn't *generally* necessary since e.g. the `gulp mozcentral` command works anyway. The explanation is rather that it's actually the source-map generation which fails for these huge sequences when building the `pdf.worker.js` file. However, not using standard `import`/`export` statements in all files means we also need to use SystemJS when e.e. running the unit-tests. This is very unfortunate, since SystemJS (or its old Babel-version) doesn't support modern ECMAScript features such as e.g. optional chaining and nullish coalescing. Unfortunately it also seems that https://bugzilla.mozilla.org/show_bug.cgi?id=1247687, which tracks the implementation of worker-modules in Firefox, has stalled since there hasn't been any updates for six months now. To hopefully address all of the above, this patch is the first in a series that attempts to further reduce our reliance on SystemJS. --- [1] The only difference being how the dependencies are handled, in the Webpack-bundled file. [2] Parsing takes way too long and consumes too much memory, thus rendering the development viewer essentially unusable.
2020-10-25 20:27:07 +09:00
export {
Add a (global) cache to the `getCharUnicodeCategory` function Given that the regular expression has already become more complex (after the initial patch adding it), it seems to me that it probably cannot hurt to add a global cache to reduce unnecessary re-parsing. Obviously the `Glyph`-instances are being cached *per* font, however in most documents multiple fonts are being used and in practice there's very often a fair amount of overlap between the /ToUnicode-data in different fonts[1]. Consider for example loading and rendering the entire `tracemonkey.pdf` document (from the test-suite), which isn't a particularily large document. In that case the `getCharUnicodeCategory` function is being called a total of `601` times, however there's only `106` *unique* unicode-chars being checked. *Please note:* In practice I suppose that this won't have a *huge* effect on overall performance, however given the relative simplicity of this patch I figured that it'd not hurt to submit it for review. --- [1] Consider e.g. how there's usually different fonts used for regular, bold, respectively italic text.
2022-01-25 00:16:54 +09:00
clearUnicodeCaches,
Handle all the whitespaces the same way when creating text chunks
2022-01-08 05:20:53 +09:00
getCharUnicodeCategory,
Change `src/core/{glyphlist, unicode}.js` to use standard `import`/`export` statements While the *built* `pdf.worker.js` file still works correctly with these changes, despite these two files being excluded by Babel[1], the development viewer does not because of issues with SystemJS[2] and/or its Babel-plugin (both of which are old). Furthermore, note also that excluding these two files from Babel-processing isn't *generally* necessary since e.g. the `gulp mozcentral` command works anyway. The explanation is rather that it's actually the source-map generation which fails for these huge sequences when building the `pdf.worker.js` file. However, not using standard `import`/`export` statements in all files means we also need to use SystemJS when e.e. running the unit-tests. This is very unfortunate, since SystemJS (or its old Babel-version) doesn't support modern ECMAScript features such as e.g. optional chaining and nullish coalescing. Unfortunately it also seems that https://bugzilla.mozilla.org/show_bug.cgi?id=1247687, which tracks the implementation of worker-modules in Firefox, has stalled since there hasn't been any updates for six months now. To hopefully address all of the above, this patch is the first in a series that attempts to further reduce our reliance on SystemJS. --- [1] The only difference being how the dependencies are handled, in the Webpack-bundled file. [2] Parsing takes way too long and consumes too much memory, thus rendering the development viewer essentially unusable.
2020-10-25 20:27:07 +09:00
getUnicodeForGlyph,
Use ESLint to ensure that `export`s are sorted alphabetically There's built-in ESLint rule, see `sort-imports`, to ensure that all `import`-statements are sorted alphabetically, since that often helps with readability. Unfortunately there's no corresponding rule to sort `export`-statements alphabetically, however there's an ESLint plugin which does this; please see https://www.npmjs.com/package/eslint-plugin-sort-exports The only downside here is that it's not automatically fixable, but the re-ordering is a one-time "cost" and the plugin will help maintain a *consistent* ordering of `export`-statements in the future. *Note:* To reduce the possibility of introducing any errors here, the re-ordering was done by simply selecting the relevant lines and then using the built-in sort-functionality of my editor.
2021-01-09 23:37:44 +09:00
getUnicodeRangeFor,
mapSpecialUnicodeValues,
Change `src/core/{glyphlist, unicode}.js` to use standard `import`/`export` statements While the *built* `pdf.worker.js` file still works correctly with these changes, despite these two files being excluded by Babel[1], the development viewer does not because of issues with SystemJS[2] and/or its Babel-plugin (both of which are old). Furthermore, note also that excluding these two files from Babel-processing isn't *generally* necessary since e.g. the `gulp mozcentral` command works anyway. The explanation is rather that it's actually the source-map generation which fails for these huge sequences when building the `pdf.worker.js` file. However, not using standard `import`/`export` statements in all files means we also need to use SystemJS when e.e. running the unit-tests. This is very unfortunate, since SystemJS (or its old Babel-version) doesn't support modern ECMAScript features such as e.g. optional chaining and nullish coalescing. Unfortunately it also seems that https://bugzilla.mozilla.org/show_bug.cgi?id=1247687, which tracks the implementation of worker-modules in Firefox, has stalled since there hasn't been any updates for six months now. To hopefully address all of the above, this patch is the first in a series that attempts to further reduce our reliance on SystemJS. --- [1] The only difference being how the dependencies are handled, in the Webpack-bundled file. [2] Parsing takes way too long and consumes too much memory, thus rendering the development viewer essentially unusable.
2020-10-25 20:27:07 +09:00
};
Reference in New Issue Copy Permalink