Allow translating from javascript files #15612

cofiem · 2024-03-17T06:25:43Z

This PR enables translating strings in javascript.

It demonstrates the js translation in use by applying it to the timeago.js js util.

Edit: added overview of the two mutually-exclusive approaches.

Approach: webpack plugin the generates js message bundles

This is the current state of this PR.

I attempted an approach similar to https://www.npmjs.com/package/@zainulbr/i18n-webpack-plugin
i.e. embed the translations into separate js bundles, one for each language.

Unfortunately, the existing webpack plugins were not suitable (issues with webpack 5 and/or did not work for the warehouse setup).
However, I did manage to create a webpack plugin that does the job (in webpack.plugin.localize.js).
The plugin checks the plural forms and embeds them into the generated js bundles.

So, the development process with these changes:

Use gettext or ngettext as appropriate in js files
Run make translations: pybabel extracts messages from js files, then a python script generates messages.json files for only the KNOWN_LOCALES and only includes translations that are needed in js
Run make static_pipeline: the webpack plugin embeds the translated data from the messages.json files
At runtime, the matching js bundle for the current locale is set in base.html, and gettext.js is initialised with the translation data. It can determine plurals and do string templating.

These are the benefits of this approach:

Uses the existing translation and js tooling - make, pybabel, locale files, webpack
Enables plurals and template placeholders
Limits or does not use the hard to cache routes nor additional http requests
Limits the number of unmaintained packages required, and works with webpack 5
Straightforward to maintain and use as part of contributions

Approach: request from client to server to get translation for a given message code

A js util fetch-gettext.js that requests the translation code returns a promise, which contains the translation text from the server, using the locale set on the server-side.

view the changes for this approach.

Changes:

enables extracting strings from javascript using pybabel,
adds a route that renders the appropriate translation using the existing machinery, and
adds js util for obtaining the translated string.

Considerations:

may make many http requests for translations

Notes

Pybabel seems to require particular js function names and the config to change them does not work, similar to this pybabel issue. This is ok, I've used the names expected by pybabel.

Relates to issues:

di · 2024-03-19T00:46:58Z

Very cool! I like this approach a lot, I think this will definitely work.

One thing I'm wondering: since we probably won't have a ton of JS translations, would it make sense to load them all for a given locale in a bundle via a single request, as opposed to making N different requests?

cofiem · 2024-03-19T08:48:43Z

One thing I'm wondering: since we probably won't have a ton of JS translations, would it make sense to load them all for a given locale in a bundle via a single request, as opposed to making N different requests?

I'm not sure I understand this.

I found a way to use the server-side rendering of translations because:

I can't see that there's a way to get just the js translation strings to make a bundle of translation strings
each translation can have up to 6 variants, which are chosen based on the num parameter
some translations have placeholders in the string, and I didn't want to have to find or make a way to fill the placeholders in js

If you're concerned about the server load or frequency of requests, the browser cache seems to work well, and there's likely some additional caching to be done in the server-side view that could reduce the need to execute the localiser.pluralize or localizer.translate.

Please let me know if I've misunderstood what you're asking.

…ture/translation-js

cofiem · 2024-03-19T13:18:03Z

I managed to get the plurals extracted.

It looks like python-babel expects particular function names.

TODO:

I'll include some screenshots of the js translations in action.
I may also have time to go through and make the rest of the text that's been identified in the other issues as translatable.
fix the couple of failing tests

Question:

Is there an existing way to cache the output of a view that depends on the request.params? That would work well for caching the translation view on the server side.
Maybe origin_cache?

Provide two functions for gettext so IDe highlighting is useful.

di · 2024-03-20T12:56:38Z

If you're concerned about the server load or frequency of requests, the browser cache seems to work well, and there's likely some additional caching to be done in the server-side view that could reduce the need to execute the localiser.pluralize or localizer.translate.

Yes, I was thinking about balancing the number of requests with the size of the requests. It's probably an over-optimization though, our CDN will cache these responses by default so it's not the load on our backend that I was concerned with, just the number of requests the user's browser would have to make after the page has loaded. I think it's fine to move forward as-is, just wanted to think through it a bit.

Is there an existing way to cache the output of a view that depends on the request.params? That would work well for caching the translation view on the server side.

By default, we cache all requests, but strip all URL parameters when caching a given view (with some exceptions). We can add an exception for this view. This happens here:

https://github.com/pypi/infra/blob/9568346c6781a6d9e754e9edc3883dd34febd97b/terraform/warehouse/vcl/main.vcl#L103-L126

cofiem · 2024-03-24T10:00:23Z

The only remaining question is what @di was asking - is the number of requests made by the browser a problem?

Some example screenshots of the js translations in use.

cofiem · 2024-03-24T10:40:06Z

Test errors:

> msg += " " + " ".join(results["feedback"]["suggestions"])
E TypeError: unsupported operand type(s) for +=: 'LazyString' and 'str'

This one could be fixed by adding str(msg), but there might be a better way?

- msg += " " + " ".join(results["feedback"]["suggestions"])
+ msg = str(msg) + " " + " ".join(results["feedback"]["suggestions"])

> return request.localizer.translate(_translation_factory(message, **kwargs))
E AttributeError: 'NoneType' object has no attribute 'localizer'

I'm not sure how to fix this error. Other form tests seem to be able to set request=pretend.stub(), but the Validator directly uses from warehouse.i18n import localize as _, which errors at request.localizer.

di

Took one last look and I have one outstanding issue here before we can merge.

bin/translations

This removes a step and allows translation updates (static_pipeline) to generate the js bundles. The messages.json files are not required when translation data is bundled like this.

miketheman

Looking good - I did some basic testing and it looks like it creates a full JS bundle per language, so once a local has changed, the only JS resource loaded is the one for that locale.

I put a bunch of comments inline - let me know if they don't make sense.

I'm also a little concerned about a JS injection attack from externally-supplied values - is there something I'm overlooking on how these are sanitized?

miketheman · 2024-08-19T15:12:45Z

webpack.plugin.localize.js

+          const denied = new RegExp(
+            "[~`^$_\\[\\]{}\\\\'\"\\.#@\\f\\n\\r\\t\\v\\u00a0\\u1680\\u2000-\\u200a\\u2028\\u2029\\u202f\\u205f\\u3000\\ufeff]+");


suggestion: add a comment to denote where this regex originated, and what it is denying

addressed in 8e2e3f5

miketheman · 2024-08-19T15:15:17Z

webpack.config.js

-    compressionOptions: { level: 9, memLevel: 9 },
+    compressionOptions: {level: 9, memLevel: 9},


thought: does our linter not check and enforce these kinds of changes?

(might be?) addressed in 7d1e8cb

Makefile

warehouse/admin/static/js/warehouse.js

miketheman · 2024-08-19T15:27:52Z

tests/frontend/messages_access_test.js

+    const data = {
+      "": {"language": "fr", "plural-forms": "nplurals=2; plural=n > 1;"},
+      "My default message.": "My translated message.",
+      "My %2 message with placeholders %1.": "My translated %1 message with placeholders %2",
+      "My message with plurals": ["My translated message 1.", "My translated messages 2."],
+      "My message with plurals %1 again": ["My translated message 1 %1.", "My translated message 2 %1"],
+    };
+    const pluralForms = function (n) {
+      let nplurals, plural;
+      nplurals = 2; plural = n > 1;
+      return {total: nplurals, index: ((nplurals > 1 && plural === true) ? 1 : (plural ? plural : 0))};
+    };


question: this looks a lot like a parametrized test, is that the case here? If so, would it make more sense to use jest's .each to express the individual tests to reduce the amount of logic baked in to the test case?

addressed in b6fc1e6

miketheman · 2024-08-19T15:31:18Z

tests/frontend/timeago_test.js

+import timeAgo from "../../warehouse/static/js/warehouse/utils/timeago";
+import {delay} from "./utils";
+
+describe("time ago util", () => {


Yay, we have tests now!

miketheman · 2024-08-19T15:33:49Z

warehouse/static/js/warehouse/utils/messages-access.js

+}
+
+export function ngettextCustom(singular, plural, num, extras, data, pluralForms) {
+  // this function allows for testing


question: is this comment still accurate, if ngettext is basically a wrapper for this, with some defaults?

addressed in d32b447

miketheman · 2024-08-19T15:35:12Z

warehouse/static/js/warehouse/utils/messages-access.js

+var messagesAccessLocaleData = {"": {"language": "en", "plural-forms": "nplurals = 2; plural = (n != 1)"}};
+
+// the value for 'messagesAccessPluralFormFunction' is set by webpack.plugin.localize.js
+// default is en
+var messagesAccessPluralFormFunction = function (n) {


question: should these be var or const/let? (again, I wonder what's happening with the linter...)

addressed in d32b447

miketheman · 2024-08-19T15:41:22Z

webpack.config.js

+    return file;
+  };
+// Refs: https://github.com/shellscape/webpack-manifest-plugin/issues/229#issuecomment-737617994
+// publicPath: "",


question: why is this previously-set value commented out? If it's no longer valid, let's remove it, along with the comment above as to why it's there, or move it to where it's now being set, or extract it to a constant that can be easier commented and reused.

addressed in 0f13ec1

di · 2024-08-19T15:58:51Z

webpack.plugin.localize.js

+        if (refs.every(refLine => !refLine.includes(".js:"))) {
+          continue;
+        }
+        result[value.msgid] = value.msgstr;


I think we need to sanitize the user input at this point.

(partly?) addressed in fd0dd80

webpack.plugin.localize.js

Using ngettextCustom makes it possible to test the functions and use the function signatured required by pybabel. Fix eslint issues.

This makes it easier to see the test cases. Add comments to make it clearer what the data is used for.

Added comment to explain what the pattern is checking.

This is important as these values will be embedded into the messages-access.js file.

…ture/translation-js

Makefile

webpack.plugin.localize.js

Co-authored-by: Dustin Ingram <[email protected]>

di

LGTM

cofiem added 8 commits March 17, 2024 15:44

add translations route for js translations

05f5601

use js translations for timeago

ece9fb5

add babel config for js translations

d70e76e

add js util to get translations from backend

9003fa5

add messages extracted from js

d2273a4

use the default pybabel js function name

291189b

typo

64f4d9a

Merge branch 'main' into feature/translation-js

a10b3fb

cofiem added 4 commits March 19, 2024 21:26

extract singular and plural translations from js

ec3b0be

Merge branch 'pypi:main' into feature/translation-js

5920aeb

Merge remote-tracking branch 'origin/feature/translation-js' into fea…

11db677

…ture/translation-js

fix lint issues and remove logging

f705788

cofiem added 5 commits March 20, 2024 21:53

add translation to more js strings

4e95739

Provide two functions for gettext so IDe highlighting is useful.

add translation to more js strings

266c1e5

add translation to more python strings

8ff40e4

update and add tests for js translations

7dc6ea4

update translations

c593101

cofiem added 6 commits March 24, 2024 16:36

Merge branch 'pypi:main' into feature/translation-js

fa2721a

fix route test

c1a7811

avoid redirect

25f2c28

fix lint error

1a6b4e0

update translation url in tests

d73ee32

update translations

b44655f

cofiem marked this pull request as ready for review March 24, 2024 10:40

Merge branch 'main' into feature/translation-js

cf57600

di reviewed Aug 15, 2024

View reviewed changes

bin/translations Outdated Show resolved Hide resolved

cofiem and others added 3 commits August 19, 2024 16:52

Merge branch 'main' into feature/translation-js

d18b04a

build js translation data as part of static_pipeline webpack build

88e027e

This removes a step and allows translation updates (static_pipeline) to generate the js bundles. The messages.json files are not required when translation data is bundled like this.

Merge branch 'main' into feature/translation-js

fd7a88e

miketheman reviewed Aug 19, 2024

View reviewed changes

di reviewed Aug 19, 2024

View reviewed changes

cofiem added 15 commits August 21, 2024 21:45

address feedback: don't translate the admin UI

0f7196d

address feedback: don't translate the admin UI

010fdd2

address feedback: fix make dependency order

43d6be1

address feedback: improve docs for placeholder js variables

d32b447

Using ngettextCustom makes it possible to test the functions and use the function signatured required by pybabel. Fix eslint issues.

address feedback: add missing webpack js files to eslint paths

7d1e8cb

address feedback: fix eslint issues

34ed5e6

address feedback: extract shared value to constant

0f13ec1

address feedback: use jest each for tests

b6fc1e6

This makes it easier to see the test cases. Add comments to make it clearer what the data is used for.

address feedback: remove old todo

6b094a2

address feedback: fix eslint issues

70f2c33

address feedback: improve plural forms pattern and check earlier

8e2e3f5

Added comment to explain what the pattern is checking.

address feedback: make sure msgid and msgstr are really strings

fd0dd80

This is important as these values will be embedded into the messages-access.js file.

Merge remote-tracking branch 'origin/feature/translation-js' into fea…

7d35f82

…ture/translation-js

Merge branch 'pypi:main' into feature/translation-js

50261ef

Merge remote-tracking branch 'origin/feature/translation-js' into fea…

7cd21a5

…ture/translation-js

di reviewed Aug 21, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

di reviewed Aug 21, 2024

View reviewed changes

webpack.plugin.localize.js Outdated Show resolved Hide resolved

cofiem and others added 4 commits August 22, 2024 19:13

Revert change to make statuc_pipelines

6fa0444

Co-authored-by: Dustin Ingram <[email protected]>

encode known problematic characters

b6edde8

Co-authored-by: Dustin Ingram <[email protected]>

Merge branch 'main' into feature/translation-js

3c91638

fix lint space issue

ef820c2

di approved these changes Sep 23, 2024

View reviewed changes

di requested a review from miketheman September 23, 2024 15:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow translating from javascript files #15612

Allow translating from javascript files #15612

cofiem commented Mar 17, 2024 •

edited

Loading

di commented Mar 19, 2024

cofiem commented Mar 19, 2024 •

edited

Loading

cofiem commented Mar 19, 2024

di commented Mar 20, 2024

cofiem commented Mar 24, 2024

cofiem commented Mar 24, 2024

di left a comment

miketheman left a comment

miketheman Aug 19, 2024

cofiem Aug 21, 2024

miketheman Aug 19, 2024

cofiem Aug 21, 2024

miketheman Aug 19, 2024

cofiem Aug 21, 2024

miketheman Aug 19, 2024

miketheman Aug 19, 2024

cofiem Aug 21, 2024

miketheman Aug 19, 2024

cofiem Aug 21, 2024

miketheman Aug 19, 2024

cofiem Aug 21, 2024

di Aug 19, 2024

cofiem Aug 21, 2024

di left a comment

		const denied = new RegExp(
		"[~`^$_\\[\\]{}\\\\'\"\\.#@\\f\\n\\r\\t\\v\\u00a0\\u1680\\u2000-\\u200a\\u2028\\u2029\\u202f\\u205f\\u3000\\ufeff]+");

		compressionOptions: { level: 9, memLevel: 9 },
		compressionOptions: {level: 9, memLevel: 9},

Allow translating from javascript files #15612

Are you sure you want to change the base?

Allow translating from javascript files #15612

Conversation

cofiem commented Mar 17, 2024 • edited Loading

Approach: webpack plugin the generates js message bundles

Approach: request from client to server to get translation for a given message code

Notes

di commented Mar 19, 2024

cofiem commented Mar 19, 2024 • edited Loading

cofiem commented Mar 19, 2024

di commented Mar 20, 2024

cofiem commented Mar 24, 2024

cofiem commented Mar 24, 2024

di left a comment

Choose a reason for hiding this comment

miketheman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

di left a comment

Choose a reason for hiding this comment

cofiem commented Mar 17, 2024 •

edited

Loading

cofiem commented Mar 19, 2024 •

edited

Loading