Attendees:
| Name | Abbreviation | Organization |
|---|---|---|
| Daniel Minor | DLM | Mozilla |
| Ashley Claymore | ACE | Bloomberg |
| Jonathan Kuperman | JKP | Bloomberg |
| Jason Williams | JWS | Bloomberg |
| Waldemar Horwat | WH | Invited Expert |
| Richard Gibson | RGN | Agoric |
| Philip Chimento | PFC | Igalia |
| Jesse Alama | JMN | Igalia |
| Chengzhong Wu | CZW | Bloomberg |
| Michael Saboff | MLS | Apple |
| Duncan MacGregor | DMM | ServiceNow |
| Keith Miller | KM | Apple |
| Chip Morningstar | CM | Consensys |
| Tom Kopp | TKP | Zalari |
| Istvan Sebestyen | IS | Ecma International |
| Sergey Rubanov | SRV | Invited Expert |
| Samina Husain | SHN | Ecma International |
| Aki Rose Braun | AKI | Ecma International |
| Chris de Almeida | CDA | IBM |
| Shane F Carr | SFC | |
| Ron Buckton | RBN | Microsoft |
| Mikhail Barash | MBH | Uni. Bergen |
| Romulo Cintra | RCA | Igalia |
| Nicolò Ribaudo | NRO | Igalia |
Presenter: Philip Chimento (PFC)
PFC: My name is Philip Chimento, I work for Igalia. I’ve been doing some work on ShadowRealm in partnership with Salesforce. This is a very fast update. to let you know where we are and what is going to happen, in the near future.
PFC: We last talked about this proposal in February at the plenary in San Diego. We described the criterion for including which web APIs to include in ShadowRealm, which I will summarize with the word confidentiality. And we have a PR open to the HTML spec, which is awaiting review from the HTML reviewers. Since then, we have collected feedback from these. We have heard feedback from the reviewers, and Mozilla about difficulties with WPT coverage. It would be more useful to have coverage of ShadowRealms created inside different types of realms, not just from inside the main JavaScript realm. For example, ShadowRealm created from inside a worker or whatever.
PFC: We’ve heard this feedback, and more feedback is welcome. We would like as much as possible to have the HTML PR in a merge-able state as soon as we can. So that’s what we will work on, in the short term here, investigating ways to simplify the criterion of what web apps to include in the ShadowRealm to get rid of the confusion of what confidentiality exactly means, and we will continue to add WPT coverage, including specifically addressing the concerns that we heard from Mozilla. And that’s it. It was just one slide.
PFC: I think DE already let me know he wanted to say something in the queue. But I don’t think he’s here yet.
DE: [inserted later] I recommend applying the criterion, “ShadowRealms should contain the intersection of what’s in all conceivable environments”, which implies that they are missing everything to do with I/O excluding import(), timers, DOM, etc. I agree with the set of things that are spec’d as Exposed=* that Igalia has put together--they seem to be following this principle already. We will need to document this design principle in the W3C TAG design principles document.
RPR: That’s right. Dan is not here at the moment.
MAG: I just—the thing I want to say is, thank you very much for all the work on trying to improve the WPT stuff. It is better. It’s in a much better state than it was. And we look forward to more work on this. But yeah. That’s about all I have to say.
PFC: Okay. Thanks.
RPR: There is nothing more in the queue. Any more comments or questions about this proposal? No? Okay. Then I think we can move on.
PFC: All right. Thank you, everybody.
- Since the last time we discussed ShadowRealm, we’ve been collecting feedback from HTML reviewers and Mozilla, but would still like to hear more from other parties.
- In the short-term, we’ll focus on addressing the confusion around the “confidentiality” criterion, addressing Mozilla’s concerns about WPT coverage, and merging the ShadowRealm HTML integration into the HTML spec.
- DE recommends adopting criterion, “ShadowRealms should contain the intersection of what’s in all conceivable environments”
Presenter: Jon Kuperman (JKP) and Agata Belkius (BEL)
JKP: I am Jon Kuperman. I am presenting today with ATO and the TG sourcemaps. I requested a longer session, opposed to my normal 0 to 10-minute update is that we’re essentially doing a lot of work in our small group and wanted a chance to dive deeper and get feedback on plenary on the things we have been working on. This update, we are hoping to cover is about our constituencies and process. Go through a list of the specification fixes that we have implemented. We have gone to TG1 for a rubber stamp. These are fixes we have merged into the draft of the specification, but have not come to you all asking for a rubber stamp approval. So not in the official approval. I have a specification question to get approval on. Talking about the three new feature proposals and their current status and switch to A—so we build a validator for validating sourcemaps generated by and a test suite to other things that are applying those things and handling edge cases and I hope not it take the whole time to get some general feedback at the end.
JKP: So the really high level updates, what I would have done normally yesterday, so this test suite we have been working on, alongside with people at Bloomberg and Igalia, and Mozilla, has merged into the Firefox devtools, the sourcemap repo and soon merged into chrome devtools. We needed a license, which we did yesterday. We added to the testing products and if you are in the area, we have a hackathon two weeks, June 24 and 25 in Munich, in Google office. A lot of us will get together and work on upping the test counts, implementation of the new proposals, all sorts of fun stuff. That is open to everybody, if you like to join, sign up on the slides or talk to me.
JKP: So yeah the goals of the update, under a year ago, I had come to plenary and asked for consensus of forming the test group. We were coming to the end of this year for a rubber stamp on the specification updates and one thing that I’ve been trying to do is find the right balance, not it take a lot of plenary time with sourcemap specific information every 2 months, but also nervous about doing too much work in a silo without communicating it well enough and I don’t want there be surprising or any wrong paths we don’t figure out until too late. I want to give an in-depth update and get feedback.
JKP: So a few just the way we have been thinking about sourcemaps and we have all this in the documentation in the TC39 sourcemaps. We have been thinking of the constituency in three terms, generators, debuggers and error monitoring tools. The generators would be tools that write sourcemaps to disks. These are like bunned letters, and transpilers, anything generating a .map file. Debuggers would be browsers but also a lot of these standalone debuggers. SHN said, replayIO are joining TC39 and has a cool standalone debugging application. Those don’t generate the sourcemap but read it and provide debugging experience. And the third are the error monitoring tools. Sentry is joining TC39. So they do things like wrap the error object in your application, and they keep track of real user errors and they also use sourcemap to help you with track traces and figuring out where the errors came from.
JKP: So I did a presentation a few plenaries ago on the process. We have a multistage process and they start off and move through via consensus vote. This gets us to our internal stage 4 and comes to plenary asking for approval and messaging it into the official annual spec. The process is pretty similar to the TG1 process. We stage 1 is a problem defined in an explainer document with or without a solution. Also, if there’s anything that people have questions on, with this stuff, feel free, this is what we currently have published. Stage 2, a set of details on the problem, and experimentation is encouraged in Stage 2. We have been adding the proposal into the sourcemap generation or the debugging.
JKP: We have this list here https://github.com/jkup/source-map-users, this is linked in the slides of all the sourcemap users, this is incomplete. But I am trying to find a list of every project that generates or reads sourcemaps. These are programming languages, and bundlers, compilers, all tools and debugging tools and error monitoring tools as well. We have a Stage 3. When the solution has been completely written and at least one implementation in a generator. So this is our first step, after we have a solution idea, we work with some of the generating tools to look into the reality of actually generating that sourcemap information in the project and again the test suite which is also in the TC39 repo. And then in a similar vein, like, when we get to Stage 3, we like to have some confidence that it would be hard to improve the proposal, without actually testing a real world scenario and sorting out things we need. Stage 4 the completion stage. That is in at least two of each of the constituents, reading it, and then 2 of these error monitoring or track traces and the test suite complete. And that’s what we will take to TG1 each other.
JKP: So I wanted to go through the fixes we have already sort of merged in. A lot of minor, but I think they have been a good exercise in taking a specification that hasn’t had a lot of attention and going—there’s a phrase that got used yesterday, documenting the reality or something like that. We have been trying to just take a survey of how everybody is currently working with sourcemaps and update the spec to be an accurate representation of what people are already doing.
JKP: We have this version field which must be the integer 3. We have the mandatory thing in the spec of like—this is for JSON hijacking. There’s a sourcemap over HTTP it should look for and remove the closing brackets. We have modified the specs, which is what the tools do and check for. We have both sourcemaps like a singular sourcemap and also the idea of index maps, which can be like a representation of multiple sourcemaps and found this URL field that was entirely being unused through the constituents. We removed it. Precedents. Multiple ways to tell the debugger where it is. One is a sourceMappingURL comment in the code itself and the other is sourcemap HTTP header. Folks with sentry, we got a test suite using all the modern bundlers to write the comments and add the HTTP headers and test on generators. The HTTP header takes precedence. So we put that into the specification as well. We have been working with folks in the WebAssembly world, making sure sourcemaps have a good debug tool for JavaScript and WebAssembly and CSS as well. We have done things noticed in the sourcemap, only applicable to JavaScript and CSS. And so we have redefined a lot of these things. What is a column in the WebAssembly world? Defining that as byte offsets as opposed to what a column is in the JavaScript or CSS world.
JKP: We have this kind of interesting thing. Where a feature came organically during the time when nobody was working on the sourcemap specification. Google added this ignore list, a list of files that ought not to be used in debugging for various reasons, maybe the library files are—basically they would be skipped. They added this experimental x_google_ignoreList via the angular project and chrome was reading them and Mozilla readed them in with Firefox. So we moved to a ignoreList and deprecating the x_google_ignoreList. It still works in browsers, but we marked that as deprecated. In general, we have had discussions about the future of these x_extensions and decided they’re not the way we like to move forward. We want people to get in the task group and we have marked these x_extensions as no longer supported.
JKP: Similarly, worked on trying to make our spec link better with other existing specifications. A lot of self-describe things, around Wasm, through the specifications and deep linking to the Wasm specification wherever possible. What is a Wasm module custom section or the names binary format, sourcemap links directly to the W3 spec for Wasm, so you have fine grained and well structured specification text over what these things are and how they work.
JKP: And similarly, we had a lot of terminology around fetching sourcemaps with no description of what that meant or how that works. We rewrote the specification so whenever we talk about fetching, that’s in terms of the fetch API. Those were we—we had this concept of correctness and trying to update the specification to match the world. That’s what we came up with there so far.
MF: You mentioned that the x_ extensions are deprecated. Can you explain what you mean by deprecating? What would that mean for an implementation to deprecate them?
JKP: For all the things we have done so far—sorry, I just asked about this yesterday. What is the opposite of normative changes when—these are editorial changes in the sense we are not requiring any changes on the generator side or on the debugger side with these things. They are still keeping the x_google_ignoreList. We used to have text around it as something like if you want to experiment with new features try using X prefix. We had the spec text that we didn’t really like. We went through the process without. We removed the spec text suggesting you use these X prefixes to experiment
MF: Okay. For the existing ones they are documented and just recommend –
JKP: The only existing one is the x_google_ignoreList. Those are still documented in the spec. They are marked dep deprecated but not removing anything. We removed this editorial suggesting how you experiment with new features.
MF: okay.
JKP: Thanks. So this is an open question that we have which is—I apologise. I am never sure what the TG1 context is. With the tools generated files, like minifier or removing types, they add a comment at the end of the type that sourcemapping URL and points to a file name. Without the HTTP headers, they locate the sourcemap file.
JKP: So the specification is extremely vague about this. It just says that a sourcemapping URL comment might be there. So we have been trying to flesh this out, and say exactly what that means, where it is, how to retrieve it. Our first step that I think NRO took was, trying to make a regular expression to parse it. Which is really hard. Maybe impossible. I think it might be impossible. Yeah. To get something you can verifiably prove is a JavaScript comment. But it’s fast. But it’s inaccurate. And then the other option would be to parse the generated file. It’s unacceptably slow for tools like IDEs to do quick lookups. We have a draft PR which I have linked in a slide, publishing for an either or approach. The way to look up a sourcemap is the sourcemapping URL comment.
JKP: And basically, giving two approaches. If you need to do it quickly, you can use a regular expression. And then if you need to be—like to be extremely accurate, you can parse the entire generated file and then walk that, which we have specified. I think this was something that came up in the task group committee, people were unsure on this type of thing hypothesized. Is it okay to have the either/or text. It has to be okay in some sense. Nobody is going to switch to one approach or the other. IDEs will not start to parse the entire generated file. And yeah. This was like something that we are kind of curious about. Trying to work through. Whether this is an acceptable solution to have two different means of getting this comment. I am not sure if people go on the queue and answer that later. Or… or not. But I have linked the issue in a later slide.
JKP: Then the three new proposals. Two of them have been progressing pretty far. And one of them is in the early stages. I wanted to talk about these. And get some feedback, or communicate—oh yeah.
LCA: Let me repeat that question. You mentioned this either or approach. I assume this is for parsers. That does include wording that the comments that tools or that generators generate must be parsable?
JKP: That’s a good question. NRO might have a better answer. We were considering this must be the last line after a new line. Something or the generators, but then immediately rounding into other proposals that have come through. At stages that also want something added to the last line. For example, people want this debug ID. I am not sure how far we have come on that side. You are correct, this spec text is only this—only for the parsers. For the debugging tools.
NRO: Yeah. The comment can also not be on the last line, there are many cases in which the last comment is close to the last line, but followed by other comments, which makes it easy for the two results to give different results. Yes, I am not sure about what is the current state of the request. It’s a good idea to require the tools and comments in a way that is—that regards to which approach you take, you get the same result.
JKP: yeah. I guess the short answer is, no. We don’t have any current proposed specification of text for the generators. We should talk about it after and ask to come up with something.
LCA: Thanks.
JWS: Yeah. I heard you mention you sort of spec in reality, and I understand there’s some generators or some—or reading the sourcemap, the version 3 was like sometimes a string or a number or it wasn’t even being looked at. The spec is just three. I was wondering what the story is about. Will you talk about doing the for in future or it’s just from now on it’s just 3 permanently?
JKP: From now on, 3 permanently. With the first question, it brings up a good point. We are trying to be extremely careful, like, this feels tricky. Where what we don't ever want to do is, for example, tell the parsers that they need to do something different. For example, in chrome, if they check version and there’s no version field, they salvage the sourcemap. When we talk about something required, we talk about it only in terms of the generator's side. Like, in order for a generator to be valid, they must have a version and it must be 3. But we don’t have any—any text of that sort for the consumer side. Saying, if the version isn’t there, you must throw. We don’t have any text like that. The parsers are free in all ways to try to salvage or assemble sourcemaps invalid because that is the reality of the situation. So whenever we make any of these speck changes, they are from the perspective of the generator, what they should be doing and less about the consumer, because they are free to do their best to salvage it.
WH: On the topic of the question currently on the screen, what are the consequences of some sneaky user sneaking in a sourcemap URL which this thing parses incorrectly?
NRO: Yeah. So well, the consequences in general sourcemap URL is that when you are looking at your code, in your browsers devtools, if you have sourcemaps enabled and they are usually enabled by default, you don’t see the source code running and the track traces. Potentially, you could I guess have some malicious code that is hidden unless you look very closely at it. And with this two approaches, I guess having the two possibility things to get the comment might make it harder for tools that want to make sure there is no sourcemap attached to a file, to prevent this code hiding potential problem because maybe they have looking for the comment following one of the process, and debugger, the developer using might follow the other approach. So, for example, you could hide the comment in a template literal. And a tool looking for the comment with a JavaScript parser would not find it. While the tool looking for the comment with a—like run a Regex on each like will extract it. So maybe the sourcemap wouldn’t have applied to the code when you try debugging it while the sourcemap tool did not catch it.
WH: So can one get browsers to fetch things from attacker-controlled URLs and can that cause problems?
NRO: So browsers only fetch sourcemaps when devtools are open. So that’s not a risk on the web. In general, yes, if the devtools are open and the browsers see the sourcemap, they will fetch, like do the HTTP request.
JKP: I think this is a really good call out. This isn’t an area we talked about. At least for me, the browsers themselves have code, security code around this, but I actually have no idea. So yeah. I do think that’s a good question. I can make an issue for that and talk to some of the consumers if there are any protections like linking to something malicious in the sourcemap URL.
WH: Okay. Thank you.
JKP: We have these 3 new proposals. Scope is a large one and range mop is smaller and debug ID is more of early stage thought than a full-fledged proposal.
JKP: So just a quick again—I am never sure what the context is. A JSON file with version, which is 3. So they have an array of sources which is an array of your original files, these could be TypeScript or JavaScript or C++ using WebAssembly. Just when you use a tool that bundles or compiles the code, the source arrays are the original files that went into that tool that generated the sourcemap. If you run three JavaScript files through a JavaScript build and it outputs an outputting, it will keep track of the 3 original files. Mr. That is the source's content. An array of the same file, but the inline content. This is an optional field but a lot of tools include this. When you do the lookup you can go to sources content. If you imagine sources had three files. Then sources content would also have three entries in the array, which would be the inline and escape content of those three files.
JKP: And then the last bit is the mappings and the mappings provide basically whenever you’re stopped at the break point in the generated code, the mappings provide an ability for these tools to look up exactly what place in the sourcemap that generated code came from. So the mappings are the bit complicated. They are VLQ data. So there are groups separated by semicolons and inside the group are the segments, tokens and point to, and comma separated. Variable link, coded and made up of 1 or 4 or 5 fields. And so basically, each one of these AAAA or all of these liken coded things contain this information.
JKP: So the first bit is in the generated file, which column are we at? When you hit a DebuggerStatement, we are on the 50th column or something like that. If you go 50 columns over in the generated file, and then the optional other ones would be to index into the sources array. Let’s say you are in bundle.JS at the 50th column. The next one, 0 with index and the sources are the first file, which would be this [p] index.JS file here. And then now in the source file. It contains a line and a column. So basically from any point in the generated file, which source file it points to and in the source fill which line and which column and so this allows a mapping from you hit a debugger and a big minified bundle and it finds which source file it came from and exactly which line and statement the debug came from in the source file. This is how they are able to map so you land on the debugger in the source file. The optional ability to add the names array. We will talk about this later. This optional array under the sourcemaps. But not well specified what you can use it for. It’s there and browsers use it for different things. If you have a generated file with multiple lines, those get represented by a sentence colon. It has a line and a column. They generated only a column. It supports line by prepending semicolons. One of the mappings is when you get stopped somewhere, you look at the Mappings list. Which line and which column in that source file. This is how these debuggers are able to do that.
JKP: So here's a visual example. You can see you can hover over any bit in either of the original code or generate code and the arrows point. A debugger statement in the generated code, line 4. And it knows it maps back to the 0 index.JS file at line 2 offset 2. Basically, if you decode the mappings, which point am I at, where does it point to and which index file? Cool. So scopes. This is a big proposal. So this is an example of a sourcemap working. You are in some devtools and you have a debugger in bundle.js and had a map. It comes from this point and is great. So you can see where the debugger was then in the source code. All the browser devtools support debug and one is the ability to display scope. Maybe there’s global or module scope and the variables you have access to, all this really useful stuff when debugging. None that is encoded in the sourcemaps. It’s this thing that customers want, users want the ability to access all this information. Yet it is not present in the sourcemaps. It’s on browsers with no specification to come up with how exactly they will implement this. And this is the way it’s been for many years.
JKP: Basically, all the browsers have taken different approaches. Chrome has big expressions. Am I in a function? Parent functions? What scope? They assemble all this. It’s quick and lead to bugs or, for example, the generated variable names could slip through instead of the course variable names. You could have these inconsistencies and Firefox loads Babel and parses the generated file. They can be more sure what the scope information is. But that’s quite a bit slower. So this is like an area where basically all of the debuggers need scope information and variable information and they don’t currently have it. So they have to go out of their way to generate it in an unspecified manner.
JKP: It’s more than just the scopes themselves. And the names. Because a lot of tools these days will do quite a few optimizations and steps. For example, terser can do constant folding or function inlining. You could end up with a file with a break point that not only doesn’t have the scope’s information but you are not aware it wasn’t a function or another function. You lose all of this information. So this is essentially impossible for the browsers right now to reconstruct. They don’t have the information about what was there.
JKP: And so basically, we came up with some goals. We merged two original proposals. The goals here are to be able to reconstruct and step through inline function. Any optimization that compilers do, we should be able to actually recreate all of the functions used to exist. We also want to do variable mapping. So a lot of tools will minifier and mangle function names and not look up the original names. So we know we are in a function called foo and we have two variables named one and two. And we have a high level of confidence that this is what was—how far it was authored in the original code. And along with that, we want to build and reconstruct scopes. Again, like I showed in the devtools, to be very sure that hey, at this point in your original code, however it was authored, here is the scope information that you had here. Here is what you access to in both of local and the global scope. And so we have a proposal to do this by adding 2 fields to the sourcemap we call originalScopes and generatedRanges. And so essentially the way this works is that here’s a sourcemap. It has the name and version of mappings and then the 2 new fields originalScopes and generatedRanges. It works, the generating tools, the bunned letters go through and add for each scope that they come across, add a scope entry to the originalScopess array. This would be like as they are parsing and bundling the file, they would keep track of each scope and its original position, end position. Like a function or a global scope, and optionally a name and a list of all the variables that are inside of it, this is a tree with nested scope. Basically, this is information that the bundlers already have access to. It’s a big part of their work flow. As they are minifying or compiling or bundling code together. This is something new they would be appending on to the sourcemap as they go.
JKP: And so yes. This—you end up with the array of nested scopes and this is hey, we had a function here. Two variables. Here are the names. And it was a function scope. At the same time, that’s like the one side of T we also have this section of these generatedRanges. These are start and position ranges. And again, just like we do with the mappings, in sourcemaps today, we basically from any debug point, we would see, is there a generatedRanges that I found the position of. If so, you know, do I have a scope that is replied to me essentially? And that links to one of the originalScopes. Basically, this would be quite a bit more information embedded in the sourcemap. But it allows a fast and accurate ability to reconstruct exactly the state you are in with the original code.
JKP: Very happy to talk more about that later. But it is like a proposal that we have been working on. I think it’s also linked to, if not, GitHub in the proposal’s fold. This is what we have been working on for the scopes proposal. This is the big one.
JKP: The next one is range mappings. We have run into this quite a bit, where I guess the TLDR on it, when you have a multiple step build process, TypeScript that generated JavaScript and terser which—when going through the multiple steps, you have to like take the sourcemap generated at each and apply them to each other combining them. And the way this apply operation works, a token has to be available in order to persist through. What you end up with is in the multistep processes, you get these extremely low fidelity sourcemaps in the end. If you ran through 3 or 4 compilation steps, removing whitespaces or comments, you end up losing a ton of information and so the effect of this would be when you set a break point in a browser debugger sometimes the best it can do it take you to that file or the beginning of that line because it’s lost a lot of the fidelity for where it was.
JKP: And so the range mappings proposal is essentially to just be able to mark them as a specific point or as a range. And these ranges would allow when the apply operation happens, to not lose fidelity. We could keep track of the—adding an additional marker to allow projects that combine sourcemaps to obtain a high level of fidelity so that it’s much easier to get more accurate debug information when you step through. I have a link here as well to this proposal.
JKP: The third one which is much earlier stage and I think this one has some trickiness to it. It requires more than just TC39 approval, but people in the error monitoring fools, they wrap the error. Object. And when your application crashes or has a bug, they send to the staff service. Along with a sourcemap. They run into a lot of these different problems where sometimes it can be extremely hard to look up which sourcemap an error applied to. Because of sourcemap URL is applied to the generated code itself, it’s hard to map—we have an error. Which file was that from? It’s hard with outdated sourcemaps. Some tools have content hash in the file name and sometimes not. Some feedback that does the stack trace analysis, it’s difficult for them to guess from which—from the stack trace which sourcemap or which version of that sourcemap to be played.
JKP: So they have come up with this proposal for the idea of adding a debug ID to the sourcemap, which we are happy to do. That seems easy for bundlers to do. The difficult part is getting the debug ID out when an error occurs. In order for this proposal to progress, we need to work with WHATWG and come up with an acceptable way of how to get it out. This is at early stages, but something we have been working on. That is it for me.
BEL: All right. We can go to the next slide. I will be talking the approach for testing sourcemaps because there’s quite of about how you normally test new premises in this community, I think. Let’s go to the next slide
BEL: Yes. So how sourcemaps actually—what is the life cycle of sourcemaps as I called it? It consists of two main steps: the first one is generating the sourcemaps. And the second one is consuming. It seems simple. But there are many ways to generate sourcemaps and many tools that do it.
BEL: And on the consumer side we have a lot of different tooling. Devtools, and also error monitoring tools that actually consume the sourcemaps.
BEL: Yeah. So generating a very simplistic diagram, you can see we have a source code. Generator takes the source code and I couldn’t tell it to put generated code and then also output a sourcemap. How do we actually test generators? This is one example. Two different generators in the same source code and outputting the same result. You can get that. But then, because of the implementation details, actually, the source code, the sourcemap that results from those two different generators is quite different. So in this case, SWC has different mappings than Babel. Because Babel is adding the names of the functions just by default, where SWC is stripping them because they are still available in the generated code. It’s not that easy to test generators based on input/output.
BEL: So here, you can see those mappings illustrated and visualized. So in Babel, you have The Bar function name. And then opening and closing brackets. But SWC, that’s just like altogether in one position. So basically, that’s why we have different mappings here. So the question; how do you actually test generators if they can generate completely different outputs? The solution for us was to validate the mappings they produce instead of having like a set of dash, yeah. I couldn’t tell you that we will check again.
BEL: The first thing that we check in the validator is to check the format. We want to—like, the sourcemap is basically including correct fields in correct format. Then the second thing we are actually checking the source files. Are they referencing the right source files? Do they exist? Are they accessible? And the third thing as long as we can, because we can’t do exact mappings, but check mappings in the sourcemap. For example, we check if the mapping is actually pointing to something that is existing in the original file? So that’s the generator. You can check it out in the repo there. It still didn’t have a title. There’s a name, so that’s why to do name. But basically, how would you use it to parse a sourcemap, parse the generated file in the original folder and validate, is it a valid sourcemap according to the criteria. That’s the first part of the journey of testing sourcemaps.
BEL: Now let’s move to the second part. So that’s consuming sourcemaps. How are they in general consumed? We have browsers. But we also have engines. So for example, Node.js can use sourcemaps because you can have TypeScript and then you need some kind of mapping to JS in stacktrace—that’s possible. Then in browsers of course we are minified bundles. The first part is you have your browser or JS engine. You have generated code. You have your sourcemap. Then debugger is using that. It’s getting the source code from the server. It could also get the source code from the sourcemap. And then on the other side we also have error monitoring. That’s usually using sourcemaps for checking the stack trace. Function names. And mapping that to be more useful for debugging as well.
BEL: So how do we test the consumers actually of the sourcemaps? Let’s go to the next slide. Yeah. We have multiple points at which—multiple steps we did this testing journey. The first one is to have a very extensive checklist of what we actually want to test. And yeah. The second thing is we want to have test data. In this case, it is different than the generators. The consumers actually can check based on specific examples and test cases. Because we basically take those and run them through different consumers. So we test the test data. The next to have a test harness. We built test harness for chrome, fair market value and WebKit. And then we also started to implement the test cases in those harnesses. Of course, for test harnesses, it’s not directly in chrome. It’s not directly in Firefox. So if chrome, it’s in the devtools content part of the stack. In Firefox, it’s in the sourcemap library of Firefox. So yeah. It’s not deep in the browser, but rather on the higher layer of the browser stack.
BEL: Okay. Let's go to the next slide. The checklist is extensive. I took a screenshot here. That’s just the beginning. It goes on for quite some time. Because it’s basically adding every case that we want to test from the spec. And that’s—that is also available in our sourcemap tests repo.
BEL: Yeah. And what does the test specification actually look like? In JSON format. We basically have an array of tests. Every test has a name. And description. So we know what you are actually testing there. Then there’s a base file. A sourcemap file. And also we have the flag of sourcemap valid or not. It's useful because basically you don’t want to test against some common cases, when the sourcemap is not valid. It should actually just be an error, but we can talk about that later in the slides, why sometimes it doesn’t happen.
BEL: And we also have test actions. So test actions are testing more detailed cases and, for example, checking mappings. Here you can see the actionType is checkMapping. And it basically takes in generatedLine and generatedColumn. originalSource. originalLine and originalColumn. And mappedName. This will basically check if that specific point is matching the point that it should match by using the sourcemap. It can also check the name, that's another thing.
BEL: Right now we have three types of actions: map—checking mappings. Also checking transitive mappings. That is perhaps when you have multiple sourcemaps. And you can just check in one is mapping to the next appointment and the next one map to consecutive point. If all that in the end will result in the—result we are expecting, the JSON. And the last thing is checking ignore list. And there is much more added in the future.
BEL: Then one thing we actually discovered and I think JKP was talking about it before, is that the consumers actually are very relaxed about how to define a valid sourcemap. Which is completely okay. Because we want the web to keep working as it was working before. Right? We don't break the web. The problem was the test cases were very strict and followed the spec. In practice, many invalid sourcemaps were not rejected. The solution for us is to either have some kind of validity of strictness in the test or at least make the spec more precise so it allows those cases that are in reality there in the wild already. Yeah. The example we were talking about as well as that most browsers don’t actually can be for field. But check it loosely or—yeah. Or just don’t. So yeah. They check the string 3 is accepted and Google devtools, I think, like, it just doesn’t check anything. It could not exist and it would be fine. So yeah. That’s one of the things we discovered.
BEL: Another thing that was revealed, more specific issues revealed through testing sourceMappings is, for example, the VLQ value it used in the sourcemaps mappings field, there’s this part in the spec which says, it should be a 32-bit limit for the value. It’s not precise what happens when you exceed that limit. Does that mean the sourcemap is not valid. Should it be capped to the 32-bits? Or what about the bit sign? Should it be included in the limit or not? There’s an issue here where we’re discussing this. Yeah. This is one of the things where the spec should be more precise. So it’s kind of—yeah. The implementation could also follow something pretty precise in this case. I think an example here was Firefox was not actually including the—was not including the sign bit in this limit, where it—we think it should, but yeah. That’s—that’s one example here.
BEL: It’s index maps. So in the spec, index maps versus basically like a few sentences. How it’s defined. And then an example of how it looks like in the JSON. But that seems not to be precise enough also to actually know how to use index maps. Index maps are basically like they have sections of different piles that are sourcemaps themselves. So it’s basically like a sourcemap that consists of many sourcemaps. But we don’t know how—how do you define the sections that must start—must be sorted by starting position and the representation section may not overlap. How do we actually define that in the JSON? It’s not really precise. Because we don’t know how they should be ordered exactly. And what does it mean—not overlap exactly. It’s similar in the part of the spec which says that the actual index maps are sharing the version, the file field, with the regular sourcemaps. And they get a sections field. But we don’t know if, for example, names field could be there. What happens if you actually have the mappings in the sourcemap? And you have the sections? Should the mappings be completely ignored. These things, it would be great to define them a little bit better or at least say which things are optional and which things like which fields should be there as required fields.
BEL: And the last one is all the other small issues that we found. Yeah. So we would like there to be specific types for field, the source root must be a figure. This could—also, the fields could be marked as optional if that really reflects the reality of how they are used.
BEL: Which means, basically, empty mapping. It shouldn’t be, but it is right now allowed by the consumers so add to the spec. There’s one specific one, which actually in chrome devtools, it came out, like, the—if you add null to the sources array, it would convert to a null string. Which is—I don’t think we want. So that’s also something that would be great to the semantics on how the sources field should look like. That’s it for the thing we found during testing.
BEL: And for the future proposals. There is highway big separate JSON file with more cases and possibly we will have new test actions for every proposal, and for example for the scopes. We are having a check scope right now the test cases are one big JSON file. For the future proposals, we can have additional field.al files with more cases and possibility will have new test actions for every proposal. Like, for these scopes, we have a check scope field.
BEL: And you can check the sourcemap spec test and the last partAnd yeah. You can check the source-map-spec text in this repo here. And the last part is what are the next steps for further testing. For consumer side, ahead the harnesses into the consumers codebases. We landed this already. So that’s in Firefox sourcemap library. We are working on landing chrome devtools harness into chrome devtools. Then we want to also make sure that the repo with all the sources is actually integrated as a git module for all the consumers. Also add more tests implementation for consumers. Get more test cases, more implementations in there. And for the generator side, better validation coverage for generators. And the last thing is to have some kind of end-to-end test between the generators and consumers to encompass the entire sourcemap life cycle and that’s basically it. Yeah. Thank you.
DE: Sorry for jumping the queue. Because the group has done a lot of work in terms of building this sourcemaps specification. And in the future meeting, maybe the next one we can come back and ask for approval. We really want to know if there’s anything you would like to know.
DE : So the scope proposal is to write detailed information about sourcemaps that will feed. Although this approval much the committee is not excited about doing what John said before, is like a rubber stamp. We want to be diligent on what we approve of now is the time because the specification is basically done. Maybe there’s a couple things to fix up. But it’s basically done. It’s there on GitHub. Next meeting, we may back some and ask for consensus on recommending it to ECMA for standardization in December. The scopes proposal is quite detailed. I am not sure if we should bring it up on the screen and review what it’s doing a little bit.
DE: You know, JKP explained the purpose. Is anybody curious to understand more details about this? Or have advice for what should be brought at a subsequent meeting for full approval? Okay. Because in some way, this is analogous with ECMA402.
DE: where the TC39 will have multiple Darth and we go through eCMA proposal in detail, and people maybe too much detail, people may end up believing that they are like better internationalization experts. Any way, I was kind of hoping that we would have at least some kind of back and forth about sourcemaps and in their technical detail.
USA: And there is a few topics on the queue, and I would request however to meet that there is four minutes left in the time box.
CM: This is prompted by DE’s question and I wanted to put my two cents in. I like the way that you all have stepped up to deal with this particular can of worms and I’m impressed by what you have done. I don’t think your presentation has too much detail, nor is it lacking in detail. I think it is fine, and I very much like the fact that TG4 is working out a style of working that is relatively autonomous compared to, say, EMCA 402, which I think could probably benefit from being a bit more autonomous, and so in general, I am in favor of the approach that you are taking.
RGN: I agree with CM and I have been following along with the proposal, somewhat on the sidelines, and am very happy with the thoroughness and the sophistication with which it has been approached. This presentation further cements that opinion and I have no other comments because the work appears to be going so great.
TKP: First, I want to second that the details work is quite nice, I am mainly curious about this version field that will never change, and mostly, everyone ignores. So why do we have it? I might have missed it in the comments.
JKP: No I did not explain that and it is a historical relic and in Google when this release a new iteration they would up the data field and that the point no changes have happened since then and then we made the decision that we don’t want to keep doing this version field thing. So as far as on the consumer side of them, I think it would be well okay for us to change the specification text that you don’t need to have that there but I suppose it is a bit tricky to be the safest possible that no one throws accidentally but before our time that the way they were doing and they were on a revision, 1, 2 or 3 on the spec maps, that is just a historical reason.
KM: Um I guess I don’t know if the version number could be used in a similar way to the Wasm version number and if there was a breaking change and then browse are can support this for a while This is less of an issue for sourcemaps because this is not like shipping on the web where there is not the same deprecation impossibility that Wasm has.
JKP: One thing I have not done a great job with is trying to write down some of our—I had a note to myself but our group so far would help to not have to do that, and we are really trying to not do that but I think it is not having the field there and like that we could especially if we are in a situation where we wanted to do that thing we have that option but we don’t want to do anything.
DE: I think webassembly is doing that and they have not incremented yet and I don’t know if we have that option with sourcemaps because so many consumers ignore the web field and because at the beginning of the error there is a different version. So maybe, we will see. Moving on.
JWS: So if I saw that you have depth indicated some google specific text, and it was similar if the reality is that consumers are not looking at the version, could that also just be depth indicated also. And I think one thing that I am really trying to work on and because I am so sort of newish though I have been working on last year and I still don’t feel confident that I have a great list of every sourcemap in existence and driven by a nervousness to remove something that might cause certain projects to throw if it is not there and my personal preference is to keep it even if we mark it as deprecated and so we have the files array that has ever been marked mandatory and no consumers that we found, and I think we do have some room, but I will definitely look for it but I will be hesitant to move.
JWS: So if you have the concept of mandatory optional text and I don’t know if that an optional spec but it could be optional.
DE: What we have been doing already for things that are not used and not generated is just removing it from the specification. So I think that if the reality is that something is not used, we already remove it and we don’t have to deprecate and deprivation is things that is still used sometimes and generated and I think with program correctness TC39 we don’t have this and so if we have a tooling space there is more opportunity in particular we want to make tighter requirement for sourcemap generator than sourcemap consumers.
JKP: Like with the in the next map and we could find nobody using it and we found comfortable using it but with the coding, it is flexible and it is checking for the version for the consumers and they will move on from that.
DE: Sorry when I say in this case I was speaking not about the verse. Not responding to but rereading the question.
NRO: I want to clarify using browser from index and for the version we suspect that maybe nobody actually checks it but right now, def tool generate the—it is more comfortable with just removing it.
PST: Um, I just want to know if there is a way I could add a consumer to the test, because XS support sourcemap types seem complete and I don’t know if sourcemaps can test—is there a document that is describing—what should I do.
BEL: So that is linked in the presentation, the sourcemap tests contain resources that you can use to implement the request and it has the big JSON files with all of different test cases and we don’t write harness for every case of course, but you can definitely use it for your project, and for the engine and yeah. Feel free to do that.
JKP: Just FYI right now the harnesses are built on and we took the source code from WebKit and took the minimum subset API’s and if you have a consumer that does not have those differently wee have to work on those but we would be happy to and to your point how to add consumer, I will make an issue to add that because we don’t have that documentation.
NRO: So if XS implements sourcemaps we will love for you or somebody else to generate multiples and mostly because we are implementing feature and so we are going to have implementers in the room that will reach out to you.
RPR: Just to echo RGN's part and this is as high quality exercise, and what is going on is that it looks like everything is on track. Also, from what I have seen, in terms of the set of generators and the set of consumers who have already been engaged or involved, that is one of the success stores is that set. And I am still left, for example, I think each has Dev Tool and we link into that and on the tooling side and generation we have TypeScript and Babel. And are there any major consumers or generators that you think at the moment have not had like radio silence or have not reached out to?
JKP: I have a personal wish list and I know everyone has been friendly and my bandwidth is difficult and my nervousness I would love to engage enough with the exacerbate and compiler community and the things that we are doing generate source notes as well. Similarly, the more WebAssembly people we have the better and I know it is an area that is into the necessarily speciality for me but we have representatives from coat land WebAssembly team assisting but I ain’t to get the WebAssembly story right and I know there is a large company that do that the stack decoders` out there and sent email to this large companies and the SAS company that do the cool air monitoring stuff I would love that and I see NIC’s comments and Apple and we don’t have anybody attending our meeting from Apple but I have spoked to people and I know our proposals are being seen, and I believe there InOrder Successor major issues right now. But at some point with the test harness we will want to put up a PR to the WebKit tools. So that is maybe another area I want to make sure we are not doing anything wrong and that is really late down the road.
NRO: I will try to submit the harness too and it is great to have somebody from WebKit that could join our meetings.
RPR: Do we know who the contact for what WebKit tools is or?
MLS: I am well aware that TG4 went people from WebKit to participate and so like they are not there yet.
USA: Okay, well that was part of queue and now we have been over our done box but we have a few time and we will take a minute to do a data summary or?
So we presented on the TG4 sourcemaps process constituents and editorial specification and active proposal for new features and we have downloaded and discussed both sourcemap validation tooling and automated assessing and not seeking advancement on anything at this time but hoping to spread awareness and engagement, and received really good actionable feedback. Thank you.
Presenter: Aapo Alasuutari (AAI)
AAI: Okay, I am Aapo Alasuutari and I was here to repeat talk that I did at the Web Engines Hackfest about maybe research engine javaScript engine that we are building with a couple of people, called Nova. So this is exploring a data-oriented engine design. And first about me, and Nova. So, I work at Valmet automation here in Finland and I hope everyone coming from abroad here is enjoying the place, and finding the University, and technical University atmosphere fun, and tech cat University student myself and I graduated and I love these places.
AAI: So at Valmet we are building a browser-based automation control system user interface and a lot of words and automation system, and it is not the system that is not based but the engineering tools. And UI performance is not like NaN second and you don’t care if it is 200 or 400 Nanosecond but if it goes into multiple hundreds of milli seconds, you will start to really suffer quickly and you can kind of maybe see where I am coming from with the data oriented side here. Um, the nova engine started in 2020 as kind of a joke—how hard could be to build a JavaScript engine? We did not have a particular goal but let’s find out. But then last year from a guy who worked in the industry and I learned about components and data oriented design and immediately I thought, wait what about a javaScript engine built with an entity component, and wouldn’t that be fun. And maybe that would actually be a revolutionary, maybe. And so okay, what on Earth is data oriented design? How many of you heard of the word? Yeah, okay, excellent. First of all data-oriented design and you must know your data, and you must know what you are working with and if you don’t know your data, you don’t know your problem. And if you don’t know your problem, you cannot solve your problem. So knowing your data and how it is common used cases is the most important software development, and if you know your data and your common use cases then you can use your data structure to support those cases and I could give dumb examples but maybe we don’t have time for that.
AAI: Your program when it runs, whatever the program does, it does not touch one thing once. It does not go to that string and read one byte from it and once and then again. It does multiple things must multiple things and common things over and over again. And it does loops and iterates and has algorithms that does this over again and as programmers we need to think of these multitude and if we don’t think about the multiples and the loops, we are doing the wrong thing. And an important point is that a computer when it loads data, if you have asked for a 64-bit integer, it will load a cache line. So when you design your data structure you should aim for the data structures support this cache line loading behavior. And you want to use the most amount of data from that cache line. If you load a cache line and use a single byte, you have wasted time.
AAI: So though it is painful you should be ignoring the singular case. If it is a one-off thing and if it happens once or rarely the performance does not matter unless you are building a performance measurement and that you are compact and doing defined property or something. So in the javaScript engine what do we know of our data? I think we have the foremost experts here so we can make some good guesses. What do we know about objects? I have one answer here but does anyone have any good answers about what happens on objects normally? Properties? Yeah. What do you know normally do with the properties? You read them and write them.
AAI: What you don’t really do that much anymore is delete them. Hash map objects used to be really common but not so much anymore. And we have proper hash maps now so we don’t need hash map object and the most common hash map object is which property will exceed unless it is looking for a prototype method, and rarely do you do hash map orbit and reopt would be the common anti-phase to this but they support maps also. And when you do an object’s property lookup, you need to access the keys and you need to search through the keys. Of course, engines do various optimization to avoid this is much as possible, but at the theoretical level, you need some way, some abstracted way to check that you have served through the key and the particular key that you match. Then when you find that match, you get the value. At most, one value. If you find no maps, then you don’t access that value and go to the prototype chain.
AAI: What about arrays, what do we commonly do with arrays? Withhold them. That is basically what I am doing all day. Withholding them. Arrays we normally access about index, right? And with by index lookup and from the spec perspective it is exactly the same as object property lookup, but from engine perspective there is not the same. You go to the elements and you know by index which place you will access and there is no key check necessary, you only need to check that it is within the length of the elements. And then you get that value. If there is a hole in there, now you go to the prototype check.
AAI: Array buffer, byte length, data view get Uint8. And any prototype method but also we have these kinds of things that we think are methods on the—sorry properties on the objects like array buffer and these never match a key and yeah somebody might put a key there theoretically we need to support the case, but it is never going to happen.
AAI: So, here is an example of an array of two properties in V8 and specifically created from a literal map from JSON but any way this is what a single cache line looks like and v8 already knows that putting stuff next to each other is, good business. So there is the prototype pointer or actually the shape pointer, abstract how the keys are laid out and elements pointer properties pointer and the two properties that were define on the object. They are in line in the same object. So you don’t need to go through the properties. How is it to get P1 object. You should not think about that. If you are thinking of the one case, you are wasting time. We can think about how well we are using the memory that we loaded. And to get to the P0, we need to check the prototype or the shape or the keys. And then we get the P0, and so we are using 16 bytes out of this cache line many and if we are mapping cache through this and getting P1 and the next property there is on the same cache line. It is not bad but it is not great.
AAI: Now here is an alternative, I have split up 8 objects into various cache lines. One cache line contains the prototype pointer for each object and one cache line will contain element pointer or each one contain property pointer for each and then I have should be actually two more cache lines where the properties are laid out. Now, if we think about mapping over these, we can actually think of the speed. It would be about maybe 400 nanoseconds and we can think of the cache plan prototype and properties, and we are caching everyone objects and every single byte is used and the two properties cache lines, yellow/red cache lines there we access every other byte essentially. Every other 8 byte section. So in total, we use every ¾ bytes, better and we can go lower, but then you know it is tradeoffs.
AAI: So data line improve cache line usage but there is a cost. What is the cost? An object cannot be a pointer. If I split the object data into different cache lines, how do I have a single pointer that points to every single cache line? That is not just really a thing. Okay I guess you could do like object offsets or pointer offsets, but that sounds kind of painful. But an object can be an index. If instead of blindly splitting to cache lines, you split to different vectors and every vector on the same index has data for the same object. Now the object can be the index into these vectors. Um, but that then means that because we are using vectors to store the heap data, now the object data itself must all the same size. We can’t change the size of our vectors kind of depending on what sort of stuff is in there. And it is build time only.
AAI: This then means if we are thinking of like is this object an array buffer or what your object your simplest object would need to be as the biggest object can ever be, which is pretty big and pretty wasteful. But what if we just have different kinds of vectors? Or each exact object we have its own heap vector, now we can indeed have like perfectly sized vectors for each type. We just need to know which vector we are accessing from. So our javaScript value becomes a tagged union and it has a byte size tagged which vector what UDP should look into and it has the heap index and what index in this vector should you access to find your heap data or of course you can have stack data in place of a heap index as well.
AAI: Um, so, as we kind of saw the previous slide we get better cache line by loading over the parts with a particular action. When we are getting stuff, we only need to get the prototype and the properties. So it is that all that we can do? No. Often we don’t really need the object aspect at all. So for the objects, what we do is that we actually drop the elements out of the engine entirely. Our objects are never array-like custom or not accuse custom to be array-like subjects. But you know you lose some and win some.
AAI: For arrays, we don’t have any properties or prototype array. That seems like I am breaking the spec here and kind of, but no. So we have the elements pointer for the array, and we also keep this kind of you can think of this as a custom internal slight. And a backing object lot and array and if do you something stupid and change the prototype or assign named properties aside from length, then we are going to create the backing object, and the internal methods of our array all kind of work by checking, “Are you doing something stupid? If so go to the backing object.”
AAI: For the other array, there is no properties or elements. And if you are doing something stupid with this, go talk to the backing object, otherwise, you will only get the array buffer features or the specific features for that object. So this is what our array heap data looks like. For or less there is a backing object and then the elements in the same structs. So this is not split for different cache size and so the backing object if it is not there, then it is a realm and we know which realms intrinsic we should use, to create it later, and that is 4 byte and a 32-bit integer basically and the elements that has a 2-bit integer which will give will length and then some stuff that will give us information on if the length is still writable and the well how to access the elements. And if we really want to kind of start optimizing and we can take the stack down to 4 + 8 byte and then we can go down to 4 + 4 + 4 if we wanted to and that would mean it would optimize length access better.
AAI: And obviously the common case for an array is to access elements or the length, and if you are looking for the elements and then you need to access it and so splitting it apart does not do much and this pessimizes, so assign it to the backing object. Any way you can think of this as like reversing object wrapping. So the elements themselves, they actually live in heap vectors. And they look like this. So, that is kind of a mouthful to say and lead but basically there is a vector arrays of a given size. So if it is 2 to the power of 1, that is 2 and then we have a vector of 2 array values and on the side you have the descriptors which is a hash map of particular index, and why? Because nobody does descriptors and why would you keep them or create them? Just throw them out, and if somebody does something stupid, then okay, we will create that descriptor for that particular index and accessing will be slow. But you should not be doing that.
AAI: So okay, we get improved cache line usage with this but this only works if the items are on the same cache line in the first place, and indeed, um, and luckily in GC system and we have axiom that most objects die young and that is why we do GC in the first place and if they were not true, GC would be mostly painful. And there is a corollary that I made up or kind of: and most objects live together, created at the same time, used at the same time mostly die at the same time. So, as we are creating all of our heap data, if they are created at the same time, they are created one much the other. They are most likely on the same cache line.
AAI: And when we GC our heap, we keep these items together. If there is—if items die from the middle and we pop them out and shift all the data down. Sounds like a future amount of work and I agree but does not mean that our vectors are always packed and the data that are together will stay together and we are likely to get this cache line performance going forward. Unfortunately, that means that our values need to all change. Our value objects points to index 300, and now 30 eye teams in index 300 before has been shifted out and now index should be 270 and you should change the value and relining the index. And this is simple to calculate and done parallel. This is not different from what most engines do. Apparently javaScript does not do this V8 has some packing and so on.
AAI: But this does mean that our identities are even likely to change in GC and we do rely on objects identity sometimes, so that is like um, let’s see, this will come up later. But okay, as a consequence, our heap does not fragment over time, and with the vectors, we actually also get kind of a separate nursery for free and we can design our nursery will start at index 100 and anything above that is new space and anything below that is in the old space. And if you are assigning stuff to old space, we mark that dirty or should mark it dirty and later in our minor GC we need to start the GC from the different parts but otherwise, this is kind of free.
AAI: So in conclusion what are the upsides of this? What do we see in the future of benefits from this kind of engine design in first of all obviously reduced memory usage because we are not creating the object-based for each array for each array buffer? We could actually slim down the object itself by 12 byte if we really tried, and we will not try that much. We also get the excellent cache line usage or the at least the potential for excellent cache line usage and especially considering this a dynamic language we are talking about and given that vectors are simply to read about and this is easily for a user to read about and if you are create object in array they will be next to each other and things work that way. Though of course, users should not rely too much on engine specific, but whatever.
AAI: Um the vector-based heap since it is just vectors, it is kind of simple to reason about. And the type union value JavaScript value that we use, it has no pointers. There is no way to get a JavaScript value from our engine and follow it back to the heap, and do the dirty stuff and there is no way to do a type confusion attack and the tags never change, and you can think of the tag as kind of being an identifier of which part of the spec is this object identifying. Is it a function with no external slots? Is a function with promise of internal slot? Each of these has own slots and internal slots and internal methods never changes. So if you are an attacker and you change the tag of some value, that does not allow you to reinterpret data in a different way. It changes both the type of your data but also where it points to. The vector, it points, the vector that you access data based on the tag so this is not type confusion. This is just a very weird way of changing your value to another value. And we get currently the value of 64-bit there. So a lot of stack value there and it is quite nice, this is quite a nice way to separate the exotic objects from ordinary objects. They just ask the ordinary object to do the stuff for them. But there are of course downsides because you know, trade-offs are a thing. So when you pessimize and so the array objects will be bad. So when you assign an index property into a normal object, it goes to the end of the properties, and I am not going to reshuffle the properties so fit the key properties, index properties at the very top. You are just doing bad things, and you shall feel bad. If you want—when you get the object keys, I will then rearrange them to the spec required order. And exotic objects if you assign properties to them, they do get an extra indirection.
AAI: And I said all of this internal slot case, they get their own tag and they get their own heap vector. And if they don’t do that, we will need to add the possibility of that stuff to the general case and that is not nice. Skip over that. Obviously the performance of GC remains to be proven. And the GC does exist, but it currently just mostly empties the whole heap. And it is not very great. Everything will fail after you have run it. Um, and whether or not the heap compaction and the index realigning can be fast enough, I don’t know, it shall be seen and I am hopeful and hash maps, hash maps damn and those will be nice and so every hash map you need to—so you will need to reinsert another slot. That might be very bad. Obviously as the heap size grows, these problems will get worse and worse.
AAI: All right, hopefully this was interesting, and that’s our ongoing work, and exploration into data oriented design using JavaScript. And there is a couple of links and so this is from a previous talk I gave and I have a ton—well not a ton but some amount of bonus slides if you happen to ask the presenter questions or if you are silent and so on. Any questions?
ACE: I was wondering if you have had a look at the structs proposal because that is a proposal that is kind of like secret thing that developers eventually learn about javaScript objects like how to fix a set of fields and don’t delete them or change the prototype and this secret that you find out and brings up first language of citizens and you don’t do these things and it does not have the vectorization in this and array does not need to contain the same structs but there is a lot of over landscape and wondering what your thoughts are?
AAI: Is that the Record and Tuple?
ACE: not that one, but similar. It is a class but you cannot change the property or the prototype and the things are basic values.
AAI: I have not seen that.
ACE: You should check that, I think you will be really interested.
AAI: That is definitely interesting. And I will hijack this to mention ABO who is one of the people working on this. And ABO has been thinking about implementing the records and tuples proposal and specifically with the idea if we can implement in nova why can’t any other engine do it?
WH: How do you represent primitives and strings?
AAI: Um, so they have their own tags. Currently our string is UTF-8 that is kind of by choice. Trying to see how big of a pain will it be to try to enter or pretend your UTF-8 will be a UTF-16 underneath and they will have heap vectors if they are heap data and we have integers on the stack, so we have a tag for integers and there is 7 bytes of data there which means we can represent all javaScript numbers on the stack without any heap data allocations but then when you go to doubles, then we do have a vector for those. So you will then have an index through the heap, which is where your float actually lives. Same thing for strings, basically.
WH: When somebody constructs strings incrementally, do you construct a new string each time?
AAI: Sorry I did not quite catch that question.
WH: I am curious about incremental string construction.
AAI: Yeah, um, currently, there is so little of the spec implemented that mostly this is like okay a future things to think about, but we do have a string concatenation implemented and a string while concat-ing and then we put it into the heap. At a new spring index and give you the index to that string with the proper string associated with it.
BEL: I have a question and you talk about optimizing caching and so it's optimizing for space. So does that somehow imply that it is also optimizing for time? Because you are kind of trying to make like the most used parts are access most easily without ever using lookups?
AAI: The performance of the engine is bad and it is very, very rough. But yes the idea kind of with the heap structure being vector-based and so on is that maybe this way, the important things like mapping over an array of items, doing a for each and those sort of things would be way faster because the things that you are normally doing are indeed those things that you the engine or the heap structure is optimized for.
SFC: Um, yeah so when area, and I don’t know if you have experienced or explored, so when you load like a cache line of 64 bytes and you are trying to store say integers in that space, you can store many more integers with variance where you basically try to bit-pack them and squeeze them in bytes as possible and this is a downsize, and for example, if you get a rust slice of issues because they are not easier to choose. And so this will get into intention what is more important bit packing or alignment and the other project I recognize which is a Rust project and there is zero vec which will favor the bit packing opposed to alignment of things because at least what we found in many cases, it is not universally true, but in many cases, you will get better performance but stuffing stuff in fewer bytes is less expensive to read so you don’t use the cache as much and that is advantageous. And I don’t know if that is something that you have explored.
AAI: No we have not. I do wonder like could we use it somewhere, so I said currently all of our integers javaScript integers, safe integers will stay on the stack entirely which is value of 64 bits at the moment. So we don’t have a heap vector of integers, where bit packing would become more, maybe more reasonable and can you actually index bit-pack into vectors?
SFC: It depends. So I guess one way to think for example a JavaScript safe integers is what 64 bit and it is not all 64. It is not super compelling but maybe you can fit 9 of them in the space of 8 something like that but more compelling you have a visible length—if you have 64 byte and the first byte is basically telling you where to get the rest of it, right. So there is definitely room and for smaller integers this could be quite a bit more efficient and we tried to implement STM which is like you have like a whole bunch of like weights into few bytes as possible and storing them in smaller bytes in less space will make thing more efficient. And it is a win/win but it is faster. And so again, it is not universally true, there could be exceptions that you need to measure it, of course. And it is just maybe an extra path that maybe an interesting area of exploration.
AAI: Yeah, I can’t think of easy places to apply it but it is definitely interesting. It might be useful. And one actually a potential place. Object-index or heap index is changing is a problem or becomes a real problem. And we might want to do a conversion where a heap indices are indirected, your index points to a vector that will point to the proprioceptor index, and then there the vector of indices might benefit from bit packing.
SFC: I mean avoiding heap is like a huge win, and so with a heap, everything runs to a halt, and you can do that and to keep the heap on the stack as much as possible and if it is bit-pack and less efficient to read but keeping the heap on the stack is a huge win.
RPR: Okay chip, you have 30 seconds until lunch.
CM: So if I understand this correctly—and I am not in the least sure that I do: A lot of what you are doing is exploiting object homogeneity, and so when you have a piece of code that the produces objects, it will produce objects that all have the same shape because it is the same code each time (unless it is doing something weird). But another thing that is common is where the origin of the object is from something like jSON.parse, which is interpreting data that originated externally, and while in fact it might be generating homogeneous objects there is no way to expect that is happening and I am wondering how do you deal with the things like this—essentially characterize the object as extrinsic and it is not visible in the probe?
AAI: The way we magic around this is the elements vectors that I mentioned. So we have a—or actually don’t have at the moment, and smallest element vectors is values of like object up to 16 values, properties. But I will take that down to 2 and 4 and 8, probably. So that the low end is kind of packed and now if you are parsing a JSON file, and you find objects that has two T’s and they will end up in the vector of two valued.
CM: So the interesting characteristic is how many properties rather than their names, so you can have one object with ABCD and another with DEFG and in your world they are the same shape—okay that is interesting.
RPR: Okay we got through the queue. And perfect timing. And so say that thank you AAI.
AAI: Thank you everyone and I said it was very nice to be invited and I will hopefully be joining lunch with you because I have missed my train.
(lunch)
Presenter: Ben Allen (BAN)
BAN: We are revisiting, rather a proposal from several years ago. At the stage of figuring out the design of it. So the main purpose of this presentation to get feedback from the folks from here on the direction we should go. Smart units
BAN: Smart units, the short version that you will see more on a later slide, is localizing measurements to the preferred measurement unit inside scales, in precision for locale. The goals just to simply discuss which of these localization is significant. Which is solvable and best solved by ECMA402. And there’s many different places where we could split off parts of this proposal. And getting a sense of whether or not people consider that appropriate, consider that the best way to proceed, would be really useful.
BAN: A non-goal is to convince you any particular part of this problem must be addressed or in a particular way. This talk is more about the discussion afterward that is necessarily about the slides. I will get through to the slide as quickly as possible. If I find myself going too fast, feel free to give some sort of subtle gesture indicating, hey. Turn it down a little bit.
BAN: All right. So here’s the problem: properly localizing content requires properly localizing measurements. It’s not just about using, say, a dot versus a comma for unit separators. And because content with non-localized values, numbers in unfamiliar scales or that are otherwise not formed the way they are, worse case, incomprehensive to users. I was going to rank those two problems. But the second problem is that content with non-localized values can seem strange to users, that is a series problem, with content incomprehensive to users. This is real problem. One of the great things I like talk about this sort of thing, is there’s all these great memes about it. Thank you. This comic I like because something towards the end of the talk. The final panel there, it also shows—indicates personal problems related to United States' aggressive cultural imperialism. All right.
BAN: I am resisting the urge to read that joke out loud. But everyone can see it and it’s funny. You can see it was definitely written by an American. The fact that guy is describing the height is with that level of precision, that’s a real tell it was an American. Also, the threats delivered at the end.
BAN: But if you look at that comic strip, the problem is primary, well, okay. Some regions use metric and some imperial. Sometimes the measurement units that are used in a given region vary based on the type of thing being measured. And this most often occurs in the commonwealth countries that use both, depending on the context. And also, the other problem is well, sometimes measurement units actually vary on them itself. They use different scales than smaller ones. It’s cross-culturally, the precision of a given numerical very well for a distance, say. It depends on, well, the size of the value itself. Typically, if you are dealing with a value that’s in thousands of kilometres, you are not going to be the precision isn’t going to be single digit—essentially, you typically don’t want to represent something that is about 1,000 kilometers as 1, 003 kilometers, unless you’re in a scientific context. Here is my other really fun meme about this. It’s one of the reasons why—no. No. I am going to tell the truth here. This meme is one of the reasons why I was excited to work on this proposal.
BAN: So the—as I said earlier, the countries that tend to use a complicated miss mash of units are commonwealth countries. This shows up in Canada and India. But if you are in Canada, and you want to seem like a Canadian, typically—side bar: a lot of the stuff in this talk I am saying things about how people do things in different cultures. And for most of those, the information I have on that is Wikipedia and CLDR. This information could be wrong. So I apologize if I am lying about your language. Please, correct me so I can then go and try to correct CLDR.
BAN: But so… At least according to this meme, if you want to seem like a proper Canadian, and you are giving a temperature use for cooking, you are going to be doing that in Fahrenheit rather than Celsius. If you do that this Celsius, you are not from here. At the bottom, if it’s related to work, use metric and not related to work, use imperial.
BAN: I alluded to this before: the precision that is used in formatting is itself context, locale and quantity dependent. So the problem isn’t simply that some places use metric and some places imperial and you want to respect that. The complexities are essentially trackal. And values that are true precision, to precise, for example, the height given in the first panel of the comic at the start they are not localized. They make it seem like you are not from around here. It’s weird to claim 1822.88 centimeters on your dating app profile. A lot of people claim to be 6’tall. If you people lie on the dating profiles, the lies should be appropriately localized.
BAN: Okay. So that preamble aside. The goal is, okay, it’s not possible to design an internationalization API to make all measurement localization automatic in all cases. We are not proposing something to all things to all people. Instead, to make the most common cases easy to localize. I don’t know. This is an analogy that probably—seems like resonated with me than anyone I talked to. But this sort of investigation of lenses from the standard measuring practices used, most often, I think, come up in context that are particularly important.
BAN: And sort of analogy is, the most commonly used verbs in languages have irregular stuff. In our day-to-day the measurements we have been doing as people in our day-to-day for a hundred years will use different measuring systems for dramatically different measuring systems than other types. The places they occur are important
BAN: Again, correct me if I am wrong, in India, most measurements are using the metric system. The height of a person, feet and inches is more commonly used. So that results in problems, but also some did you understand in the EU prefer using the height of a person of a mixture of metres and centimetres rather than simply metres. In this case, representing a height as centimetres, it’s going to seen as, this is probably written by someone who isn’t from here.
BAN: Here are some other quick examples. One place where this often shows up is with fuel consumption. So feel consumption in Canada for cars is actually most miles for imperial gasoline. And all summary one and a significant one is most regions will measure feel consumption in terms of litre per hundred kilometres. But many use litre per kilometre instead. So that’s less of a comprehensibility problem, but something that cause us to trip up when reading text.
BAN: I will go through these fairly quickly. So tell me if this is wrong. In Finland, the wind speed measurements given in metres per second.
EAO: Yes. It’s correct.
BAN: Okay. And yeah. This is a fun one. This is the Scandinavian mile used in informal context. This is 10 metres. From the outside, that seems like it’s primarily a thing that exists to play pranks. But—yeah. [sop] of these context are actually formal context. The example that I have seen is that there are tax forms in Sweden related to travel reimbursements that relate to the Scandinavian mile. Many regions will use the measurements scales for lengths of infants and the height for adults. In Canada, if you describe the height of a person, you use feet and inks. The 6-month-old, you would do it in inches.
BAN: Okay. So these are some contexts in which might be particularly important to localize content such that it respects the different cultural preference about. So weather applications, I think health-related applications might be typically important. Any sort of large city hospital system will have patients from hundreds of places. And it’s very, very important to have medical information available to people in a form they can read themselves. Another one I just sort of thought up about, let’s see… During another conversation about what I want say 12 hours ago, one thing with environmental organizing in the United States is that people will not take temperature values given in Celsius 3°C is no big deal and 5.5 Fahrenheit is a big deal. Even though they are the same value
BAN: Fortunately, all of the data to build an internationalization API is also in CLDR. I mean, it is CLDR data. It’s not necessarily reliable. But information on context and quantity dependent units for all locales in CLDR. And also the constants used for conversions. This is supplemental/units.xml. We’re dealing with about 8 kilobytes of data. There are some screenshots of these conversion constants. Pretty much all of the relevant ones and some fairly obscure ones which themselves could be relevant. Here is information that they have on which locales use which things. Again, the most common suspects are the United States and commonwealth countries. But there are variances in this sort of thing that aren’t directly related to metric versus imperial, and that occur in a lot of places.
BAN: And this is just a screenshot of some of the text on precision based on the size of the value. So that “geq” property that is a threshold, above the threshold, and if the value is above the threshold and then—there’s no other threshold above it that hasn’t crossed, it will view a specific precision and sometimes units change too. For example, in Great Britain, you will typically give distances in yards. Until you get up to half a mile. Then you start giving distances in miles.
BAN: The goal to provide users with content that uses comprehensible and cultural skills and only when localizing the measurements can be done reliably and deterministically. As an example of a thing we wouldn’t localize is clothing size. Measurements related to clothing size would be really, really handy to convert between European and American measurements. Something that claims to be one size from one company can be nothing like that size, the equivalent size from another company. Sometimes even manufacturers will internally have inconsistent styles because it’s not actually reliable. It’s not possible to say what a given clothing size is. Even though it’s really handy to be able to localize documents, localize content, it uses the appropriate clothing measurement scale for each locale. That’s something that is not on the table. Maybe someday in the far future, manufacturers will standardize clothing sizes, but not any time soon.
BAN: Okay. Another non-goal is, we don’t need to support every possible context-related measurement system variation. So, for example, it’s very frequent, I know this happens in the UK, I suspect this happens in other places. The measurement scale that you use for measuring quantities of beer differs from the measurement scale used for other liquids. So again, often foods—ingredients used in recipes will have idiosyncratic scales. We don’t have to support all of them and have to fully measure like a Canadian. We don’t have to have a way to flag something as a pool temperature that should be there for representing Fahrenheit.
BAN: And another non-goal is you probably went and looked at units.xml. There’s a lot of stuff in there. We don’t have to support everything. The goal here is to solve common localization problems, better some combination of common and important. Okay. We also have—I lost my place.
BAN: This is important, though. When we get back to the actual content of the API, what we’re going to talk about is not any sort of object for doing unit conversions. We do not want to provide a tool to do arbitrary unit conventions. If we were, it’s in the context of 262. Rather than 402. We are simply providing something through localized measurements. And we want to discourage its use for purposes other than internationalization. Because typically, and there’s people in this room who worked on 402 for a long time who have very, very specific examples, it’s relative objects to do non-internationalization work and that results in code that is breakable.
BAN: So using this for general purposes conventions is not our goal. Instead, our goal is localizing documents for specific locales.
BAN: Okay. Don’t need to support everything. Okay. One convenient non-problem, you could have an API for doing all of those conversions. We talked about so far… without providing any new fringe printing surfaces. It uses information that is already revealed. Specifically the locale that is being requested. And so yeah. Expressing these preferences and receiving documents in this way, doesn’t require revealing things about yourself. This does not provide new fingerprinting services.
BAN: Okay. So the proposed API is simply: add a usage parameter to Intl.NumberFormat that takes a string representing the usage. This string, it could be the name of it, in CLDR, so if you are formatting a person height, include the—the property person-height. The output and the rounding will be determined by what was the input unit, what locale, what was the value, which matters because the appropriate and what locale. Here is just a bashed together—this is what it looks like to use it.
BAN: You will notice something, so the expected I couldn’t tell would be 5 feet and 11 inches. That value is a little bit under 5 feet 11 I think. If it were—this is not properly localized because that’s not something that anyone would ever say. [?? This seems mistranscribed]
BAN: Okay. So there are a number of questions we need help with and this is the point of this talk. Question number 1: you might have noticed a lot of the preferred measurement scales for a lot of values most especially as we have seen that person height tends to be given in mixed units. So if I recall from the very, very, very, very start of the talk, the American, the thing this gets the American in the comic really mad, is given a height as 74 inches. Because very few Americans hear 74 inches and know how tall that is without doing mental math. This requires supporting mixed units
BAN: Question number 2: I said at the start. That refers to the topic of United States' cultural imperialism. We don’t need to solve all the problems caused by the United States. But we do have one big problem. A lot of people—what’s a good number? 70% of Mozilla users in Indonesia use the en-US. It’s possible that’s an outlier. Worldwide there are a lot of people who use that localization to the browser. And because the United States uses a unit system that is just truly baffling to most people, we don’t want to show people site content tailored for a measurement system that is not to the rest of the world. This is a real problem. I am interested in hearing people’s ideas on how to deal with this problem or whether the problem is available with.
BAN: My main thought or first thought was, well, this is something where there needs to be a permission pop-up. This seems silly, can we use your camera? Use your microphone? Do you really want to use feet and inches? Okay. This is something that has come up with conversations we have had mostly this week. This could maybe be used as a sort of site for user education. So if a pop-up says, you can provide the option of downloading something that is tailored for your region.
BAN: And question number 3. So we are proposing adding something to Intl, doing calculations. Should this be something that happens in 402? I think it should. And my sense of that is, NumberFormat is supposed to format numbers. We need to properly format those numbers to properly localize content. Otherwise, you get content that is kind of localized to your locale. But you can’t read any of the figures. So properly formatting a number can sometimes require doing calculations. And perhaps we should do the calculations that are needed for well formatted, well localized output and nothing else. Do everything we can do to keep people from misusing this for non-internationalization purposes. No way to specify the locale, to specify precisions not used in the locale and so forth. It’s not something doing generalized unit conversion. But formatting numbers for different locales.
BAN: That said, sort of the first idea that I have seen people have when I go reading through notes on the history of this proposal is, well, why don’t we just separate out of the conversions. Have it so people can flag given quality as the height of the person or as the distance travelled by road. And then let some other library handle it. I am thinking that doing it with 402 would let us better prevent this use. Providing the tools to do exactly what is needed for localization and nothing more than relying on one library to do it.
BAN: And then there’s questions we need help with, number 4, all of the questions we haven’t taught to ask. So at this point, I am going to throw it to the group for questions. I found since I started working on this, that this topic is a wonderful conversation starter. People like to think about the ways things are measured in their region. I would like to throw it over to you.
MF: Thank you for that fantastic presentation. You have already answered many of the questions I wanted to ask, which is great. And I especially appreciate the focus on doing as much as we can but not necessarily trying to hit every single possible thing that we could. So a question I have remaining is about discoverability. Who is the target user of this API? Is it a professional in localization? Is it the average developer? I ask that because if we want just anybody to be able to use this and properly describe the units they're working with, how would they know what kind of granularity to describe things at and what is available for them to describe about it?
BAN: Okay. There are people in this room who have thought much more deeply than me. My sort of shallow thought is, and roughly that same user category. It should be on MDN. There are better answers, significantly better. I am going to stand here awkwardly and wait for someone else to respond. I promise you, I can make it particularly awkward.
DE: I think MDN is a good answer for this question. And in particular, there’s a lot of surface area already for NumberFormat. MDN docs are pretty good, it’s possible to contribute more examples and things like that for the particular set of units that support it. Given that everyone’s working from CLDR, all the implementations, I think it would probably make sense to include the list of supported units in MDN, because you can’t just sort of make up a unit and expect it to work. So I guess I kind of want to ask behind that question: is there any change that we should be considering to the API based on making it more discoverable? I like the shape presented, but maybe other people have other ideas
BAN: What I was aiming for is the absolute minimal change, so the absolute minimal change. Just adding one option to NumberFormat.
MAG: So thank you for linking to the actual units file. Peeking through it, it’s interesting to see what is included, who is included, what is not included. It’s clearly been maintained in a pretty spotty, like, ad hoc fashion. So I wanted to know a little bit about—so how was this being maintained and then as it evolves, how do you see uptake happening through committees of like additions to this? So, for example, like, there’s no electric field fuel consumption there. I file a pull request on CLDR. And then when could somebody use it in a browser?
BAN: Right. That’s an implementation context—sort of like an implementation question relating get into IC4X getting browsers to use it and so forth. That from the context that there’s pull requests to be put in and I see. A lot of ones I don’t see. If you look at that data, there is a lot of fine grain detail related to imperial units related to UK and much less fine grain related… This is a difficult problem. One of the reasons I like this proposal, one, it improves localization. 2, it gives sort of a locus for improving our localization practices.
BAN: It’s a good reason to reach out to folks that are not currently represented in the CLDR database, not currently participating, and reaching out to people and helping people get into CLDR. Instead of making it an Anglo-centric document in a lot of ways.
SFC: The original proposal that added unit formatting, which was 2019, 2020, we picked a subset of exactly 45 units from the CLDR, that are currently supported in NumberFormat for unit formatting. And since then, we have basically left the door open for people to file issues. Why is this unit not supported? We received about a dozen requests since then, some more motivated than others to get a signal for which ones we leave that that might be useful for people. One thing that came up, in order to build an interoperable intl API, it’s important to have that list. If you don’t have that list, then basically, everyone—all engines have to do what Chrome does. It’s important to have that list because if you don’t have the list and Chrome supports a different set than Firefox or than Boa, then all the others, then like it makes it basically impossible to write interoperable code which is the point of our work. I think that with this case we will probably use some metric to select what subset of unit contexts are most relevant and useful to people on the web platform. Right? There’s some that I think that are definitely likely to be included, person height, those are popular. Also other context that wouldn’t be included. You know, and then there’s what's in the grey. Establishing some metric for how we decide what to include and not include is important. We established that metric back when we picked the 45. We should do the same thing here.
BAN: One thing like there’s a number of those values and units in—yeah. That are related essentially to scientific calculations. And we want to avoid those at all costs. Because it’s an internationalization tool, rather than something that is to be used for scientific calculations. I was thinking, there’s one odd ball one in here that might be worth including. One example is there are Differences in how countries measure blood glucose. Not commonly used, but for the people for whom its support is extremely important.
USA: Yeah. On the topic of how often we update with respect to that, every engine has its own kind of their own strategy, and most of the engines that I have come across use ICU to supply CLDR data. Boa uses ICU4X. But Google, Chrome and—Chromium and Firefox both package a version of ICU they upgrade. And Safari uses the system ICU. Every system update could—have information.
MAG: So Shane, you said that there was a metric when you do the first 45. What was the metric you chose? How do you figure out what the 45 were?
SFC: So the way we—the metric we used at the time and—was that we looked at what were the common quantities to measure, length, time, value and so forth and looked at what are the units used for measuring the quantities and the locales and that’s the set we got with the 45. That’s how we derived 45.
SFC: We looked at—we actually used some data from units from XML, we looked at all—at all regions. And, for example, the 45 units include all units used for measuring weight and measuring distance in all locales. So basically, the quantities to measure, once we picked the quantities, we picked the units based on the qualities, based on the localeData. We don’t have all length units, but we do have the length units used for measuring things. According to units at XML. It’s not a perfect metric, but it got the job done.
EAO: So one of my main concerns with this proposal is not at all the internationalization parts, but the usages separate from internationalization. A lot is about being to have input 1.8 metres and from there, get an output like 5 feet and 11 inches. And JavaScript developers will see that as a built-in unit conversion tool for which they will absolutely have use cases beyond localization and internationalization, and they will use this tool, however, they see fit. So my concern is that if we are defining this sort of capability to be happening internally, in NumberFormat, we need to account for this use or abuse by making it work really well also for unit conversion for by providing some other alternative way of getting the unit conversion output out of this.
EAO: So I think that needs to be a part of the proposal, from the get-go, before it can really go further. If we need to—because this seems like it’s a lot and it might depend on a couple different parts that could be packaged separately. I would think that a unit conversion which is not necessarily an ECMA402 thing, part of this should be one of the first parts of the whole proposal, to proceed.
BAN: And my sort of—my sort of naive take there is that one, it should be as useless as possible as a generalized unit converter. Essentially, the way I’ve been trying to think of it is that this is something that should provide a format unit and as little capability as possible for generalized number conversion. That said, so, for example, not being able to specify precisions outside of what is appropriate and so forth. That thought is, yes, of course, people will misuse it. So—had he been. What was the exact wording of the second part of the question?
EAO: So given that JavaScript developers have real world needs for unit conversion, they will use an API even if it’s clumsy, to do unit conversion, if that is a thing that is packaged into the standard library
BAN: So the numerical values for people who want to use it to abuse it that way, the numerical values are currently available via formatToParts. They could be surfaced in additional ways as well. Yeah.
EAO: Yes it’s possible to use formatted parts for this. But formatted parts is not the API that one would use if we were developing a unit conversion API. And given that part—this proposal includes within itself what is effectively a unit conversion API we should include within this a good unit conversion API, not one that we possibly make as hard as possible to use
BAN: I see. That goes beyond things like providing methods to pull out numbers, then using formatToParts.
EAO: I am not offering an opinion on the shape of the API should be. I am stating that there needs to be an API that is not an internationalization API or, like a null locale, provides an explicit way to say if you do unit conversion, this is how you get unit conversion out of this thing that also does unit conversion.
BAN: I see. So there’s no sort of like clean way to keep it as something that is 402, rather than outside of 402
EAO: If you provide a tool that does unit conversion it’s used for unit conversion.
SFC: Yeah. So we have definitely learned as Ben noted in his presentation and Eemeli just highlighted when we add things that do functionality that is in the exposed in a nice way, people use and abuse it. For example, people love to use Intl.DateTimeFormat to do timezone conversions, as it’s the only way in the platform to do that. Only way to do calendar conversions. They love Intl to do that for them. And like if we have unit conversion here, people are going to try to use it, which means which—one thing we learned to approach this consciously, and be like, well, we are going to have unit conversions. Programmers can abuse it to do their own unit conversion, they are not doing formatting. We have approached it knowing that this is just going to be the way. So I think that formatting is stable formatting. We don’t need to support high precision unit conversion. Because we have full control over how the units are rounded. We can round them by the XML rules for rounding. This can’t be used for unit conversion. If you want to abuse it, the best is maybe rounded to the nearest integer or whatever they are in unit inside XML. I know like use stable—so basically, like—we don’t have to expose a full powered unit conversion API. Only expose a way you do unit conversions in ways that are intended for end-user consumption and people want to use table formatting, that data accessing the numbers and results, they can. It’s not going to serve most cases for—people using conversions—they should still continue to bring their own library to do that.
BAN: I wanted to add, there’s two different paradigms at play. I like that one that you are giving. Specifically, because it’s not just that oh well. If people use this for generalized unit conversion there isn’t anything more precise than the integer. If people use this for generalization for distances, they will not get anything precise to the integer, and get anything more precise than a power of about 10. And then above a threshold, and so forth and so forth. Yeah, the limitations that can be imposed to—it is possible to have limitations for this that make it very useful for internationalization. Users will find lots of ways to use anything.
DLM: First of all, I would like to thank you for putting together the presentation. I think you did a good job making a case for this. I did appreciate your decision chart, it’s radically simplified in a lot of ways. One area where I think it is important to note that and I haven’t had the chance to look at this, anything related to construction is imperial in Canada. So lumber is in feet and inches. If you are ordering soil or anything, it’s sold by the cubic yard. And I think that may be an important use case, something we want to look at for CLDR. I am not sure.
DLM: I appreciate the API design as well. Like that feels minimal and the person height use case that you presented, I feel like that is actually fairly motivating. It’s easy and I had some fun going through a rabbit hole but this is sufficiently useful and not intrusive as an API. The one thing to question you about is unit pricing. Say, I am selling coffee or something like that. This API would let me show in pounds or grams, depending on the user. I could see that being a place where someone might also want to present how much of this is going to cost per 100 grams or per pound. That might be a place where people are tempted to parse the output.
BAN: There’s probably a counterexample I am not thinking of. It’s something that would be absolutely beyond the scope of this anyway because their conversions—if you are localizing a document to show grams, or and also a locale that uses pounds and ounces, those locales will use different currencies. I think the thing we were talking about right before this, this would be a more severe problem if there was any chance at any time and anything like the near future that the UK uses Euros. So that problem becomes a particularly sharp problem when you’re in regions that use different measurements scales and different units to represent weights, for example, or lengths. And also, the same currency. So yeah. That said, also, show—so 1., it’s something we couldn’t do with this anyway. 2, people will pull out the data to do the currency conversion which is a problem we are discussing in general. 3, there is something I would like to add. I should have looked more closely at unit.xml, which includes a lot of constants for things that are in units per something. So some of that—things that could be useful for that are already in CLDR.
SFC: The last time we raised this to committee was 4 years ago. I think it was in the Hawaii meeting. We presented this, and one of the ones later that year, it’s been about 4 years since we presented this to the committee. And one of the big pieces of feedback we got at that time was this problem of—it’s a problem that Ben highlighted in the presentation, question number 2 in the presentation, which was, a lot of people use en-US is the localization locale. They get US units. Did we solve this problem? We spent a lot of time investigating this problem and we—there is also a comment on the queue earlier that I have also heard around which is that units can also change and drift over time. Sometimes it depends on what age you are, what province you grew up in. It’s not region-based either. So like when he—basically, we can figure out what the units—like should be, we can use whatever information we have which is basically your locale, your language region. If you have fall backs listed, we use that to be slightly smarter. But it’s never going to be perfect. So the best this can really do is serve as an approximation, a best guess for what the units should be.
SFC: So the topic that I have is called snap poll involving units, thanks to Eemeli we had a really wonderful community event on Monday, that attracted close to a hundred local Finns who are JavaScript developers. And Eemeli gave me about 7 minutes of meeting time so sort of like, you know, give a give pitch for the smart audience. Is this proposal useful for this? Would you use this? Would you use it to spite like not having very good personalization of units and being able to guess and infer? And not a scientific poll by any means. But it was approximately about 70% by visual inspection. You know, raised their hand. Is this a useful proposal for? 70% is good. Would you use this despite the understanding that like anyone who has eu-US, it went to 40%. It’s motivated and useful. But there definitely is this gap where a lot of developers see this as, you know, like a proposal you’vesful, but it’s going to give—favor ENUS localization, that could be something that makes them less motivated. That’s why we looked over the last, few years like can we add locale extensions to the [step] language? Do other ways to sort of have a way that these units can be localized in a better way? And this is sort where the locale intention proposal came from and we have investigated that space a lot. I think there could be a room in that area. But it’s definitely been an area where we faced a lot of resistance in order to get that direction adopted. That’s why it’s good to take a step back and go back to the smart units proposal. I think the smart units proposal is well motivated by itself and I also think that locale extensions would be motivated if smart units was in the language. There’s small things we could do with locale intentions. Adding en-US with world units, one extra local locale and solve the biggest chunk of pain point. So I think there’s definitely room to improve this. So yeah. That’s my topic.
SFC: The short version; I think this proposal is still motivated and hopefully move forward and if anyone raises concerns, I think this proposal is fundamentally flawed because of this en-US issue, my response; I think it’s motivated enough and I do hope to solve the en-US issue in the future, once this lands.
BAN: I wanted to add: the problem with this is, yeah. People in a given locale, based on the age or subregion, can have preferences that differ from the preferences that CLDR have. And the preferences that are more generally common in their locale. So this is where we start rubbing against the line between localization and personalization. That said, presumably, the data we are using will continuously be updated and CLDR 2024 is not the same as CLDR 2022 and that’s good. For the purposes of improving localization. But also, in a weird way, it discourages people from using it as a number converter. In the units used in Canada for, let’s see, what is good example? Over time, fewer and fewer people use feet and inches to describe length values in Canada, then well, from year to year what the output of the—the output of number format will change. And that will kind of punish anyone who is using it as a unit converter. I have worded that incredibly awkwardly. But yeah. I would agree that the central problem seems to be the en-US problem, to me. And I really, really like to get a sense of how other people feel about that particular one.
JGT: So having gone through some of the pain that Shane talked about, about people misuse the API and this seems like an incredibly difficult problem to solve. I mean, simply formatting dates or numbers without numbers as is, is incredibly hard and incredibly abused. And so one thing that occurs to me, certainly when you like at why developers are excited about Temporal, one of the main reasons, they won't have to ship a 50k library with their app. And one thing that occurred to me is it might be difficult—it’s going to go large time to succeed for a platform technology, but something you could do is help libraries. So instead of shipping a 50k unit conversion library, you could ship a 2k unit conversion library and provide the underlying plumbing to allow the libraries to be successful. You might want to consider: instead of going all the way in the last mile to the end-user developer, could you do something that gets most of the way there, but you rely on user-land libraries to change more frequently and won’t be the kind of thing where you can never change the Swedish date format because it’s the only way to get ISO 8601.
BAN: There’s sort of a minimal version of this that would simply be the capacity to supply what the usage is of a given number so the library can pick it up. I don’t know. It resonates with me because if we are doing exactly what we can do, in 402, aiming for smaller rather than larger, that could actually reduce misuse. Rather than sort of like, we’re trusting a library to handle all the conversions.
SFC: Yeah. Next on the queue, I think it’s a good point. It’s something that we haven’t really spent as much time investigating as we probably should have. Not for this but for other types of libraries. API Intl.getUnitConversionFactor, it doesn’t do any conversion, which gives us the data in a way to package is interesting. I also worry it opens a can of worms. We should do that for basically everything else in Intl. Why are we doing this for unit conversion factors but everything else in intl? Right? I would eventually like to move in the direction of being able to have data-driven APIs. That’s no secret.
SFC: The last question was, Ben brought 3 questions to the committee and we haven’t answered any of them which I think is kind of bad. Because we came here with questions. We didn’t get answers to them, especially question 1.
RPR: I think we should begin summarizing now
BAN: Yeah. I mean, it is something, remarkably difficult to summarize. Because it seems to me that the sense in the room, is, oh, yes, this is something that would be very useful and the sort of pitfalls that we’ve identified are in fact serious pitfalls. So, like, what, in that case, what is a summary? What’s a non-objectionable summary that the room expresses interest in, has serious concerns how to or how not to make it useful as a generals all conversion today. It is unsettled whether or not it should have a minimal version and did not do the conversions but applies the plumbing for doing the conversions. That is very, very, unsummarized summary.
RPR: Okay, and are there any next steps you would like to state?
BAN: Oh, jeez, that’s a fine question. It seems like next steps might be determining specifically the things that are going to be supported, and additional work on ways to prevent PE problem.
RPR: If people want to work with you on this, did you have any venue that you’re running this in? Or should they just DM you?
BAN: So, there’s, I set up my repo on this. Currently the agenda as a link to the repo from about five years ago. But yes, I will update that link to my repo. And I’ll raise questions with the various people about performing.
Presenter: Daniel Ehrenberg (DE)
DE: I’ll be going through the presentation because RBN isn’t feeling so well. But he helped a lot with developing this content. It has been several years since we discussed cancellation, or previously cancellable promises. I wanted to review the current state of the world and what problems are solved and aren’t solved so we can think about what should come next.
DE: Cancellation mechanisms should help avoid unnecessary work. Like if you do a fetch and you no longer care about its results, then you can cancel it and then if it hasn’t already completed, maybe that will allow some network bandwidth to be used for something else. It can also be important to save memory by unlinking data structures, especially by taking certain callbacks and making them not referenced from certain places. Cancellation is often used in a certain nesting structure, which I will get into later.
DE: With cancellation, there is an important factor, separation of concerns. There’s both something that’s controlling the cancellation, a source of it. And there’s several things that might be listening to that, so they are cancelled when the cancellation is triggered. In order to actually make cancellation be used—in order to actually avoid the unnecessary work, it has to be threaded through a lot of operations. So ergonomics are important. I would like to solve the problem of having things we create in TC39 be able to work with a cancellation mechanism, which isn’t something that we currently have been able to do.
DE: The current cancellation mechanism in the web platform is AbortController, AbortSignal. There are two kind of capabilities, two objects. One is the controller which you can go and abort. controller.abort(). And the other is the signal that you pass down to things, which will be aborted when it’s canceled.
DE: There are also nice convenience methods in WHATWG DOM or HTML. AbortSignal.timeout is a shortcut, you don’t have to make a controller, do the setTimeout, abort the AbortController. This gives you a pre-made thing, because it is quite a common operation. That lets you have a fetch with the timeout as fetch(..., { signal: AbortSignal.timeout(5000) })
DE: AbortSignal.any is a new capability which allows several AbortSignals to be combined and triggered when any of them aborts. It turns out you cannot implement this in pure JS because of abort itself because of memory issues. Here is an example of an API that takes a signal, an AbortSignal as an argument to make something happen when the abort signal is canceled, you use addEventListener("abort", ..). so then if you want to use it, you can pass in that signal and, you know, it works just like with fetch.
DE: Let’s revisit how well AbortController and AbortSignal meet the goals. This is not a TC39 API, but an API we have discussed previously in TC39, or a problem space we discussed several years ago. I want to take this discussion to WHATWG as well. But if it is of importance to JavaScript also, so I wanted to discuss it here also.
DE: We want to avoid unnecessary work with AbortController. The .abort() method expresses this “don’t care” property. Propagate this message, you propagate the abort signal as a manner, to register the event handler in the abort signal as in the previous examples. For room for improvement, comparing this to other cancellation API, such as the Bluebird promise cancellation, there is kind of general need for a “reference counting” mechanism where you have several effective things that express interest, when the reference count equals zero, if the various things kind of end up being uninterested it gets canceled. I think this would actually fit well with signals. This is something I want to prove out over time. But the signal unwatch call back that computeds can take, may be a good way to model this, kind of naturally falling out. That’s “citation needed”, but I think this can compose well.
DE: Unlinking data structures: You may think that JavaScript garbage collection means that developers don’t need to about disposing of resources all of the time. But actually the way that web frameworks work where they have, their event handlers, they are always thinking about disposing of resources, because, for example, when something gets removed from the UI, you have to unlink the event handlers from that, otherwise there will be various sort of cyclic and partially connected data structures and things will be held alive longer, even just in memory, even if nothing ever gets triggered on them. The way addEventListener takes an AbortSignal as a parameter is a pretty nice API to compose with all of this. Because several things can be passed to the abort signals and provide as way to unsubscribe as a group.
DE: Room for improvement in unlinking data structures: RBN has point out, sometimes when you have a cancellation it gets sort of past the point of new return when it is not going to be canceled. Memory is potentially wasted when we hold on to the canceling reaction. Those event listeners for things that won’t be triggered. And even if you do trigger cancellation, you better pass { once: true } to the reaction so that disposes of itself, and again, examples in MDN don’t even do that. There is currently no API to drop all cancellation reactions related to an AbortController.
DE: Nesting structure: AbortSignal.any allows us to nest cancellation as a tree. And it might be a little bit upside down from how you would initially picture a tree. The idea is you have one AbortSignal and AbortController kind of at the top. And each child of it consists of making a new AbortController for cancelling this subtree, and then use AbortSignal.any([the parameter, the controller’s signal]) as your signal for work in this subtree. So then, if the parent is canceled, it will cancel all of its descendants, but also, you can cancel just a particular subtree using their own AbortController.
DE: So if you want to assemble a program, this could potentially be a DOM tree. Or it could be a call tree of basic functions. You often have pieces within there that have their own reason for canceling or you may cancel the whole broader thing. This is a common structure, it is important to have AbortSignal.any as a non-leaky way for this. But the AbortSignal.any pattern may be difficult for developers to understand. Maybe we want a convenience API for this common tree case.
DE: Separation of concerns. We discussed AbortController triggers cancel and it reacts to the cancel. From AbortControllers, you can actually get the AbortSignal. So, having an AbortController implies having the AbortSignal capability, but not the other way around. That is good. It is important to have that separation.
DE: But AbortSignal being based on EventTarget permits various different kinds of accidental misuse. I mentioned the way that you really should be passing { once: true }, there’s another thing which is that you can just call dispatchEvent on an AbortSignal and simulate an abort event even though that should be outside of what your capability can do. That should only be done by the AbortController.
DE: And actually, inside of the DOM, there is a built-in mechanism for reactions which doesn’t go through all of this. That separation is actually observable in sort of the ordering of when things run. Further, you can also override onabort from an AbortSignal. So it should be when you pass AbortSignal into things, they have the ability to subscribe to the cancellation, but they don’t have the ability to unsubscribe other people. But by overriding onabort, you do end up potentially being able to do that [if they failed to use addEventListener properly].
DE: Another question is how much will it actually be adopted by applications? This is one where, I’m not sure if between RBN and I we have a fully common answer yet. But it is something that I personally care about. I think, we would get higher adoption of AbortSignal and AbortController if we pass it through implicitly. Right now, every layer of the call stack has to pass the AbortSignal through. There’s a convention to use an options bag. This gives some potential for adoption. And I have been seeing some more adoption and some frameworks doing more to pass it through.
DE: But if we can use an AsyncContext variable, maybe, one that is provided by the platform and the variability isn’t exposed, but it is just set by the platform to hold the current ambient AbortSignal and be read by operations maybe if they opt into reading through this signal inherit option.—RBN was explaining that we don’t want new programs with new APIs, but with this argued it is still easier to use this than explicitly adopting abort signals down in the graph.
DE: Finally, we currently have not been able to use cancellation from TC39 proposals. We’ve been chatting informally about TC39-defined timers in Matrix, informally. You may want to unsubscribe from those. Another thing is signals: the proposed API does have unsubscription through this unwatch API, but could not use AbortSignal because it is a web API.
DE: There are a few possible strategies. One is do everything in the web platform. I think there is value in TC39, we have good integration into ihe language that enable cancellation functions if those make sense. We have a good connection with the community and diligent about doing things, I hope we can come up with a way this group can also define cancelable APIs.
DE: One strategy is to define host hooks such that AbortController and Abort Signal remain web APIs, but we can accept AbortSignal as an argument to TC39-defined APIs and react to cancellation. That feels a little uncomfortable to me, that means that JavaScript needs that web API, but an option to consider.
DE: The other one is to a move AbortController and AbortSignal to TC39, but that seems messier given the EventTarget integration. Making that happen about a way it is not observable changes it seems difficult and very messy.
DE: Or we can even consider defining a different cancellation API, which would have to work well with the AbortController and AbortSignal. It would be pretty rare and unfortunate to define multiple APIs for the same thing. I think the name chosen for AbortController is particularly unfortunate.
DE: For next steps, I want to work together with other people, including RBN and others, on investigating these problems and explaining them in more forms. Working with ecosystem libraries such as Ron’s library cancel token API, and server-side frameworks, and see what we can do to work between TC39 and WHATWG on cancellation. If you want to get involved, please get in touch with RBN or me. This is a very early effort.
RBN: Some concerns with the idea of ambient AbortSignal. I’m providing some of that feedback in chat. It is something we discussed on-and-off in various meetings. I think, the last time this may have been discussed was around 2017. But again, if you want to talk more about this and my concerns you can probably talk to me offline or on Matrix. There are a lot of different bits and pieces in this to go into that are more what we have In the short discussion today.
DE: Yeah, I wanted to bring up the discussion at the early stage because I’m also curious whether people see this as an actual problem. Maybe I’m making a big point out of nothing in particular.
NRO: Yeah, I’m happy to say, this is being explored again. Personally, as a developer, I have been using AbortController quite frequently since I discovered it exists. I never had problems with it being there, because it is present in both Node.js and the browser. But it still feels like the context where I use it are not really anything Node-specific or browser specific. So it feels weird to have to go to platform APIs to separate from the language itself.
DE: Anybody used abort control other than Nic? How has it worked out for you?
CHU: It’s especially useful for business. Because, to, to remove them. Because the other way, to remove them it is awkward, like they have to have t
LCA: One thing we use them for is our HTTP server, takes a request object that passes in a signal that gets aborted when the user aborts the request, which happens very frequently. Users can pass the AbortSignal to downstream requests they make, for example, get aborted. We see actually quite a lot of adoption of this. But it requires support from libraries. If they have a library that makes the call under the hood and not called through the signal, they have no way to abort this. There has to be an upstream API to wire this through. I don’t think we have a good solution for this. I agree with RBN, the signal needs to be an opt-in, if it is inherited. If there is a way not to record that, that would also be very nice. It would mean that users would not have to think about cancellations written libraries and someone that needs to make cancellations could make it work without upstream changes.
KG: Yes, I have used AbortController. Yes it is very useful. To answer Dan’s question.
PFC: I haven’t used AbortSignal and AbortController on the web platform so much yet. But I have used the equivalent on other platforms where you have to thread an object through all of the async operations. And for me, it makes a lot of difference whether you come from a position of wanting everything to be cancellable by default or want it not cancellable. There is no need for something to be cancellable if there is no UI cancel button for the user to press. But on the other hand, maybe your goal is to get people to design more UIs with more cancel buttons. I was curious whether you want this API to take a position on that?
DE: Well, it has to take a position one way or the other. The current position is not threading through. I thought it might be useful to do the automated threading through, but there are these costs.
RBN: And add onto that, my concern with automatically threading things through is that, for everybody that wants, that looks at a package and says this package wasn’t written to by cancelable, therefore, I have to commit an upstream PR to make sure it can be. There are passages that have aPIs that except to complete in the best case scenario, like say a three-phase commit transaction in a distributed transaction system, you want to make sure that I’m going to submit the request and go through barring a worst case network interruption, you don’t want someone navigating away to break that in an SPI, where the actual, like, code is still running and still resident in memory. There is no reason it should stop, but suddenly that code taking—or supporting cancellation even though it wasn’t written to handle it is problematic because it could break an existing assumption about how code runs, now I have to write defensive code around my code that I run without any type of cancellation, I’m back to the same thing we have to do today for saving off primitives and intrinsics at the start and not making sure that someone is poly filling or hijacking my code, now I have to do it for every API. So I have to write defensive code around something that should be ideally passed in, because that is the appropriate separation of concerns.
DE: Yes, that’s why in the slides I’m suggesting explicit { signal: “inherit” }.
- DE raised several issues with
AbortController/AbortSignalwhich affect its reliability and usability. - Many TC39 delegates have found
AbortController/AbortSignalto be a useful API. - Some interest in ensuring that there is an API for this functionality “in JavaScript”.
- RBN explained that threading through
AbortSignalimplicitly, via AsyncContext, would cause unexpected effects on existing code, possibly affecting their soundness. This may be mitigated by explicit opt-in (e.g., { signal: “inherit” }). - LCO, PFC saw both sides to the threading question (about whether the
AbortSignalshould be threaded through via AsyncContext and used by default). - Unclear whether committee members have run into any of the other issues described aside from implicit/explicit threading; no discussion about this.
- DE and RBN to work with WHATWG and TC39 on these issues and invite others to join them.
Presenter: Michael Ficarra (MF)
MF: This is joint iteration for stage 2.7. So as of when it was last presented, there are two methods being introduced. zipToArrays and zipToObjects. zipToArrays takes an iterable of iterables that is self-explanatory. And zipToObjects takes named iterables, that is an object whose own properties are iterables. zipToArrays produces tuples that align with the input iterables. And
zipToObjects produces records whose fields align with the names of the input iterables. Each of these methods also takes an options bag as a second optional parameter. There's a longest option, which changes the behavior from the default of shortest—which means stop iterating once any iterator that you’ve been passed has completed—to longest, which means we can continue until all of the iterators you have been passed have completed. There is also a third mode of operation that can be enabled with the strict option. If all of the iterators don’t yield at the same time, after the same number of elements have been yielded, we will throw a type error. And the final option that can be passed is padding. This is to be used with the longest option as the values that are used to, you know, used in place of what would have been yielded by already exhausted iterators.
MF: Hopefully, that explanation was clear. I felt like I was a little rambly. So there are three open questions at the moment, which I think we can resolve in real time and still achieve stage 2.7 today. The first open question is the names. We’ve gone back and forth on this a little bit. I originally proposed that these two methods were actually the same method that distinguished based on the type of the first value. But committee generally leaned towards splitting them. So we split them into these two methods, zipToArrays and zipToObjects. That’s what they are currently called now in the proposal repo. Some floated the idea of maybe calling zip and zipToObjects. Personally, I would lean towards keeping things the way they are, zipToArrays and zipToObjects. The reason being, I feel pretty strongly that when using these zipping methods, once you get past, you know, two iterators, that it is best to start naming them so that you can better align the use site with this zipping site. But I guess I’m open to change it, if the committee is overwhelmingly in the other direction.
MF: The second open question, how do we treat strings? Historically strings are iterable. But with iterator helpers we have diverged from that. We can actually test when an iterator passed is a string and not iterate it. So this pull request, the open pull request number 25 does that. I can go either way on this. I can see good arguments in either direction for consistency, of course.
MF: And then the third and final open issue is whether we pass longest and strict options, these are the options that control the behavior around when the iterators yield a different number of values. Whether we pass those two options or instead pass a single option that unifies them into this three-state option. There are pros and cons here. Obviously, we want to make illegal states unrepresentable. The two Booleans make four states or three legal states. That means there is an illegal state where longest and strict are both true. The alternative is passing a mode that is the union of these three strings. The downside of that is like, technically now, we have an infinite number of possible values if we just say this is typed as a string. You know, you can mistype shortest or whatever. Maybe we could just make these constants on the Iterator object or something, like they do with DOM node types. But you know, that doesn’t make anything better, that just adds a layer of indirection. I’m not really sure. I do want to solve this problem, but I don’t think we have good solutions. We just have these two. I think, either way we go would be fine, though. And it’s a very straightforward change if we do move from one to the other in the spec text. So, I also don’t think that would prevent us from going to 2.7 today.
MF: I wanted to notify the committee of this decision that was made. This was a change from the last time this was presented. Previously, when you passed no iterators to these methods they would yield an infinite iterator of empty values, which, depending on your mental model of how this operates, it is consistent with the cases where you do pass iterators. It just depends on the mental model you have. But we changed it to instead never yield any values. So, they return already exhausted iterators. And that is more consistent with the vast majority of other languages and libraries. So I felt this was a pretty clear-cut decision to align with them.
MF: It was brought up in a couple of our meetings discussing this proposal by JHD that we should have array specific variants of these. As a reminder, these do work on arrays, arrays are iterable. But array-specific ones would work through different means, through array indexing instead of iteration. That may have pros and cons. And there also may be pros and cons with the clarity of the code produced if you’re referring to arrays when operating on arrays. But, at the last meeting, I agreed that if it was completely uncontroversial within committee that I would include it and we could send both of these through together, but on the thread that I directed people to, we were unable to resolve those differences and it’s clear that it is still controversial. It may still advance in its own proposal, but it will have to be independently justified.
MF: So otherwise, this proposal has not changed since stage two. I have full spec text. I have a polyfill in the repo. A demo on the repo’s website. I have basic tests. Not test 262 level, but all of the tests for the expected functionality. I received reviewer sign-off, from the reviewers jHD and NRO. I would like to ask for stage 2.7 once we have resolved the three questions that I have run through. I’m ready to go to the queue.
RPR: Okay. Ashley is on the queue.
ACE: Hi. Yeah. So when we talked about this in Bloomberg during our review, we had a preference for just zip, I think that’s, so—much precedent in utilities and other languages, I can see being explicit has advantages, but the preference, the shortness of zip for the common case. We would like, if the group of you feel the same, adding our voice to that group. Move onto my next thing. We will set a preference for the string enum rather than the kind of Boolean session.
MF: For the mode?
ACE: Yeah, for mode. Yeah.
RPR: And—obviously, you already have the items, Ashley. So John?
JKP: Sorry, I don’t think I saw Ashley was typing as well. I was voicing a preference for the mode with the multiple strings over the other. Thank you.
RPR: RGN?
RGN: Yeah. I think the, the string valued mode, it already has precedence in 402 as we’ll hear about later there. Are also in plans in Temporal as well. I don’t think we should have any reservations about extending that pattern.
JHD: Yeah. I guess, I’m confused that it’s controversial. I mean, for the array-specific variant, it is certainly fine if the proposal proceeds without it. But the semantics of an array-specific proposal or basically set by this one, so I’m not sure other than, than having separate conversations about independent justification, I’m not sure what else there would be to discuss. It’s clear from previous plenaries that it would not be an array-type method. So it would basically be Array.something, and something is whatever the method names are in this proposal.
JHD: So, if we’re fine with this, I guess, yeah, I’m not sure I see the benefit of splitting it up. And if, if I were to make a new proposal, it seems like it would almost immediately go, you know, to a somewhat advanced stage given there’s virtually no design space in the wake of this proposal.
RPR: All right. This is SYG's prepared statement. I’m assuming he is not on the call right now. It says:
V8 has the following concerns for Stage 2.7, but not blocking if they enjoy committee consensus otherwise:
- Dislike the current method names
- Dislike using two boolean options for mutually exclusive options instead of one option with string constants
- Supports omission of Array method because
toArray()existsSee this issue for details.
JHD: The link to the issue doesn’t discuss the array thing at all, but it does apply there is a comment in a different issue, which I’m trying to look at right now. The, the last comment is essentially, from what I’m inferring is that the last statement of SYG’s comment on the issue number one on this repo is that toArray is not that inconvenient. So in other words, the general arguments I have heard is that using the iterator form and then using toArray is acceptable for some and unergonomic for others. I haven’t heard of a lot of strong arguments that I recall against the array variant beyond that. So it would—you know, be helpful, I think, if someone, if anyone who had any, like if you have an argument that we haven’t, the arguments we already heard are you can already do it because arrays are iterable. And it is fine to do two array on the iterator variant. If someone has an additional argument, I would love it if they can throw it on the queue. At this point I’ll be done speaking. Thank you.
Mathieu has a reply.
MAG: Yeah, there we go. Just I kind of don’t want to try to do the array prototype thing every time we try to land something on the array prototype.
JHD: I was saying Array.zip a static method because of the prototype methods.
MAG: I heard you say prototype and I thought you meant prototype method. Withdrawn.
JHD: I was echoing the dislike you’re describing, because it’s been shared in previous plenaries, so I’m assuming there would have been really compelling reasons to did an appropriate type method, and in this case, a static method is perfectly fine.
LCA: Yeah, I have a weak preference for zip and zip to object, but don’t really go either way.
MAG: Yeah, just I kind of don’t want to do the array proton type thing because every time
LCA: It is not array prototype.
JHD: I was saying array.zip at this time. The array problem. I was ACK laying the dislike that was described so I was assuming there would be really compelling way to do array prototype method but in this case static array method is totally fine.
Okay good. LCA?
LCA: I have a weak preference to array for the thing of whether we should—what to do, but string case. I think there’s—I think as a committee we should make a decision on this and apply this generally, we have talked about whether we want to coercing things or asserting a couple of things a few meetings ago and general conclusion that we will not curse things and detour of what is wrong and we have a similar discussion a couple of weeks ago and the conclusion for now there is also that we do not want to sort of do anything—we do not want to implicitly iterate strings and adone done. So I have a preference to not implicitly iterate strings and I think as a committee, we should make a decision whether we want to apply this to future things.thing or we do not want to inclusive iterate strings and rather inclusive abdomen, so we have a preference not to iterate things or whether or not we will apply this to future things too and finally on the string case I think we should use the string and using them there is many ways that can use string in fetch and many other one too and think there is—yeah people are familiar with this and it is easy to use and it removes this case of specifying two arguments.
MM: Okay, so I am just concerned that to make sure that the iterator helpers and the iterator helpers are coordinated that people have minimal surprise going to the other and not a detailed answer but what is expectation that parallelism between like proposal of iterators in what you expect to show up for these writers?
MF: That's a good question. For the things that are in the stage 3 iterator helpers proposal, we have a very good idea of what their counterparts in async iterator helpers look like. For some of these follow-on proposals that I have done since, I haven't really tried to make an assumption about how they may appear in async iterator helpers. And I don't know how helpful it would be to do that without actually going the full route of pursuing a proposal because, as we've seen when we started working on concurrency in async iterator helpers, and as we've now started to see when we've started working on the unordered space, concurrent task management, these solutions might look a lot different than we initially expected them to until we did a lot of work. Maybe it'd still be worthwhile doing some amount of work to try to get an idea, but it's not going to be very reliable.
MM: So that is easy for me to volunteer to do work but I will say that really I think that some moderate amount of work to do a check to see if expected similar API’s seem reasonable for this iterator helpers and enough to at least spot any problems because if we do something that accidentally makes things more different than the corresponding thing simile iterator helpers that would be a shame.
MF: I agree, and I think that it is worthwhile to spend a bit of time at least thinking about it.
MM: Okay, thank you. Um, and if you would like to do some separate brainstorming on that, that would be fine. Thanks.
MF: Can you repeat that, I did not hear that?
MM: It sounds like it might be something that you know some interactively brainstorming and it seems like it is not very much examined yet. So talking through the issues through and volunteering to the spring board.
MF: I am still having a bit of trouble actually hearing but I think you are offering to work together on that?
MM: Just to you know, essentially yeah. Just not a lot, but yeah enough to do initial exploration. Yes.
MF: That is exactly as much as I would like to as well.
LCA Yeah, I just wanted to say, I think we should do this analysis, and I hope that we can just point this over to once we figured out concurrency for iterators but yeah I don’t think there is anything blocking us but we should definitely take a look.
RBN: I will say this for message one and I have a preference for zip mostly because that is what it is called in every other library that you look at and especially what developers are familiar with so looking at zip 2 array and adding the extra thing for what is most likely going to be the most common case seems like unnecessary overhead for users. So I don’t think we need to array part of it, that is fine. And for the other topic, discouraging implicit array position and I held that the implicit iterator on the string prototype has been a mistake. I think string iteration is great and I think we could have easily had string prototype values and be done with it. Mostly because it is a poor developer experience for most cases. Because JavaScript is untyped, and not type checked and we don’t have an individual thing that represents a character in a string. It is very easy to it iterate over a string thing you don’t reaped that you are iterating over a string and not something else and it is easy to get the wrong thing in many cases. And if we already have this explicit iterator that can you get by calling string.values to get the actual characters manually and you can iterate over and that is generally what we suggested iterator helpers API and we worked with.
[ WRITER SWITCHOVER ] object iterators and that is why we have had to add things like symbol into an array. And we recognize this is something to avoid and continuing to avoid it is a good thing and we should discourage API and it is they made a mistake in their code and not what they are expecting and if that is something they want and they manually reach for it and say this is something I want to do.
KG: This is on the first topic. I am fine with the names zip and zipToObjects but I want to point out that encouraging zip the array form is the more convenient thing is a little bit unfortunate because the natural way of using it, is to destructure the results with iterable destructuring with the structure for destructuring whereas the natural way of using zipToObjects is to do the sort of braced object destruction thing for the thing the zip iterator produces which does not invoke the iterator protocol. It's unfortunate to invite use of the iteration protocol when it is not necessary and it is unfortunate if we agree to give zipToArray the good name it will encourage that overhead. With that being said, I do degree that zipToArray is known as zip any other language, but it is unfortunate that we are giving the more convenient name to the less performance thing.
MF: And it's not just about performance, there's also readability associated with it, which I mentioned earlier. Once you have more than a couple of them, it's really hard to align visually. You have to count, this is the fifth iterator that I've passed, so it's going to be the fifth thing that I destructure. It's just really, really hard to do as a human.
RBN: I would say most of my use cases of zip anywhere else has never been to destructure the results. They have been to merge two things and work with the results usually by continuing to do more things to the query, almost never have a case of destructuring resulted zip.
KG: Well but a common thing is that you zip two things, and then you map over the result. And what you do with the mapper function is, you take the two items out of each result and the way you take those two items is by destructuring.
RBN: I see, yes.
MF: So in my experience, that's almost every single time I use it.
CDA: SFC?
SFC: Yeah, I was on the queue earlier and plus one of string and Boolean are bad with one and we had examples where we had options and we decided to change Boolean with Boolean plus string because it is boolean is not expressing and this happens over and over again and basically they are like every API is like it is clearly a Boolean but not a Boolean, but stringing them is definitely the way to go. Duncan?
DMM: Oh so, sorry. I definitely don’t like introducing more implicit string iteration because it is what we use as an API and it is normally confusing. We have a string proton type symbol iterator that I think this is an area where we could improve the language because when you do want to explicitly it rate over a string by characters or something like that, that does not give you the things that you want. Because your code is now too complicated for that to be the thing to do. So I think in what we should think about on the string iteration is swift API for iterator a string that will give you the actual principal characters which what user of API will get a hold of and it is a tricky thing to write. And get correct, and it is something that we should probably consider putting in language.
CDA: That is it for the queue. Is there a reply for Dan?
DE: As—oh we have graphing iteration in the language and we have Intel segmenter where you can pass the graphing flag and it will iterate through graphings. And any way, I agree with others that not iterating through strings sounds good and I support stage 2.7.
MF: Well, um okay I would like to just review the decision points here. It sounds like on naming what I have heard is that we should use zip instead of zipToArrays. It is not my preference but we will make that call. We are all in agreement that we should not iterate strings and use that to set a precedent for further API’s that accept iterators. As far as the mode specification, the overwhelming majority opinion, if not unanimous, is that we pass a string for the mode option and not 2 Booleans. With that being said, MM did ask for us to spend some time looking into how this would align with a possible async iterators variant, and I am not sure if we should be asking stage 2.7 for this. I don’t know if KG was on the call when Mark asked for that and if he has thought about this since KG has been working on async iterator helpers.
KG: I have, and I have thought about it. And I am pretty sure it works completely naturally. You like each call to dot next on the result of the iterator zip just as one call to dot next on each of the underlying things and bundles up the results. And you don’t like wait for the earlier things to settle. It is just completely the obvious thing. I will not promise that is how it is, but I don’t think any of the—I don’t think I see a way that the design of synchronous iterators could possibly need to change in order to accommodate the sync iterators.
MF: How would you think of the risk here moving forward with Stage 2.7 with only the amount of thinking that we have done so far?
KG: I am not worried about it.
MM: So yeah let’s check. The fact that KG has thought about this and has that fairly confident impression is what I am looking for. I just need a sanity check, and it sounds like Kevin has already on that. I would like to do more exploration, but at this point, it’s more of a just in case something is overlooked which is lower probability. So, yeah, I’m happy to go forward with 2.7 now and then we just double-check it in case there’s a problem.
MF: Okay, if you’re happy with it, then I’m happy asking for 2.7.
CDA: Okay, we are just about out of time. JHD, do you want to be brief? I see you’re on the queue.
JHD: Yeah, just so with those three changes MF just discussed, I do support 2.7. But just to be very clear, my intentions are to come back at the next plenary. I was going to do it this plenary but I didn’t because I thought it would still be included, but I’m coming back with Array.zip and Array.zipToObject proposal. So my intention to ask for some form of rapid stage advancement depending on if I discover any issues or not. Because it will just be cribbing the semantics of this proposal. So if somebody has a new argument against it besides duplication and, you know, ergonomics, I would love to hear it between now and then. Thanks.
CDA: All right. So you have support for 2.7 from JHD, from MM, from DE. Are there any other voices of support for 2.7?
MF: We have LCA raising his hand in the room.
DE: I think thumbs up from Chip.
CDA: We got a +1 from Daniel Minor. All right. You have 2.7. Would you like—I know you kind of summarized a little bit some of the issues a few moments ago, but do you want to provide a summary and conclusion for the notes?
MF: I’ll do it real quick. On the naming issue, we choose to rename zipToArrays to zip. On the string iteration issue, we chose to not iterate strings that were given as input. And on the mode selection issue, we chose to replace the two Booleans with a single mode option, which is one of three strings: "shortest", "longest", or "strict". And that’s it, advance to Stage 2.7.
Presenter: PFC, JGT
SYG's prepared statement:
V8 strongly supports this scope reduction and thanks the champions for being so open to late-stage simplification.
V8 will take all the reduction it can get: the smaller the proposal, the higher the likelihood it can be implemented and shipped. That said, we don't feel super strongly about any particular method's reduction. For example, we see where developers are coming from for wanting
subtract(): it is a ubiquitous convenience, even though subtraction everywhere else also means addition of the negation.
PFC: For the rest of the afternoon, we are going to be talking about Temporal. Hopefully we can finish a bit earlier than 90 minutes, but we have a lot to discuss. My name is Philip Chimento and I work for Igalia and I’m doing this work in partnership with Bloomberg, and one of my co-champions, Justin Grant, is going to be presenting some parts of this as well. First a short progress update. I don’t remember if we announced this in the last plenary or not, but the long awaited standardization of the ISO string format with timezone and calendar annotations that we’ve all been waiting for is now an official RFC. It’s RFC-9557. A big thanks to the USA, among other people, for keeping that moving through the long and not entirely pleasant process. We’ve got some active implementations going on in SpiderMonkey, which is being done by ABL, I believe. There’s an implementation going on in Boa, which is being done by a few people, including JWS over here in the room. There’s a polyfill implementation being pursued by Adam Shaw as part of the Fullcalendar organization. And all these have been incredibly helpful for finding both editorial issues in the spec as well as some of the bug fixes that we’ll be discussing today. So one part of this presentation is going to be presenting two minor normative PRs to address issues that have been reported by implementers. One is an arithmetic corner case bug and the other is a non-ASCII character this the ISO 8601 day string grammar. And then the bulk of the presentation will be talking about how we are responding to feedback from implementations.
JGT: And, boy, have we got some feedback. So, you know, so we just sort of screenshotted the purpose of Stage 3 here, which is to understand when people are going to build production grade implementations to, you know, help expose issues and to get feedback from implementers. So what we’ve really been absorbing over, I guess, the last six months or so has been some very strong feedback. I would say from all, you know—from Google, from Apple, from Mozilla, and others. That Temporal is too big. And it’s interesting in that often times in the committee, we focus on sort of what is the right thing to do for the language. And we have to convince each other that these things are the right things, and in this case, the pressure is really coming from outside, right? It’s coming from not necessarily people in ECMAScript, but people who are responsible for Apple Watch or for Android, who are concerned about the additional added size that a proposal the size of Temporal will provide. There are some engine-specific things, so, for instance, the number of functions turns out to be particularly expensive for V8 because of the way V8 stores each individual function. And, you know, obviously we’re all using laptops that have, you know, many gigabytes of storage, but if you’re talking about a low-end android device, even something like 5K or 10K can be a significant issue for engines. And so in addition to size, just complexity itself in terms of the difficulty of implementing for implementers, and two things that have sort of bubbled to the top here is custom calendars and timezones in Temporal involve calling back into user code. We’ve heard a number of concerns from implementers. I think Dan, you’ve—on the Mozilla side, you guys have definitely raised that. We’ve heard similar concerns from Google. And so today we’re going to go through and try to propose some solutions to these concerns.
PFC: That’s going to be the bulk of the presentation. But first I’d like to go through the normative issues and discuss those first. Get that out of the way. So the first one is a bug that Adam Shaw, the polyfill author, found, in an edge case when you take the difference between two plain date times, near the end of the month, when one has an earlier date and later time of the day and one has a later date and earlier time of day. You can be off by two days because of a sign pointing in the wrong direction. Thanks to Adam we have a fix for this and also improved test coverage as part of fixing this bug. Here you can see a code sample of the exact input that would trigger this bug and how it is not correct to count one month from February 28th to April 1st. It’s one month and three days. So this is the case where we just gave the wrong answer and we need to correct that.
PFC: The other pull request that we’d like to discuss today is dropping support for parsing the Unicode minus sign in iso 8601 strings. This is a suggestion from Boa. The proposal contains a grammar of the entire ISO 8601 string format, plus RFC3339 and RFC9557 now, and the exact variations that are permissible by those standards, which of those variations we support and which of those we don’t. ISO allows minus signs in these strings to be Unicode minus signs rather than ASCII hyphens. This is not something that the RFCs allow, so it’s kind of ambiguous and probably not used very much in the wild. It complicates writing parser code that could otherwise operate only on single byte ASCII strings. Now, one place where minus signs might crop up is in the names of UTC offset time zones, which since not so long ago, are supported in ECMA 402 as legal time zones for Intl.DateTimeFormat. So this is a normative change that applies outside of Temporal as well. There is a PR for ECMA-262 changing the grammar for offset timezone names. You can see in this code sample here that there is an observable difference in the code, even for engines that don’t yet implement Temporal. So, yeah, before we discuss the removals, I’d like to ask if there are questions and call for consensus on these two normative changes. I’ll give a moment for people to add themselves to the queue, if needed.
RGN: I’m specifically hoping we have comments from one or more implementers on the second one, because it is already widespread that we’ve got support for this, so if we are going to change it, they’re affected more than they otherwise would be.
DE: So I thought that this was, for the second one, borderline whether it was normative change or not, mainly because dates have that otherwise secret grammar that each engine can make up for themselves, and presumably, this would be inside of that, just not the Temporal one.
PFC: Do you mean the secret grammar of Date.parse?
RGN: Yeah, but that is not the case. It is unequivocally a normative change.
PFC: But you could still support it in this secret grammar of Date.parse if you wanted.
DE: Right. Right. I mean, sure, it’s a normative change. Fine. But it doesn’t actually require that any engine make any change if their implementation. Just reduces the number of guarantees that a programmer has?
PFC: It does. This code snippet here would have to produce a different result, normatively.
JGT: And to provide some context, right, this is a very recent addition to the grammar. And it is almost certainly not used widely. Both because it’s an obscure Unicode character you’d have to get to, and second is that this format, this—for offset time zones, really is only around for compatibility with Java's ZonedDateTime, and because EMCAScript does not have Temporal yet, the usage of this particular format of offset timezones, as opposed to name timezones, is going to be very unusual. So even though this is, in theory, a normative change that will have observable effects, in practice, I find it incredibly hard to imagine that this would break anybody.
PFC: The only use I could think of is, like, copy/pasting a UTC offset from a Word document where Word replaced the dash with a minus sign automatically or something like that. I don’t even know if recent versions of Word still do that.
JGT: In the interest of time, because we have got a lot to cover, could we ask if there are any objections, and if not, can we get consensus for this.
CDA: We do have a couple folks in the queue. Daniel Minor support for normative changes.
SFC: The offset formatting example that you gave is relatively new. It’s very new. I remember we just recently proposed that. And, so, this would break existing behavior, but only this very, very new existing behavior, which is I think an important note that should be noted here. Yeah. Thank you, by for highlighting that is technically reachable without Temporal.
PFC: That’s thanks to RGN. I didn’t realize it myself either. Should we call for consensus on these two normative changes?
JGT: That’s actually a great transition to our next slide, so do we have—I think we have consensus for those two changes so we can move on? And can we --
CDA: Any objections?
CDA: Seeing nothing in the queue, please continue.
JGT: Okay, great. So before we get into actually what we want to—how we want to reduce the scope of Temporal, I think it’s helpful to set some context for everyone first, and this is really what we’ve learned over the last six months or so about the process of getting new EMCAScript features and actually shipping them, especially in browsers, right? So if you put yourself in the shoes of someone who— who is looking across a browser, right, ECMAScript is just one of many components that they have to deal with, and these browsers, they need to ship in resource constrained environments like the Apple Watch or a low-end Android device, they need to be really concerned about the size on disk, and especially if you’re in storage constrained device, they need to be concerned about download size, right, and growing that over time. They need to be worried about run time, RAM consumption of the code itself, and we’ve learned, you know, things like if you have a website with a lot of ads on it and each ad is in an iframe and each iframe has a copy of the ECMAScript built-ins, these things sort of add up a lot, even if they’re not necessarily related to the design of the language. And that if you imagine one of these folks, EMCAScript is one of, you know, there’s DOM, there’s CSS, there’s the video APIs, the sound APIs, there’s hundreds of components that these folks need to wrangle and think about, so even if we are adding, you know, a relatively small amount to the size, it’s not like they would think of it that way. But rather, the right way, if you’re building one of these browsers, is essentially to put everybody on a budget and to say that, okay, of you hundreds of components, nobody can double their size in a year without approval from the CEO or however that works, right? So there’s both a technical challenge and a sort of human process challenge here. The good news is that time heals this, right? So every year, every release of the Apple Watch will have more storage. Every year, new Android devices are coming online at the low end that have a lot more capability than the previous years and the old devices are being recycled. And so what that means is we can add things that we’re proposing to take out, but we just can’t add them all at once. So a good way to think about what we’re discussing today is that we’re deferring things, not permanently cutting them. And in reality, right, some of those things might not come back. There might not be enough community demand or champion interest in bringing them back, and that itself is a signal, and so—and finally, over the long term, engines will optimize, right? So we mentioned before that V8 has a problem with function count. Well, as, you know, Temporal or no Temporal, we’re increasing the number of functions in EMCAScript, right? And at some point, it will become feasible or sensible for V8 to optimize that problem somewhat, right? But we can’t hold all our proposals and, you know, that might be years from now, so we don’t think it’s a good idea to hold Temporal for every engine to redesign themselves. And finally, so my day job is I help to build enterprise software, and there’s a—in all large companies, they have a—what’s called a procurement office. And their job is to negotiate with suppliers and essentially extract discounts from those suppliers. And that the people that work in procurement, literally their performance reviews are based on how much money can they squeeze out of various suppliers. And you could think of the people deciding which components go into browsers as kind of like these procurement officers, right? Their job is to ensure that they can squeeze as much out of or at least limit the growth of all of the components that are in their purview, and you can think on the other side of that is someone like SYG, right, who’s, you know, we have champions and proposals. Well, SYG is the ECMAScript champion inside of Google trying to lobby the rest of Google to be able to say that ECMAScript should be able to grow. So what we have designed this process of taking stuff out as a way to give those EMCAScript champions like SYG the ammunition to go back to their browser procurement officers and say, you know what, we’ve done all that we can, we have really squeezed these guys, make sure they can reduce as much as possible.
JGT: So next slide. Great. And so the process we went through is we literally went through every single Temporal function and every argument and essentially tried to understand what would be the impact if we took them out, right? And in some cases, and we have, you know, issues are all in the repo. They’re all split out. We got lots of helpful suggestions, so, like Frank from Google was very helpful in suggesting some ideas that we hadn’t thought of. And we’ve had lots of community feedback, so the proposals that Philip is going to go through in a little bit really represent the last several months of a very focused effort from the entire champions team and many of you here and many others in the community to try to come to what could we take out that causes the least harm to the success of the proposal. So our goals in doing this, we wanted to address, as I mentioned, implementer concerns around size and complexity. Ideally, if we were going to hurt things, we wanted to hurt things that were uncommon, right? That were advanced use cases were less commonly used. What we really didn’t want to do was to make the API harder to learn for the vast majority of developers out there. One thing we also didn’t want to do is redesign anything. This proposal has been in the works for like seven years. We don’t want it to be another seven years. So, you know, our goal is either remove it or leave it in, not to crack it open again.
JGT: We also wanted to make sure that we didn’t make any future incompatible changes, right? So everything that we’re proposing today are things that if there’s community demand, we can put back. So that was very important to us. And in some cases, the work of going back and really closely examining every function, some of which we hadn’t really touched in three or four years, and we got some good ideas from the community as well, what better solutions could be. A good example is in the intervening time when designed the timezone API and now, there are now open standards that define declaratively what a timezone API could be, and that might remove completely the need, in a future proposal, to call out into user code. So next slide. So I’m going to hand it over Philip here. Hopefully this is 90 minutes and we’re going to try to keep you interested by going back and forth. And Philip is doing the hard work with going through all the changes.
PFC: You may be familiar with the old saying, tell ‘em what you’re gonna tell ‘em, then tell ‘em, then tell ‘em what you told ‘em. This is the part where I’m telling you what I’m gonna tell you. Here is an overview of the things that we want to remove. We’re going to talk about removing custom calendars and custom timezones. We’re going to talk about collapsing the implementations of the valueOf and toJSON methods into shared function objects. We’re going to talk about removing a bunch of functions that are mostly ergonomic APIs but have easy workarounds if they’re not present. And we’re going to talk about removing the relativeTo option from Duration addition. And so what this gains us is a net removal of 96 functions, which is just under one-third of the proposal. So there were about 300 total functions before. And there will be about 200 after. Here is the numbers of each—like, how many functions each topic removes. If you want to calculate it yourself.
PFC: So that’s a lot of things to discuss. We think that some of these, people are going to want to discuss, and some of them people are not going to feel so much like they have to discuss it. So we want to give everybody the opportunity to discuss where they feel it’s needed, but also move along quickly if nobody has any comments or questions. So the way we cooked up to do this is we’ll give each topic on each slide a letter. It will be in a big circle like this one. So if you want to discuss the A removal, then just put yourself on the queue with A. And then when we get to the end, we’ll see which letters are on the queue and those are the topics that we’ll talk about. And all the other stuff, we’ll just skip. We have at the end of this, a slide with some proposed time boxes for each topic, and depending on how many topics we need to discuss, we might adjust those time boxes.
PFC: Okay. The first one is calendars. The current state of the proposal is that there are built-in calendars. These are the ones that are supported natively by the implementation, and these are the same calendars that you can find in CLDR and ICU. The built-in calendars are represented as strings when you make API calls that take a calendar. There’s also a Temporal.Calendar object, which is primarily present so that you can make your own custom calendars by writing a class that extends that class. And then each type that carries a calendar has a getCalendar method where you can get the Temporal.Calendar object. So we would like to remove this. Not only the ability to define the custom calendar, but also the Temporal.Calendar class, and we will just use the built-in calendars, and those will be represented as string identifiers. So there are use cases for custom calendars. In the course of examining this removal, we also figured that it may be possible to add custom calendars back in a later proposal with less cost for implementations and less complexity. Frank had one nice idea that we’d like to explore in the future, but, you know, as Justin said, we are removing, not redesigning, so, you know, if we want—if there’s demand for custom calendars, it might make sense to add them back and then redesign them. But that is not something that we want to do right now. One big sort of deficiency of the custom calendars that we had is that they don’t integrate with the internationalization APIs. The Intl.DateTimeFormat only takes the built-in calendars and you’ll get an exception if you try to format a date with your custom calendar. We also, maybe if we consider adding this back in the future, like, to discuss how it would be possible to integrate custom calendars with date time format. Right, so what does this mean for users? We’d like to defer custom calendars until a later proposal. You can still implement a calendar that’s not in CLDR by making your own object that either composes with Temporal.PlainDate or extends Temporal.PlainDate. And the main thing that you’ll be missing is the ability to polyfill custom calendars as if they were built in. We now don’t recommend this, but if you needed to do it, you would have to monkey patch. On all of these slides, I’ve got links to the commits that show the exact normative change in the spec. There’s a link on each slide that you can click. For calendars and timezones, these removals are done in the same commit, so the links to these four commits, they apply to the next slide as well.
PFC: What we propose to do with calendars we’re also proposing to do with time zones. It was possible to create a custom time zone, and there is also a Temporal.TimeZone class that you could extend in order to do that. The—there were also a couple of advanced features on Temporal.TimeZone that you might have wanted to use, like looking up the next UTC offset transition. We’ll keep this functionality, but just move it to a different place in the API on ZonedDateTime. So just as with calendars, we actually think there’s a better design possible in a future proposal. Something that didn’t exist when we first designed this was jsCalendar and jCal. There’s actually a JSON format for declarative custom timezone API instead of the design that we have here, where you call into user code, and it didn’t exist when we came up with this API, but it would make sense if we were bringing this feature back to use something like that where we wouldn’t have this reentrancy problem, and the timezone would be completely defined in a declarative way rather than by implementing methods. So all of the methods that existed on Temporal.TimeZone, you can still get that functionality even without the timezone class, even if it’s less ergonomic in a couple of cases. Except getNextTransition and getPreviousTransition. So as I mentioned, those will move to ZonedDateTime and consolidate to one method. We chose to consolidate them in order to sort of squeeze out a savings of one extra function there. So you can see in this code sample, instead of getNextTransition, you call directly on the ZoneDateTime getTimeZoneTransition and pass a direction of ‘next’. This is actually slightly shorter than what the old API was. So maybe that’s a win. Another reason we choose remove these now, we thought we had to add them at the beginning we wouldn’t be able to add them later, because originally the time zone and calendar protocol called user methods even for built-in calendars and timezones. Last year, we made this change to optimize those user calls away for built-in calendars and time zones. And so now the original reasoning of we wouldn’t be able to add this in the future is no longer valid. That’s not the case anymore, that we wouldn’t be able to add it in the future. So that’s another reason why we are proposing these for removal.
PFC: We’d like to remove all of the getISOFields methods. We originally added getISOFields as a convenience for implementers of custom calendars because you may have a date in a certain calendar, and you need to get the underlying fields in the ISO 8601 calendar in order to get the calendar calculation. When we made that change last year of optimizing built-in calendars and timezones, you know, we discovered another use for this method. You can also use it to check if the calendar or time zone is built-in or custom. So without custom calendars and timezones, neither of those use cases make sense anymore, so there’s no reason to keep these methods. There are six types that have getISOFields methods, so that’s a savings of six methods. You can still get the ISO fields by changing your calendar using the withCalendar method and looking at the fields on there.
PFC: We would like to collapse all of Temporal’s valueOf methods into an identical function object. All of these methods, all of the valueOf methods on Temporal types, do exactly one thing, they throw a TypeError, they do not do a brand check because if you failed the brand check, you would throw a TypeError anyway. So literally, the only thing that the spec text for these methods says is throw a type error exception. You know, there are eight of these methods. They can all be the same function object, and this function object would even be reusable by future proposals that wanted to have a valueOf function that throws, such as Decimal for one. The only observable change here is what you can see in this code sample, where the valueOf method of one Temporal type is identity equal to the value of method of another Temporal type. Another case where we figured we could do this same collapsing is the toJSON method. Basically, all that the toJSON method does of a Temporal type is call toString without any arguments, but without observably looking it up. So what we propose is changing these to, again, an identical function object that’s identity equal on different types that does a brand check and then switches on the brand and, you know, does the appropriate toString call but without observably looking up toString. Another option we considered and discarded was to look up and call toString with no arguments. That would have made the body of this function simpler, but would also introduce an observable lookup.
I will pause here and relay some concerns from ABL about implementing these method collapses in Firefox. It sounds like it might be inconvenient though it is not impossible, because you need to do more manual work to connect everything up. I don’t know the exact nature but it sounds like intrinsic %ThrowTypeError% has some implementation problems because it is hard to get right. And there are a couple of questions he brought up: (1) should we recommend that future proposals use these as well? I mentioned that would be possible but do we want that to be recommended? (2) suppose we wanted to uncollapse these in the future so they would be distinct functions, it would be good to get consensus as part of this on whether that would be acceptable or not to do in the future. So that we know exactly what we are signing up for.
PFC: All right, on to the simpler removals. And we are proposing to remove the subtract methods. We have add methods and subtract methods. add adds a duration to another type and subtract subtracts a duration from another type. You can achieve the same result by adding the negation of the duration that you are going to subtract. Just like A - B is the same as A + -B. This is obviously less ergonomic. If you want to subtract you will look first for a method called subtract. But you know, for all the reasons that were said earlier, we suggest removing it for now and investigating if there is more demand for the ergonomics to bring it back in a future proposal.
PFC: Same thing we are going to do for since methods. Our types have until methods and since methods and there is a similar relationship between the two where one gives the negation of the other. until takes a difference between the two objects of same type and you write a.until(b). And since has the opposite sense, and so you can see in the code sample here, a.since(b) can be replaced with a.until(b).negated(). You could also consider replacing it by b.until(a)
and that will work for time units (which is the default) but not calendar units. We propose removing for the same reasons as subtract because it is a method that is a version of another method but goes in the opposite direction.
PFC: There are two methods named withPlainDate on PlainDateTime and ZonedDateTime. We propose removing those. It is a convenience to create a new object of the same type with all of the calendar units or all the time units replaced at once. We expect withPlainTime to be more commonly used than withPlainDate, withPlainDate is basically for symmetry. And also, if we were to remove withPlainTime, it would be less convenient to work around it with a property bag like you see here in the code sample. withPlainDate is relatively easy because you only have to replace year, month and day.
PFC: Another thing that we are proposing removing is two methods from PlainTime, toPlainDateTime and toZonedDateTime, which allow you to combine a plain date with a plain time and optionally a timezone if you are having a time date object. There are two ways to do this, you can use the PlainTime method and supply a date or can you use the PlainDate method and supply a time. So those are again two methods that do the same thing, but in opposite senses for convenience, and you don’t necessarily need to have this convenience. Instead of adding the date to the time, add the time to the date. So really the only drawback of removing these is that if you are looking in your autocomplete pop up in your IDE, you type .to and you will not see .toPlainDateTime or .toZonedDateTime and that is relatively small cost to removing two methods that do exactly the same thing.
PFC: We have exact times such as Instant and ZonedDateTime and we define those in terms of time elapsed since the Unix Epoch. And in order to examine those we have four properties, epochSeconds, epochMilliseconds, epochMicroseconds, and epochNanoseconds. Millisecond APIs are very important, because that is how you interoperate with legacy JS date and many APIs around the web, and in other libraries. Nanoseconds APIs are important because that is the resolution of the type and nanosecond is the granularity in which we count the elapsed time since the Epoch. Seconds and microseconds are not used as much, so we will propose to remove those because you can calculate them yourself. It is easy to round milliseconds to seconds; you divide by 1000 and do Math.floor. Nanoseconds to microseconds is a bit more difficult because they are bigints, which have no floor division, there is only truncating division, so you need to be careful when you are dealing with exact times before the 1970 epoch when the number of microseconds and nanoseconds are negative, as you can see in the code sample here. We think that this is going to be a bit of an annoying code snippet that people will have to copy around, but there are not that many APIs that use microseconds and so it seems like another candidate for adding back in a future proposal if there is demand for it.
PFC: Another thing that we are proposing to remove is the methods that let you convert directly from PlainDateTime and ZonedDateTime to PlainYearMonth and PlainMonthDay. You can still accomplish this by converting via an intermediate PlainDate. We think it is probably pretty uncommon that you want to go directly from a type with full date and time to a type of only month and year. We figured these will not be missed very much and if you miss these, there is an easy alternative to reach for.
PFC: There are four places where we had pairs of methods where one was suffixed with ISO and returns an object with the ISO 8601 calendar, and the other one has a required calendar parameter. We’ll propose removing the latter. Specifically, remove Instant.prototype.toZonedDateTime, Now.zonedDateTime, Now.plainDateTime, and Now.plainDate, while keeping Instant.prototype.toZonedDateTimeISO, Now.zonedDateTimeISO, Now.plainDateTimeISO, and Now.plainDateISO. We had these for a good reason which was to make sure that you did not get a didn’t get an ISO calendar date when you wanted a human calendar, but they have caused confusion. Like most other things we are removing, they have an easy replacement like you see in the code sample, instead of supplying the calendar directly you call the ISO method and call withCalendar, which is hardly longer.
PFC: All right then, finally, we are proposing removing the relativeTo option from Duration.add. This does not remove any functions but removes a lot of complexity that we think is not going to be often used and this particular functionality has had normative bugs in the past. So rather than keep spending the resources on wondering if we have bugs in something that is unlikely to be used a whole lot, we will just remove this option. There is an easy workaround with add and until. So instead of performing addition of two durations and passing the relativeTo option, you add the first duration to the relativeTo and add the second duration and take the difference with the original relativeTo.
JGT: So that was a lot. And I think we are probably going to have a lively discussion here. One thing that we did set up some time boxes to hope that we get through everything and if you add up all the numbers you realize it is a lot more than 48 minutes and we did it like a plane flight and that it is overbooked and some will be coming in little under. So can we get help from somebody who is not a temporal Champion to be our stop watch to help us enforce the time boxes?
CDA: That is our job.
JGT: Okay great. Thank you so much.
CDA: I presume we are starting on A right now?
JGT: In a moment but before we dig into these: we have had six months to go through the stages of grief about taking all of these things out and we are asking you guys, it is sort of a tall order, right? To go through it right now, and so, one thing that helped me, helped all of us in understanding this, this is not a permanent thing. We are simply trying to figure out how do we stagger the use of functionality so we can ship Temporal at all and that the most important thing here and this is something that has been driven home to us and Google, and it is just really hard to ship things in a big company and incredibly hard to ship big things in a browser and we are asking for your help to get the first version of Temporal out of the door to make things better later and what we would choose to do, when you add them altogether and it represents others that SYG and others that we can ship it and so with that, let’s go to the queue.
PFC: Looks like we have nobody signed up for C, H, I, J, and K. So we will talk about A, B, D, E, F on the left side and G and L on the right side, sounds good?
CDA: I think we have some general non-letter specific commentary as well. So let’s just jump right into it. Rob are you there?
Prepared statement from SYG:
V8 strongly supports this scope reduction and thanks the champions for being so open to late-stage simplification. V8 will take all the reduction it can get: the smaller the proposal, the higher the likelihood it can be implemented and shipped. That said, we don't feel super strongly about any particular method's reduction. For example, we see where developers are coming from for wanting
subtract(): it is a ubiquitous convenience, even though subtraction everywhere else also means addition of the negation.
NRO: Thanks for doing this work, I see different removals and even the great implication could say this is actually not very good and this is good and I would expect removing calendars which would be good and other matters not much and I have absolutely no idea of how much you can save by managing this method. So it would be great if the implementation could help us understand this.
DLM: It is hard to say offhand, like I have an item in the queue later on but I’m skeptical about squashing valueOf and toJSON together. All I can say is that the state of implementation in SpiderMonkey it is relevant to experiment with anything that is removal and anything that adding functionality will require more work and I think one could experiment in a way some reduction off each other and with that being said, these reductions where they are coming from us and it is valuable to get that feedback from SYG.
JGT: That is one of the things that we have learned from this process that a lot of these things are engine-specific. And so functions are expensive in V8 and that is not the case in SpiderMonkey so I would want to follow up for a more formal response by SYG. But you know in any order of few K per function and in terms of code size and there is some RAM cost as well, and so again, if it does not seem that much but you add it up 96 times and so SYG has like 200K per year. This would be like several years worth of their current budget expended on just Temporal which would obviously make them unhappy.
SFC: So I think SYG says everything that we need on this topic but it is both per context memory use and binary size that is important here as far as I understand it. And you know function removal is the biggest way to improve those metrics.
PFC: Move on to the specific topics and we have about 45 minutes and the time boxes are a bit optimistic so let’s take half of the time box for each one.
JHD: Yeah I mean, so this is about the calendar stuff. One of the points and I don’t know if you can put the slides back up while we are talking about this. One of the points you made was about—where is it? I believe this requires monkeypatch, and a lot of JavaScript developers and outside of TC39 has stopped doing, and you know disrupted a lot of proposals on our own and I think that is a huge downside. And separately, I think that the idea of having the Calendar class and the TimeZone class is that they serve as a non-string identity for a thing. For a place where like for a time zone and for a calendar where future data and properties and methods and accessors can be added as needed, but also it helps avoid making a program stringly typed. In other words, just using strings all over the place and making all of your branching based on that. And it feels really important to me to be able to have authoritative object identity or shared prototype for ideally with internal slots for a calendar and a time zone. These are important concepts that merit a primitive. I feel like these two classes constitutes a lot of methods, and you are trying to remove—you are trying to kind of count numbers on the score boards of how many functions you can take off to appease the feedback, I get that. But so it would be unsympathetic specifically for calendar that there is a possibility of a better design if it is deferred now. And I think that is the strongest argument in favor of it. Setting aside the implementation feedback. But it feels—yeah for maintainability and the code base for you know type systems for googling, for learnability, it feels much more usable in API where it is not stringly typed.
DE: So the thing is we already kind of are living in the stringly typed world even without the removals, both through Intl which supports the timezone and pass in strings, and Temporal because it does accept strings in calendar and timezone and it is optimized for this case. If you pass in a TimeZone or a Calendar object it will go through kind of complex, and suboptimal paths in the spec. So, I would want to hear about benefits besides conceptual or typing in this, given that we are already also incurring the cost of string base pads and reason for that is to both minimize the overhead of always allocating this extra object with an extra identity with each instance which adds up to a lot, as well as to make sure that code works reliably, although you are reliably calling the original methods and sort of matching developer expectation. Any thoughts on that?
JHD: I don’t think anyone would be surprised to learn that I prefer to avoid observable lookups and I value the simplicity of the string over the object in the case to the case of the actual spec case performed, I am not arguing in favor of the full protocol or favorability and passing it as strings is fine when the strings represent a canonical object. And when there is a transformation between them because then I can use the string and say this is a calendar string, now I want to make a real calendar and do stuff with it, but the string is the serialized format. And in this case, serialized not for the wire but passing it to functions and what not.
DE: So what should we do given that neither the current state of proposal with calendars included nor the proposed state with the calendars excluded will meet your goals there?
JGT: Could I maybe jump in there, and that I think JHD, you mentioned that we have a much better design that is possible for both of these features and if a future proposal so can we defer this discussion until that future proposal? Which we are pretty certain there is going to be community demand in time zones in particular and calendar. But for time today, could we have this discussion later?
JHD: I would say if enough of us, myself may or may not be included, are convinced that it will be possible to add a better designed thing in the future then yes, but then if we add the strings now and don’t get to adding a better designed thing that has an ergonomic adoption path from the initial version of the proposal, then I think that would be a very bad outcome. So I am not currently convinced and I am being careful of not saying I am not convinced but general convincing of that is what is important.
CDA: If we can, I would like to get to Shane’s comments, and then move on to the next topic.
SFC: Yeah, so, um you know just in case this does not occur other people and non-remaining calendars is remained supported with this change, and that is the most important thing from my perspective coming from Intl. I used some use cases for custom calendars in preparation for this slide. And basically all cases that I found like were better served by working with CLDR, and then you have the calculation, and you have formatting and you have the support S and other ecosystems. So therefore, I think that you know I look forward to possible future proposals about this but I think that for Temporal it is not a necessity.
JGT: One other thing, PFC has done some work here around actually preparing for this, recognizing that composition seems to actually be effective, certainly compared with monkey-patching which will now be so hard that honestly, nobody is likely to do it. Whereas composition of things seems to be a much more reasonable approach, so that is something that we have learned as part of this that we feel pretty optimistic that if you want custom calendar and time zone behavior, you can compose pieces of Temporal to help you make that work.
DE: So I am not sure we should restrict this to ten minutes. This is an important area. The topic that JHD raises is important and can we have the calendar and so when they were first added, we were always calling the methods on those things. And we thought that if we added it later, then that would be inconsistent so we had to add it in the first place—Philip are you queuing up to correct me?
PFC: Yeah I want to say maybe JHD could clarify but I don’t think his position is so much about custom calendar APIs, but about having a reified calendar and time zone object, even if it can’t be customized.
JHD: That is generally correct, I would rather ship a reified placeholder now then custom objects can be added in the future. I am concerned about delaying the reified form.
DE: Just to finish my point, when we made the change later to call the original intrinsics for when you are using built in calendars, that changed the calculus of whether we could add custom calendars later and we proved to ourselves that yes we could and that is what made this a strict removal than a change in semantics and I think custom calendar and custom API should be used for Intl and the level this was at the wrong level conceptually.
PFC: And I can go so far to say that, I’d need to verify this at a later point, but I think if we now had a reified calendar object that does not call custom methods, that would make it more difficult to add that in the future. So my recommendation would still be to support only the string if we are not going to support custom calendars.
DE: So any particular—the current spec uses whether you have instance of the object as its cue if it should call the methods or not and we use the object and did not call the methods, where would we go next and that is unclear.
JGT: That is a reasonable point is that the entire purpose of the calendar class is to create custom calendar and there is no functionality in 7 years on working on this proposal other than helping people build custom calendars and that is why the removal is tied together. For time zones there is one thing you cannot do, which is to find the next or previous time zone transition. We have been working on that and we found that it's more ergonomic to put it on ZonedDateTime instead of TimeZone, and I am pretty confident that the functionality that exists on calendars and timezones can—without ability to do custom, we have not found the use cases there and if we do find use cases we strongly agree with JHD we need a home for those use cases but for today, we have not found them yet. Can we move on?
CDA: We have about 25 minutes to get through a long queue. JHD?
JHD: I said my piece on timezone or calendar and I think the feedback would be the same so we can move on.
SBE: I come from time zone as a prospective consumer. Custom time zones are a pretty critical use case for supporting iCalendar which is the predominant standard for exchange of calendar data. And iCalendar requires that—there is no provision in iCalendar for referring to a common time zone database. The time zone is encoded within the format. So custom time zones are pretty critical there. I do think a database sort of declarative format would be an improvement, but my concern is that I think that the statement was made that removal, not redesign, was the foremost goal here. But I think that a removal of this scope is sort of a de facto redesign. And it is very important as a potential consumer of this API that we be clear that it’s possible, not only to add custom time zone support in a future proposal in the desired shape, but that it be possible in the meantime to kind of backfill that functionality into the current shape of the API without losing functionality or having to layer additional semantics from types from the API.
JGT: That is really a big challenge. And I think particularly in the case of time zones, I am now convinced that the current time zone design is a bad one. And that we can use—especially for the iCalendar case and because there are existing standards and with jsCalendar there is the same schema that moved into idiomatic JSON. We would love to start a proposal now, to go build custom time zones for Temporal based on the exact schema that you are depending on. So, if you are interested, I think we would love to work with you on building that proposal literally starting tomorrow. With that being said I think it would be a mistake to take a design that we don’t think is the best one, and keeping it in Temporal while it is too big; would be short-sighted and this is an API that we have to live with for decades.
SBE: I want to clarify I am not suggesting that we should ship existing implementation, I do think that taking time to do more appropriate declarative of that would be a net positive and my only concern is making sure that the remaining shape of the API is conducive to a proposal.
PFC: I want to say thank you to Sean here, we talked a couple of days ago, and he explained the iCalendar use case in more detail and if you are interested, I have a PR for the Temporal cookbook up on the proposal-temporal repo with a class that implements iCalendar time zones based on some test data that Sean gave to me. The class is composed with Temporal.ZonedDateTime and you can see how that looks with the remaining API in Temporal.
SFC: I want to say data driven APIs are the future. I think they are better and I would like to see more of these.
JHD: Just to clarify, a question, the shared valueOf method is going to be a distinct identity from the existing %ThrowTypeError% intrinsic?
PFC: It is currently distinct but if you have a good reason for why it should not be distinct, I am fine with that.
JHD: I don’t know if an implementation is capable of writing in a function that throws an error with a different message depending on what the receiver is. So it is fine to have an intrinsic, and I was curious on what was current, thank you.
PFC: I experimented with collapsing methods in V8 earlier this year, and that is possible. I am pretty sure SpiderMonkey is capable of that as well. I’m not certain about JavaScriptCore and the other engines.
SFC: So the shared prototype proposal is basically HasStringLikeSerialization. It is a prototype that could be used by other objects. That's one way to conceptualize it.
DE: I just wanted to clarify because Mark had an item on the queue previously, about replacing toJSON with toString. The reason why not was not obvious to me at first. It’s that toJSON is called with a particular argument and toString may interpret the argument for something and we need to make sure that toJSON does not misinterpret its argument. That is why it has to be a distinct function.
DLM: I am a bit skeptical with this one because unlike the other items, this is not a pure removal. It requires a change to implementation and I have not had a chance to review the comments that ABL raised but I do trust him and if it could be a source of problems and errors in the future, it seems it would be something good to avoid. So I guess my request is to defer on this one until we have tried all the other ones. If this is the last thing we need to do to make Temporal ship, then sure but otherwise I would like to see us hold off.
PFC: That sounds fair enough to me.
DE: Subtract is kind of nice ergonomically. Okay Shane you can rebut that comment.
SFC: For subtract besides this is compelled by the size reduction concerns, and then the usage concerns and so that motivates the proposal but’s itself, and additional thing I have had a lot of experience with developers being confused with negative duration and they think that they are always positive and if they see .add you—you can subtract a negative duration to go into the future and you can add a negative duration and go to the past, there are two ways to do the same thing and I think this is a foot gun for developers and I support this removal not only because you know it improves the metrics for V8 but it is clear to do the right thing.
JGT: Can I suggest that just looking at the temperature of the room that we have heard also on the repo concerns about subtract and we are hearing concerns about subtract here and in the interest of time if we should just withdraw removing subtract? And move on? (no) Okay.
NRO: So we can still do negative duration.
JGT: Yes.
ACE: I think removing subtract on my graph of value-add and how often I will use, it is in the quadrant I will use this way more than the others and removing it cuts down much less than cutting down other things and I see the point you make Shane about adding negative durations, and like NRO said, I don’t think this will reshuffle it because we still have to think about negative duration and I think people can learn adding negative numbers. And so would like to keep it.
CHU: I think it is the date level we use today and they are offering subtract methods and I would rather keep them instead of using negative durations and I have the feeling, I mean does it reduce the complexity of the—because internally aren’t they just using the date to the negative durations. So I don’t see so much benefit in removing them.
LCA: So for the complexity, it was not implementation complexity, but there is a fixed overhead for every function that it adds and this adds several more functions. But with the other comments I completely agree.
SFC: Yeah to clarify on that and these functions do the replacing code internal and it is the definition of bloat because it brings no additional value. There is ergonomics, for that I will argue it is negative for ergonomics and there are people that can disagree and that is fine but yeah.
MF: Moving on from subtract, to since and until. I think it is reasonable to expect that somebody can understand subtract would just be adding the negation, and for them to know those identities and just figure that out and then use that in the absence of subtract. I don’t think it's the same for since and until. I think this identity is hard to discover and apparently not an identity. Not as simple at least. And I don’t want people to have to be experts to know how to not—it seems—
JGT: You mean removing since.
MF: Yeah and to get the functionality back by using the until method.
JGT: So in the comments and champion group we heard one interesting feedback that several developers were confused by “since”, and from an English standpoint that is clear—the developer I am thinking of said “I never really understood what since meant” and it was harder for them to grasp. One of the potential benefits of removing since is that it has one way to do things rather than two ways to do things.
NRO: Same with what Michael said I was confused but that was short—there were some other people confused on Matrix, so can you identify why since and until are not reversible.
JGT: It has to do with the month. When you are moving on from one month to the next and say you start on the 31 of January and you want to measure the distance to February and February has only 28 days and there is a clamping that happens at the end. That is necessarily not reversible.
PFC: Yeah, that is exactly it. They are equivalent if you are not dealing with calendar units. If you are dealing with calendar units, you can get clamping.
JGT: The default options do not give you clamping. You have to opt in to the calendar unit to dig out the problem; you need to work at it.
PFC: Another way to look at it: the until and since methods use the receiver as the reference point for the calculation. So, if you switch the order from argument to receiver and receiver to argument, you are using a different reference point, whereas if you switch the method and then take the negation, you are using the same reference point. I don’t know if that makes it clear?
NRO: If this event happened say X time ago, am I looking at doing since or until?
JGT: It depends on how you are thinking about it and if you use until, you would say “that event until now.” Or if you use since, you would say “now since that event.” Does that make sense? And this discussion is an example of the exact confusion that we are in repo and because there are two ways to do things that are similar enough, and my current thinking is that it will be easier to teach developers there is one way to do this and you start at the receiver, and you go forward. And then it would be to continually explain the difference between the two similar ways to do something that are opposites.
SFC: Yeah also in the queue and I agree with everything that was said. One way to do things is better and the definition of API and so since is not equivalent to reversing the arguments and one way to difference is that it is there.
DE: Yeah I think this discussion makes clear that keeping both forms also does not help non-experts get this right. And the main defense that we have is I forgot what it is called, that balancing parameter by defaults is the same thing will not refuse to be ambiguous.
RGN: For anyone else who is interested there is a concrete example in Matrix #tc39-delegates:
Temporal.PlainDate.from("2024-06-30").until("2024-08-31", { largestUnit: "months" });
// => P2M1D
Temporal.PlainDate.from("2024-08-31").until("2024-06-30", { largestUnit: "months" });
// => -P2MLCA: This was the thing we got the most out all of these things here, and I looked through internal code that does not use Temporal but use some Rust date libraries, and yeah, we have not a single use of until and many-many uses of since, like tons of usage of since and not a single use of until, so I don’t know. It can’t be that confusing.
SFC: I definitely strongly support this and I don’t agree with the code that is on the slide. I can follow up later. About the code because that code is not calendar safe.
JGT: Good point.
SFC: But I agree with the change. Just not with the stated code.
DE: The question for LCA, does the since method you are using use this particular calendar month balancing operation?
LCA: I do not know.
DE: Okay, so it sounds like it doesn’t use since/until in a meaningful way because that is the difference that we are talking about. Maybe it is about argument order but that is the superficial part.
LCA: I don’t understand that argument either because if there is a difference between these two, that assumes that there is a reason for there to be a difference between these two existed at some point because there was a reason for it to exist.
DE: You can get the other behavior by switching the arguments and doing the negate. But the only time that you care about that is when it is in this particular month calendar case.
JHD: Yeah, it is just a quick comment on letter L and it seems fine to remove it. But the code example that you have shown is not complex. I think the reason was explained on Matrix that there is other complex that will make it complex and the transformation that you showed on the slide is because there is no other options, is that right?
PFC: The transformation is here on this slide. It is kind of a complicated answer and I will summarize it as quickly as I can. It looks like a simple replacement but what actually happened in the Duration.add method internally is that there were three different code paths that it could go through. One path for if you don’t pass relativeTo, one path if you pass a PlainDateTime relativeTo, and one path if you pass a ZonedDateTime relativeTo. So removing the relativeTo option does actually remove those extra code paths in favour of the until methods of those respective types. So this actually removes a place where control can go through three different paths and there is a certain amount of duplication.
JGT: Complexity is internal and it was very complex and we would be happy to see it in the rear view.
JHD: So my next item should be last to make sure nobody else wants to talk.
JGT: The queue is empty.
JHD: Okay, so I wanted to start to talk about overall process concerns with Temporal. And it has been in stage 3 for 4 years. And certainly stage 2.7 is brand new, and so that is why it has been stage 3 and not 2.7. The rate of change was slowing down and it even had its first meeting in 4 years this year without normative changes, and it was looking good, and now there is a lot of change. And we should still make it because—we should still consider these implementation concerns because they are still requesting it and regardless of statement the implementation does not—your slide has two things, code size and complexity, something like that. Yeah, so complexity and code size like neither of these things was hard to predict in the years prior to stage 3, and we did not need to implement to know it was going to be a lot of stuff. Certainly you needed to implement if it was going to be within the specific limits and these concerns I would hope in the future these kinds of concerns are brought up much earlier in the process when things are still being designed. And as you explained, the sort of corporate rationale for why the person concerned with the size may be different than the person in TC39 and I hope those things are coordinated in the future. And then the other thing is, we are all very interested in having Temporal to get to shipping and browsers as fast as possible. I really wanted that, and I have wanted that for many years, and with that said, we have tried in recent years to try to make the stage a proposal is in, actually match the reality of where it is. And it seems like it might be worth considering that Temporal belongs in 2.7 until these sort of changes are finished and so the move back to 3 can actually signal to the world that it is ready. Whereas like everyone week I feel calls in and I feel questions is lacking and I think it is simple for everyone to say once it is stage 3 again, it is ready.
JGT: We are up on the time limited and that would be a relatively long discussion item. So I am willing to have it but I would love to see if we can get consensus on which of these A-L has consensus today so we can actually start executing and then if there is time, to talk about stage. So we would like to go through and I don’t know the right way to do this process wise in a show of hand and asking for objection and we will go through A-L. And do we need to get explicit support for each one or should we?
CDA: It would be great to be acclamation for the entire batch of everything. If we cannot do that, then –
PFC: We are going to defer D until we get a chance to consider feedback.
DE: Are we going to defer E as well?
JGT: I think it would be more effective to go letter by letter and I don’t think we are going to get acclamation for everything. And I think—We start with A? Any objections to removing Temporal.Calendar and user calendar?
JGT: B? Temporal.TimeZone and user time zones? Okay.
JGT: For C, I assume there are no objections because C is only there for the purpose of A and B; any objections?
JGT: D we just agreed to remove or to defer until Mozilla has a chance to investigate.
JGT: Do we have objections to removal of subtract? I see a few hands here.
SFC: Can we write down who those are for the purpose of notes? These are all recommendations from the champions group. State the names so they will appear on the notes.
DE, LCA, and CHU, ACE object to removing subtract
JGT: Removing since? Any objections?
NRO: Can you remove since giving it is a single method and that could happen synchronously—never mind.
??: No reduction –
DE: It is not about a specific thing. I don’t know if there’s any further question to ask.
CDA: RBN prefers to leave since.
RBN: What is described to me about the asymmetrical nature of a.until(b) or b.until(a) vice versa and it is asymmetrical because the start and end and since has a value because people are likely to reach for the wrong thing and since will give them easier way to do the right thing.
CDA: Okay. Objection to since.
JGT: One thing we should consider in the future is remove until and leave since if that would address some of those concerns.
CDA: We may be able to revisit since() later so let’s continue.
JGT: Any objections for withPlainDate? For all of the remaining ones, these are less contentious, are there any objections to G, H, I, J, K or L?
JGT: Okay I think we did it.
PFC: All right I guess we are a couple of minutes over time, and that remains to ask the chairs if it is possible to discuss an overflow item to discuss JHD’s topic.
CDA: We will do our best, but I cannot promise.
JGT: Of A-L, D is deferred and E and F have not been removed and rest has consensus. ABC and G through L have all reached consensus and D is deferred and E and F did not get consensus. And thank you all for slogging through this, for the discussion, we are grateful for your time and your brainpower and feedback. It has been a long road and hopefully together we can ship this thing.
PFC: Thanks everyone. Let’s give it back to the chairs.
CDA: Thank you and we are past time and I understand that you are going to be ejected by security from the University very soon, so we will see everyone tomorrow.
- Consensus reached on fixing an arithmetic bug, tc39/proposal-temporal#2838.
- Consensus reached on removing Unicode minus signs from the ISO string grammar, in tc39/proposal-temporal#2856 and the corresponding ECMA-262 change tc39/ecma262#3334.
- Consensus on removing Temporal.Calendar, user calendars, Temporal.TimeZone, user time zones, getISOFields() methods, withPlainDate() methods, PlainTime’s toPlainDateTime and toZonedDateTime methods, epochSeconds properties, epochMicroseconds properties, Instant.fromEpochSeconds, Instant.fromEpochMicroseconds, PlainDateTime and ZonedDateTime’s toPlainYearMonth and toPlainMonthDay methods, Instant’s toZonedDateTime method, Now.zonedDateTime, Now.plainDateTime, Now.plainDate, and the relativeTo option from Duration.add.
- Before collapsing valueOf and toJSON into identical function objects, Mozilla would like to investigate how feasible it is on their end. We will bring this item back to the next plenary.
- Removal of subtract() and since() methods did not reach consensus.
- JHD considers it important to have reified Calendar and TimeZone objects so that we don’t have “stringly-typed” APIs and functionality can be added to them in the future. This can be considered for a future proposal, especially if new, non-reentrant designs can be taken into account.
- SBE considers user time zones an important use case and encourages work on a proposal to re-add them using the iCalendar data model.
- While there is no consensus to change the stage of Temporal, JHD considers that complexity and code size could have been evaluated much earlier.