Iterables and iterators in ECMAScript 6
This blog post is outdated. Please read chapter “Iterables and iterators” in “Exploring ES6”.
This blog post is part of a series on iteration in ES6:
ECMAScript 6 introduces a new interface for iteration, Iterable. This blog post explains how it works, which language constructs consume data via it (e.g., the new for-of loop) and which sources provide data via it (e.g., arrays).
Iterability
The idea of iterability is as follows.
Data consumers: JavaScript has language constructs that consume data. For example,
for-ofloops over values and the spread operator (...) inserts values into arrays or function calls.Data sources: The data consumers could get their values from a variety of sources. For example, you may want to iterate over the elements of an array, the key-value entries in a map or the characters of a string.
It’s not practical for every consumer to support all sources, especially because it should be possible to create new sources and consumers, e.g. via libraries with data structures or with new ways of processing data. Therefore, ES6 introduces the interface Iterable. Data consumers use it, data sources implement it:
Given that JavaScript does not have interfaces, Iterable is more of a convention:
Source: A value is considered iterable if it has a method whose key is the symbol [2]
Symbol.iteratorthat returns a so-called iterator. The iterator is an object that returns values via its methodnext(). We say: it enumerates items, one per method call.Consumption: Data consumers use the iterator to retrieve the values they are consuming.
Let’s see what consumption looks like for an array arr. First, you create an iterator via the method whose key is Symbol.iterator:
> let arr = ['a', 'b', 'c'];
> let iter = arr[Symbol.iterator]();
Then you call the iterator’s method next() repeatedly to retrieve the items “inside” the array:
> iter.next()
{ value: 'a', done: false }
> iter.next()
{ value: 'b', done: false }
> iter.next()
{ value: 'c', done: false }
> iter.next()
{ value: undefined, done: true }
As you can see, next() returns each item wrapped in an object, as the value of the property value. The boolean property done indicates when the end of the sequence of items has been reached.
Iterable and iterators are part of a so-called protocol (methods plus rules for using them) for iteration. A key characteristic of this protocol is that it is sequential: the iterator returns values one at a time. That means that if an iterable data structure is non-linear (such as a tree), iteration will linearize it.
Iterable data sources
I’ll use the for-of loop (which is explained in more detail later) to iterate over various kinds of iterable data.
Arrays
Arrays (and typed arrays) are iterables over their elements:
for (let x of ['a', 'b']) {
console.log(x);
}
// Output:
// 'a'
// 'b'
Strings
Strings are iterable, but they enumerate Unicode code points, each of which may comprise one or two JavaScript “characters”:
for (let x of 'a\uD83D\uDC0A') {
console.log(x);
}
// Output:
// 'a'
// '\uD83D\uDC0A' (crocodile emoji)
Note that you have just seen that primitive values can be iterable, too. A value doesn’t have to be an object in order to be iterable.
Maps
Maps [3] are iterables over their entries. Each entry is encoded as a [key, value] pair, an array with two elements. The entries are always enumerated deterministically, in the same order in which they were added to the map.
let map = new Map().set('a', 1).set('b', 2);
for (let pair of map) {
console.log(pair);
}
// Output:
// ['a', 1]
// ['b', 2]
Note that WeakMaps [3] are not iterable.
Sets
Sets [3] are iterables over their elements (which are enumerated in the same order in which they were added to the set).
let set = new Set().add('a').add('b');
for (let x of set) {
console.log(x);
}
// Output:
// 'a'
// 'b'
Note that WeakSets [3] are not iterable.
arguments
Even though the special variable arguments is more or less obsolete in ECMAScript 6 (due to rest parameters), it is iterable:
function printArgs() {
for (let x of arguments) {
console.log(x);
}
}
printArgs('a', 'b');
// Output:
// 'a'
// 'b'
DOM data structures
Most DOM data structures will eventually be iterable:
for (let node of document.querySelectorAll('···')) {
···
}
Note that implementing this functionality is work in progress. But it is relatively easy to do so, because the symbol Symbol.iterator can’t clash with existing property keys [2].
Iterable computed data
Not all iterable content does have to come from data structures, it could also be computed on the fly. For example, all major ES6 data structures (arrays, typed arrays, maps, sets) have three methods that return iterable objects:
entries()returns an iterable over entries encoded as[key,value]arrays. For arrays, the values are the array elements and the keys are their indices. For sets, each key and value are the same – the set element.keys()returns an iterable over the keys of the entries.values()returns an iterable over the values of the entries.
Let’s see what that looks like. entries() gives you a nice way to get both array elements and their indices:
let arr = ['a', 'b', 'c'];
for (let pair of arr.entries()) {
console.log(pair);
}
// Output:
// [0, 'a']
// [1, 'b']
// [2, 'c']
Plain objects are not iterable
Plain objects (as created by object literals) are not iterable:
for (let x of {}) { // TypeError
console.log(x);
}
The reasoning is as follows. The following two activities are different:
- Examining the structure of a program (reflection)
- Iterating over data
It is best to keep these two activities separate. #1 is relevant for all objects, #2 only for data structures. You could make most objects iterable by adding a method [Symbol.iterator]() to Object.prototype, but they would lose this ability in two cases:
- If they are created via
Object.create(null). ThenObject.prototypeis not in their prototype chain. - If they are data structures. Then they need iterability for their data. Not only would you not be able to iterate over the properties of, say, arrays (which are also data structures). But you couldn’t ever later add iterability to an existing class, because that would break code that iterates over the properties of their instances.
Therefore, the safest way to make properties iterable is via a tool function. For example, via objectEntries(), whose implementation is shown later (future ECMAScript versions may have something similar built in):
let obj = { first: 'Jane', last: 'Doe' };
for (let [key,value] of objectEntries(obj)) {
console.log(`${key}: ${value}`);
}
// Output:
// first: Jane
// last: Doe
It is also important to remember that iterating over the properties of an object is mainly interesting if you use objects as maps [4]. But we only do that in ES5 because we have no better alternative. In ECMAScript 6, we have Map.
Iterating language constructs
This section lists all built-in ES6 programming constructs that make use of the iteration protocol.
Destructuring via an array pattern
Destructuring [5] via array patterns works for any iterable:
let set = new Set().add('a').add('b').add('c');
let [x,y] = set;
// x='a'; y='b'
let [first, ...rest] = set;
// first='a'; rest=['b','c'];
The for-of loop
for-of is a new loop in ECMAScript 6. One form of it looks like this:
for (let x of iterable) {
···
}
This loop iterates over iterable, assigns each of the enumerated items to the iteration variable x and lets you process it in the body. The scope of x is the loop, it doesn’t exist outside it.
Note that the iterability of iterable is required, otherwise for-of can’t loop over a value. That means that non-iterable values must be converted to something iterable. For example, via Array.from(), which turns array-like values and iterables into arrays:
let arrayLike = { length: 2, 0: 'a', 1: 'b' };
for (let x of arrayLike) { // TypeError
console.log(x);
}
for (let x of Array.from(arrayLike)) { // OK
console.log(x);
}
I expect for-of to mostly replace Array.prototype.forEach(), because it is more versatile (forEach() only works for array-like values) and will be faster long term (see FAQ at the end).
Iteration variables: let declarations vs. var declarations
If you let-declare the iteration variable, a fresh binding (slot) will be created for each iteration. That can be seen in the following code snippet where we save the current binding of elem for later, via an arrow function. Afterwards, you can see that the arrow functions don’t share the same binding for elem, they each have a different one.
let arr = [];
for (let elem of [0, 1, 2]) {
arr.push(() => elem); // save `elem` for later
}
console.log(arr.map(f => f())); // [0, 1, 2]
// `elem` only exists inside the loop:
console.log(elem); // ReferenceError: elem is not defined
It is instructive to see how things are different if you var-declare the iteration variable. Now all arrow functions refer to the same binding of elem.
let arr = [];
for (var elem of [0, 1, 2]) {
arr.push(() => elem);
}
console.log(arr.map(f => f())); // [2, 2, 2]
// `elem` exists in the surrounding function:
console.log(elem); // 2
Having one binding per iteration is very helpful whenever you create functions via a loop (e.g. to add event listeners).
let-declared iteration variables in for loops and for-in loops
Two more loops get one binding per iteration if you let-declare their iteration variables: for and for-in.
Let’s look at for with a let-declared iteration variable i:
let arr = [];
for (let i=0; i<3; i++) {
arr.push(() => i);
}
console.log(arr.map(f => f())); // [0, 1, 2]
console.log(i); // ReferenceError: i is not defined
If you var-declare i, you get the traditional behavior.
let arr = [];
for (var i=0; i<3; i++) {
arr.push(() => i);
}
console.log(arr.map(f => f())); // [3, 3, 3]
console.log(i); // 3
Similarly, for-in with a let-declared iteration variable key leads to one binding per iteration:
let arr = [];
for (let key in ['a', 'b', 'c']) {
arr.push(() => key);
}
console.log(arr.map(f => f())); // ['0', '1', '2']
console.log(key); // ReferenceError: key is not defined
var-declaring key produces a single binding:
let arr = [];
for (var key in ['a', 'b', 'c']) {
arr.push(() => key);
}
console.log(arr.map(f => f())); // ['2', '2', '2']
console.log(key); // '2'
Iterating with existing variables, object properties and array elements
So far, we have only seen for-of with a declared iteration variable. But there are several other forms.
You can iterate with an existing variable:
let x;
for (x of ['a', 'b']) {
console.log(x);
}
You can also iterate with an object property:
let obj = {};
for (obj.prop of ['a', 'b']) {
console.log(obj.prop);
}
And you can iterate with an array element:
let arr = [];
for (arr[0] of ['a', 'b']) {
console.log(arr[0]);
}
Iterating with a destructuring pattern
Combining for-of with destructuring is especially useful for iterables over key-value pairs (encoded as arrays). That’s what maps are:
let map = new Map().set(false, 'no').set(true, 'yes');
for (let [k,v] of map) {
console.log(`key = ${k}, value = ${v}`);
}
// Output:
// key = false, value = no
// key = true, value = yes
Array.prototype.entries() also returns an iterable over key-value pairs:
let arr = ['a', 'b', 'c'];
for (let [k,v] of arr.entries()) {
console.log(`key = ${k}, value = ${v}`);
}
// Output:
// key = 0, value = a
// key = 1, value = b
// key = 2, value = c
Therefore, entries() gives you a way to treat enumerated items differently, depending on their position:
/** Same as arr.join(', ') */
function toString(arr) {
let result = '';
for (let [i,elem] of arr.entries()) {
if (i > 0) {
result += ', ';
}
result += String(elem);
}
return result;
}
This function is used as follows:
> toString(['eeny', 'meeny', 'miny', 'moe'])
'eeny, meeny, miny, moe'
Array.from()
Array.from() [6] converts iterable and array-like values to arrays. It is also available for typed arrays.
> Array.from(new Map().set(false, 'no').set(true, 'yes'))
[[false,'no'], [true,'yes']]
> Array.from({ length: 2, 0: 'hello', 1: 'world' })
['hello', 'world']
Array.from() works as expected for a subclass of Array (which inherits this class method) – it converts iterables to instances of the subclass.
Spread
The spread operator [5] inserts the values of an iterable into an array:
> let arr = ['b', 'c'];
> ['a', ...arr, 'd']
['a', 'b', 'c', 'd']
That means that it provides you with a compact way to convert any iterable to an array:
let arr = [...iterable];
The spread operator also turns an iterable into the arguments of a function, method or constructor call:
> Math.max(...[-1, 8, 3])
8
Maps and sets
The constructor of a map turns an iterable over [key,value] pairs into a map:
> let map = new Map([['uno', 'one'], ['dos', 'two']]);
> map.get('uno')
'one'
> map.get('dos')
'two'
The constructor of a set turns an iterable over elements into a set:
> let set = new Set(['red', 'green', 'blue']);
> set.has('red')
true
> set.has('yellow')
false
The constructors of WeakMap and WeakSet work similarly. Furthermore, maps and sets are iterable themselves (WeakMaps and WeakSets aren’t), which means that you can use their constructors to clone them.
Promises
Promise.all() and Promise.race() accept iterables over promises [7]:
Promise.all(iterableOverPromises).then(···);
Promise.race(iterableOverPromises).then(···);
yield*
yield* [8] yields all items enumerated by an iterable.
function* yieldAllValuesOf(iterable) {
yield* iterable;
}
The most important use case for yield* is to recursively call a generator [8] (which produces something iterable).
Implementing iterables
The iteration protocol looks as follows.
An object becomes iterable (“implements” the interface Iterable) if it has a method (own or inherited) whose key is Symbol.iterator. That method must return an iterator, an object that enumerates the items “inside” the iterable via its method next().
In TypeScript notation, the interfaces for iterables and iterators look as follows (based on [9]):
interface Iterable {
[System.iterator]() : Iterator;
}
interface IteratorResult {
value: any;
done: boolean;
}
interface Iterator {
next() : IteratorResult;
return?(value? : any) : IteratorResult;
}
return is an optional methods that we’ll get to later (so is throw(), but it is practically never used for iterators and therefore explained in a follow-up blog post on generators). Let’s first implement a dummy iterable to get a feeling for how iteration works.
let iterable = {
[Symbol.iterator]() {
let step = 0;
let iterator = {
next() {
if (step <= 2) {
step++;
}
switch (step) {
case 1:
return { value: 'hello', done: false };
case 2:
return { value: 'world', done: false };
default:
return { value: undefined, done: true };
}
}
};
return iterator;
}
};
Let’s check that iterable is, in fact, iterable:
for (let x of iterable) {
console.log(x);
}
// Output:
// hello
// world
The code executes three steps, with the counter step ensuring that everything happens in the right order. First we, return the value 'hello', then the value 'world' and then we indicate that the end of the enumerated items has been reached. Each item is wrapped in an object with the properties:
valuewhich holds the actual item anddonewhich is a boolean flag that indicates whether the end has been reached, yet.
You can omit done if it is false and value if it is undefined. That is, the switch statement could be written as follows.
switch (step) {
case 1:
return { value: 'hello' };
case 2:
return { value: 'world' };
default:
return { done: true };
}
As is explained in the follow-up blog post on generators, there are cases where you want even the last item with done: true to have a value. Otherwise, next() could be simpler and return items directly (without wrapping them in objects). The end of iteration would then be indicated via a special value (e.g., a symbol).
Let’s look at one more implementation of an iterable. The function iterateOver() returns an iterable over the arguments that are passed to it:
function iterateOver(...args) {
let index = 0;
let iterable = {
[Symbol.iterator]() {
let iterator = {
next() {
if (index < args.length) {
return { value: args[index++] };
} else {
return { done: true };
}
}
};
return iterator;
}
}
return iterable;
}
// Using `iterateOver()`:
for (let x of iterateOver('fee', 'fi', 'fo', 'fum')) {
console.log(x);
}
// Output:
// fee
// fi
// fo
// fum
Iterators that are iterable
The previous function can be simplified if the iterable and the iterator are the same object:
function iterateOver(...args) {
let index = 0;
let iterable = {
[Symbol.iterator]() {
return this;
},
next() {
if (index < args.length) {
return { value: args[index++] };
} else {
return { done: true };
}
},
};
return iterable;
}
Even if the original iterable and the iterator are not the same object, it is still occasionally useful if an iterator has the following method (which also makes it an iterable):
[Symbol.iterator]() {
return this;
}
All built-in ES6 iterators follow this pattern (via a common prototype, see follow-up blog post on generators). For example, the default iterator for arrays:
> let arr = [];
> let iterator = arr[Symbol.iterator]();
> iterator[Symbol.iterator]() === iterator
true
Why is it useful if an iterator is also an iterable? for-of only works for iterables, not for iterators. Because array iterators are iterable, you can continue an iteration in another loop:
let arr = ['a', 'b'];
let iterator = arr[Symbol.iterator]();
for (let x of iterator) {
console.log(x); // a
break;
}
// Continue with same iterator:
for (let x of iterator) {
console.log(x); // b
}
An alternative is to use a method that returns an iterable. For example, the result of Array.prototype.values() iterates the same way as the default iteration. Therefore, the previous code snippet is equivalent to:
let arr = ['a', 'b'];
let iterable = arr.values();
for (let x of iterable) {
console.log(x); // a
break;
}
for (let x of iterable) {
console.log(x); // b
}
But with an iterable, you can’t be sure that it won’t restart iteration if for-of calls the method [Symbol.iterator](). For example, instances of Array are iterables that start at the beginning whenever you call this method.
One use case for continuing an iteration is that you can remove initial items (e.g. a header) before processing the actual content via for-of.
Optional iterator methods: return() and throw()
Two iterator methods are optional:
return()gives an iterator the opportunity to clean up if an iteration ends prematurely.throw()is about forwarding a method call to a generator that is iterated over viayield*. It is explained in the follow-up blog post on generators.
Closing iterators via return()
As mentioned before, the optional iterator method return() is about letting an iterator clean up if it wasn’t iterated over until the end. It closes an iterator. In for-of loops, premature (or abrupt, in spec language) termination can be caused by:
breakcontinue(if you continue an outer loop,continueacts like abreak)throwreturn
In each of these cases, for-of lets the iterator know that the loop won’t finish. Let’s look at an example, a function readLinesSync that returns an iterable of text lines in a file and would like to close that file no matter what happens:
function readLinesSync(fileName) {
let file = ···;
return {
···
next() {
if (file.isAtEndOfFile()) {
file.close();
return { done: true };
}
···
},
return() {
file.close();
return { done: true };
},
};
}
Due to return(), the file will be properly closed in the following loop:
// Only print first line
for (let line of readLinesSync(fileName)) {
console.log(x);
break;
}
The return() method must return an object. That is due to how generators handle the return statement and will be explained in the follow-up blog post on generators.
The following constructs close iterators that aren’t completely “drained”:
for-ofyield*- Destructuring
Array.from()Map(),Set(),WeakMap(),WeakSet()Promise.all(),Promise.race()
More examples of iterables
In this section, we look at a few more examples of iterables. Most of these iterables are easier to implement via generators. The follow-up blog post on generators shows how.
Tool functions that return iterables
Tool functions and methods that return iterables are just as important as iterable data structures. The following is a tool function for iterating over the own properties of an object.
function objectEntries(obj) {
let index = 0;
// In ES6, you can use strings or symbols as property keys,
// Reflect.ownKeys() retrieves both
let propKeys = Reflect.ownKeys(obj);
return {
[Symbol.iterator]() {
return this;
},
next() {
if (index < propKeys.length) {
let key = propKeys[index];
index++;
return { value: [key, obj[key]] };
} else {
return { done: true };
}
}
};
}
let obj = { first: 'Jane', last: 'Doe' };
for (let [key,value] of objectEntries(obj)) {
console.log(`${key}: ${value}`);
}
// Output:
// first: Jane
// last: Doe
Combinators for iterables
Combinators [10] are functions that combine existing iterables to create new ones.
Let’s start with the combinator function take(n, iterable), which returns an iterable over the first n items of iterable.
function take(n, iterable) {
let iter = iterable[Symbol.iterator]();
return {
[Symbol.iterator]() {
return this;
},
next() {
if (n > 0) {
n--;
return iter.next();
} else {
return { done: true };
}
}
};
}
let arr = ['a', 'b', 'c', 'd'];
for (let x of take(2, arr)) {
console.log(x);
}
// Output:
// a
// b
zip turns n iterables into an iterable of n-tuples (encoded as arrays of length n).
function zip(...iterables) {
let iterators = iterables.map(i => i[Symbol.iterator]());
let done = false;
return {
[Symbol.iterator]() {
return this;
},
next() {
if (!done) {
let items = iterators.map(i => i.next());
done = items.some(item => item.done);
if (!done) {
return { value: items.map(i => i.value) };
}
// Done for the first time: close all iterators
for (let iterator of iterators) {
iterator.return();
}
}
// We are done
return { done: true };
}
}
}
As you can see, the shortest iterable determines the length of the result:
let zipped = zip(['a', 'b', 'c'], ['d', 'e', 'f', 'g']);
for (let x of zipped) {
console.log(x);
}
// Output:
// ['a', 'd']
// ['b', 'e']
// ['c', 'f']
Infinite iterables
Some iterable may never be done.
function naturalNumbers() {
let n = 0;
return {
[Symbol.iterator]() {
return this;
},
next() {
return { value: n++ };
}
}
}
With an infinite iterable, you must not iterate over “all” of it. For example, by breaking from a for-of loop:
for (let x of naturalNumbers()) {
if (x > 2) break;
console.log(x);
}
Or by only accessing the beginning of an infinite iterable:
let [a, b, c] = naturalNumbers();
// a=0; b=1; c=2;
Or by using a combinator. take() is one possibility:
for (let x of take(3, naturalNumbers())) {
console.log(x);
}
// Output:
// 0
// 1
// 2
The “length” of the iterable returned by zip() is determined by its shortest input iterable. That means that zip() and naturalNumbers() provide you with the means to number iterables of arbitrary (finite) length:
let zipped = zip(['a', 'b', 'c'], naturalNumbers());
for (let x of zipped) {
console.log(x);
}
// Output:
// ['a', 0]
// ['b', 1]
// ['c', 2]
Frequently asked question
Isn’t the iteration protocol slow?
You may be worried about the iteration protocol being slow, because a new object is created for each invocation of next(). However, memory management for small objects is fast in modern engines and in the long run, engines can optimize iteration so that no intermediate objects need to be allocated. A thread on es-discuss has more information.
Conclusion
In this blog post we have seen that even with just the foundations of ES6 iteration, you can already do a lot. Generators [8] build on that foundation and help with, among other things, implementing iterators.
The JavaScript runtime library is still missing tools for working with iterators. Python has the feature-rich module itertools, JavaScript will eventually get a similar module.
Further reading
- “Exploring ES6: Upgrade to the next version of JavaScript”, book by Axel
- Symbols in ECMAScript 6
- ECMAScript 6: maps and sets
- “Pitfalls: Using an Object as a Map” in “Speaking JavaScript”
- Destructuring and parameter handling in ECMAScript 6 [includes an explanation of the spread operator (
...)] - ECMAScript 6’s new array methods
- ECMAScript 6 promises (2/2): the API
- ES6 generators in depth
- “Closing iterators”, slides by David Herman
- “Combinator” in HaskellWiki
Comments
Post a Comment