While writing Frontend code, You may have a scenario where you have to fetch all records from an API using pagination. You can break the code into multiple functions to do so. However, Maintaining so many variables and passing states from one function to another function, is not simple as it looks. The async nature of the data source makes it more complex. However, You can use the async-generator functions to simplify it. In this article, I will explain how you can break this complex logic into simplified functions.

1. Create a mocked data lake/source

To test out the functionality you required a data source. To mimic the real-api data source, I will create a list of dummy users.

const users = Array(1000)
  .fill()
  .map((_, i) => ({ name: `user${i}`, id: `id_${i}` }));
const TOTAL_RECORDS = users.length;

console.log({user: users[0], TOTAL_RECORDS});
// { user: { name: 'user0', id: 'id_0' }, TOTAL_RECORDS: 1000 }

2. Fetch users API function

You can use a promise-based API function to fetch records. However, I will use async-await function to create this Fetch user API function

//service.js

const delay = () => new Promise((r) => setTimeout(r, 1000));

/**
 * fetchUsers
 * 
 * @param {page, limit} current page number, limit of the records to fecth
 * @default {0, 100}
 * @returns 
 */
const fetchUsers = async ({ page = 0, limit = 100 }) => {
  const start = page * limit; // start index of the records
  const end = (page + 1) * limit; // end index of the records
  await delay(); // virtual delay of 1000ms
  return {
    data: users.slice(start, end), //Slice the records from start to end
    done: end >= TOTAL_RECORDS,
    start,
    end,
  };
};

// [optional]
//export { fetchUsers };

The above function fetchUsers takes options like page and limit. Page is defined as the current page of the pagination and limit is defined as the limit of records to be fetched. Delay is just and virtual delay of 1000ms to mimic the actual network.

3. Fetch all records using Generator Function

Traditionally, You can write a function using recursion to solve the problem. However, the Writing recursion version is too complex to understand. You can also use for-loop/while-loop using async-await. Check out the below example.

async function main() {
  let records = [];
  let options = { page: 0, limit: 100, end: 100, total_records: Infinity };

  while (options.end < options.total_records) {
    const response = await fetchUsers(options);
    records = records.concat(response.data);
    options.end = response.end;
    options.page = options.page + 1;
    options.total_records = response.total_records;
    console.log(records[records.length - 1], response.start, response.end);
  }
}
// { name: 'user199', id: 'id_199' } 100 200
// { name: 'user299', id: 'id_299' } 200 300

Output:

As you can see using a while-loop function, Your main function has to worry too much about other variables like records, paging options. In such scenarios, Generators shines well. Let's try to convert the above example into generators.

/**
 *
 * @param {page, limit} start page index, limit/chunk of the records to fecth on each call
 * @default {0, 100}
 * @returns
 */
async function* fetchAllRecords({ page = 0, limit = 100 } = {}) {
  while (true) {
    const records = await fetchUsers({ page: page++, limit });
    yield records;
    if (records.done) return;
  }
}

async function main() {
  const api = fetchAllRecords();
  let records = [];
  for await (let record of api) {
    records = records.concat(record.data);
        console.log(records[record.end - 1], record.start, record.end);
  }
}

Output:

As you can see, Using generators makes code look much simpler and easy to read.

Note: If you noticed line for await (let record of api), Here in this line we are looking at the generator looping asynchronously. This await signals for-loop that api object is async iterator function to iterate.

Codesandbox

Conclusion:

Async await does not make your API run faster. It just simplifies code and increases the readability of code. As someone wise once said.

“Any fool can write code that a computer can understand. Good programmers write code that humans can understand.” — Martin Fowler