In some circumstances, generating the data feed can be too costly for your server. We are going to learn how feed pagination works in Doofinder so you can implement a custom paginated feed.

Feed Pagination

Doofinder’s feed downloader can use paginated feeds, so it doesn’t try to download the whole feed in one single request, but downloads smaller pieces of it in multiple requests, so the feed server, instead of generating one big data feed, generates smaller feeds on every request, which is less expensive.

In order to activate that, you have to let us know so we can enable that feature for your feed, but also your server has to comply with some requirements.

Doofinder expects to find two parameters, in order to retrieve a paginated feed:

  • limit: This the size of the page. Defaults to 1000 items.
  • offset: This is where to start reading the page. It starts with 0 and incremets in <limit>; items on each request.

Basically, doofinder starts gathering pieces of the feed using the limit and offset parameters until it gets an empty response.

With the default limit, Doofinder would start requesting offset=0, then offset=1000, then offset=2000 and so on, until an empty response is received. Of course, along with the offset parameter goes the limit parameter which specifies the size of the page (1000 in this case).

Learn by example

– Let’s say we have a feed at the url, which has a size of 6300 products. – Let’s say we set a pagination of 2000 items.

Then the first request our downloader would do would be:

This would fetch the first 2000 items:

  • when the feed is a XML, this would also fetch the declarations at the beginning of the XML file.
  • when the feed is TXT, this would also fetch the header of the file.

The next request would be:

This would fetch the items between 2000-3999:

  • special care should be taken when the feed is a XML feed so the beginning of the XML is not sent again with the data.

The request with an offset of 6000 would fetch the last 300 items. When the feed is a XML feed, this would also fetch the XML closing tags.

The last request would get an empty response, telling the Doofinder downloader that must finish the download process.