Rendering 400k rows

Tags: #<Tag:0x00007f8b1dc43b50>
#1

I am trying to render 400k-500k rows with around 35 columns. These columns contain no formula.
I also want to export this data to csv.I am able to render the data and export it to csv but it is sluggish and sometime hangs web UI.

The requirement is to export all the data into csv. The data can be displayed in pages to make it responsive but should export all pages when clicked.I could not see any plugin for paging.

The data is retrieved from search , so it could be just 1000 rows or max 500k rows.

Do you have a snippet/jsfiddle for the above issue?

#2

That is true. There is not official paging plugin yet.
Export plugin exports data from getData() set. If you use pagination on front-end like this http://jsfiddle.net/AMBudnik/6bknc3oa/ you will get only one page of results. To work around it, you can load all the data for a split of second and page it again when the export plugin finishes its job.

If, you use hiddenRows to hide the rows to get the paging effect you can turn on the exportHiddenRows option for the export plugin https://handsontable.com/docs/8.0.0/demo-export-file.html

#3

Hi Aleksandra, thanks for your reply.

Regarding hidden rows: Is there a limit on number of rows we can hide? I mean if I hide around 300k rows but export it using export plugin with exportHiddenRows option. Do you think this can work?

Thanks

#4

loading more than 300k rows and keeping them hidden may degrade your project performance. However, the best thing is to try it. Here https://jsfiddle.net/AMBudnik/n5dtLf4c/ is s simple demo where you can manipulate with the dataset size (line 5)

data: Handsontable.helper.createSpreadsheetData(10000, 10),

and the amount of hiddenRows (line 1)

var myRows= Array.from(Array(1001).keys())

ps. there are no limits for hiding rows.

#5

Hi Aleksandra,
I am trying as you suggested to load the entire data set for just a second to export it.Basically to make it responsive I am just loading 10k rows instead of 50k but want to export 50k rows and change it back to 10k for display purposes.

I am able to load 10k rows initially instead of suppose 50k rows but when I want to export it, it still does not export the entire data when I am assigning full 50k rows before exporting.
here is the snippet
afterFilterSet = 50k rows
vm.recSearchData = shown on UI

                var hot = hotRegisterer.getInstance('Search');
                vm.recSearchData = afterFilterSet;
                hot.Data = afterFilterSet;
                
                hot.render();

                var exportPlugin = hot.getPlugin('exportFile');
                exportPlugin.downloadFile('csv', { columnHeaders: true, filename: vm.FileName });
               
                if (afterFilterSet.length > 10000) {
                    vm.recSearchData = afterFilterSet.slice(0, 10000);
                    hot.render();
                }

Let me know if I am missing something or there is another way to pass data to export functionality.
Thanks

#6

Would that https://jsfiddle.net/AMBudnik/pcrqtag8/ meet your requirements? I’ve just paused the plugin and brought back the hiddenRows settings after calling the plugin.

#7

The hidden rows really slows down the control/UI and it makes unusable.
It becomes really slow for 50k rows and I assume will not be user friendly for 300k rows.

#8

I guess that the most performance-friendly solution would be to overwrite this part of the code https://github.com/handsontable/handsontable/blob/0ec29f9cab9e0d9df1af99287c1211e87779737a/src/plugins/exportFile/dataProvider.js#L40 to allow using external dataset.

You can also check this PR https://github.com/handsontable/handsontable/pull/6837/files where we create a POC for pagination. I might be providing some better coding solutions, but 300k rows it quite a large dataset so an external source of data would work definitely faster.

#9