Problems with importing/loading large tables

  • Resolved Shane.C

    (@shanec-1)


    11 years, 3 months ago

    Hi Tobias,

    I am having some issues with using TablePress on large datasets (10,000+ rows with each row being +/- 650 bytes).

    First of all the import function fails – I have to reduce the number of records significantly and then import them in small chunks.

    Secondly, when the data has been imported, the initial display of the table plus the search box – the display of which seems to indicate that the pre-processing is complete – can take several minutes.

    I guess some sort of indexing is taking place so as to allow for the near instant filtering. Is there any around these issues? If so I’d be very interested to hear how to do it. I would be particularly interested in knowing if it is possible to import the data in another way e.g. directly via a SQL procedure.

    Thanks,
    Shane

    https://wordpress.org/plugins/tablepress/

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Author Tobias Bäthge

    (@tobiasbg)

    11 years, 3 months ago

    Hi Shane,

    thanks for your post, and sorry for the trouble.

    Unfortunately, I have bad news here. TablePress is not really suited for huge tables like this. It simply was not designed to be a database replacement system.

    The server probably stops the import script when it reaches some sort of memory limit. That’s why the import of the large files fails.

    For the initial display: You are correct here. Basically, the entire data set (which is at least 6.5 megs, from your numbers) has to be sent to the browser, and then the DataTables JS library has to be invoked on that. That just takes time in the “client-side” approach that TablePress takes for this.

    As you are planning to use data from an SQL table anyway, I’d recommend to not use TablePress here, but to look into custom PHP/SQL solutions directly. With those, you could then also use the DataTables JS library (see http://datatables.net/ ) directly, and even use the server-side operation mode that it offers. With that, only the required parts of a table would be sent to the browser, which makes the displaying much faster.

    Regards,
    Tobias

    Thread Starter Shane.C

    (@shanec-1)

    11 years, 3 months ago

    Hi Tobias,

    Thanks for your honesty! My last question on this topic before I go in search of another solution is…

    Can the underlying table – that TablePress loads and does it’s client-side processing on – actually be a query i.e. the ‘table’ that is loaded by TablePress is a generated-on-the-fly query based on prior selections made by the user?

    If not, (and I know this a bit like going into a Ford dealership and asking them to recommend a BMW!) could you recommend a particular PHP/SQL plugin that can handle ‘big-data’ and facilitate querying it.

    Ok, and very very last question, what is the best tool for database management work – creating tables, queries, indices etc. – on WordPress.

    Right, that’s it.

    Thanks,
    Shane

    Plugin Author Tobias Bäthge

    (@tobiasbg)

    11 years, 3 months ago

    Hi Shane,

    querying tables or similar is not really possible in TablePress either. Is it’s not a database tool, it’s more aiming at static tables. One similar function to querying is the Row Filter Extension from http://tablepress.org/extensions/row-filter/

    I’m not aware of directly usable PHP/mySQL plugins, unfortunately. There are probably not that many, as they would have to cover very many different requirements. One commercial one seems to be http://wpdatatables.com/ but I have never used that. Another option might be to use plain PHP, e.g. with the help of the examples from the DataTables website.

    For creating tables, queries, etc. you should not use a WordPress plugin, but a database tool, like phpMyAdmin.

    Regards,
    Tobias

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘Problems with importing/loading large tables’ is closed to new replies.