Added shallow search for data.table in tables()

[manmita]

Closes #2606

added arg depth = 1L to tables() one for shallow search
if depth is 0 then its the data.table
if depth is 1, we loop through list-like objects using is.list and which are not data.table
if depth > 1, we throw error

added name for the nested list found parent[[1]] or parent$child
pre-allocating info to avoid reallocation cost

[manmita]

Hello,

I created a new PR in replacement of #7568

Reasons: There was some git issue there and the merge became too complex and I changed the algo because I didnt know previously that rbind or cbind would cost for re-allocation

The current PR considers that part and avoids appends

Previous PR : creating seperate data.table called info and rbind at the end
This PR: pre-allocates for a total-sized data.table and fills the info

[manmita]

In reply to previous comment of @jangorecki

An example of when this new feature could be useful?

To support lists which occur due to split.data.table or fread like the following

list(data.table(a = 1, b = 4:6)),
      data.table(a = 2, b = 7:10))

The original code supported data.table() top level and this code adds support for list(data.table) if the arg shallow_search = TRUE

[manmita]

Example of the original code and the new feature is as follows

> A = list(data.table(a = 1, b = 4:6),
      data.table(a = 2, b = 7:10))
> B = list(data.table(a = 1, b = 4:6), 1:5)
> C = data.table(a = 1, b = 4:6)
> tables()
   NAME NROW NCOL MB COLS    KEY
1:    C    3    2  0  a,b [NULL]
Total: 0MB using type_size
> tables(shallow_search = TRUE)
     NAME NROW NCOL MB COLS    KEY
1: A[[1]]    3    2  0  a,b [NULL]
2: A[[2]]    4    2  0  a,b [NULL]
3: B[[1]]    3    2  0  a,b [NULL]
4:      C    3    2  0  a,b [NULL]
Total: 0MB using type_size
> D = list(d = data.table(a = 1, b = 4:6), x = 1:5)
> tables(shallow_search = TRUE)
     NAME NROW NCOL MB COLS    KEY
1: A[[1]]    3    2  0  a,b [NULL]
2: A[[2]]    4    2  0  a,b [NULL]
3: B[[1]]    3    2  0  a,b [NULL]
4:      C    3    2  0  a,b [NULL]
5:    D$d    3    2  0  a,b [NULL]
Total: 0MB using type_size

tables() work same as before and tables(shallow_search = TRUE) searches 1 level

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.03%. Comparing base (6c6615c) to head (e2ae2ce).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #7580   +/-   ##
=======================================
  Coverage   99.02%   99.03%           
=======================================
  Files          87       87           
  Lines       16896    16934   +38     
=======================================
+ Hits        16732    16771   +39     
+ Misses        164      163    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

• HEAD=feat/adding_list_search_to_tables stopped early for DT[,.SD] improved in #4501
  

Generated via commit e2ae2ce

Download link for the artifact containing the test results: ↓ atime-results.zip

Task	Duration
R setup and installing dependencies	2 minutes and 48 seconds
Installing different package versions	43 seconds
Running and plotting the test cases	3 minutes and 31 seconds

[MichaelChirico]

this test should also include a further-nested table to demonstrate that depth=1L is honored:

xenv2$LL = list(list(data.table(a=1L, b=4:6)))

There, we'd need depth=2L to find the data.table, AIUI.

[MichaelChirico]

why xenv2? Where is xenv?

[manmita]

Hello @MichaelChirico , I have added the suggested changes.
regarding the coding style for data.table as data.table(a=1L), in my previous (PRs) tests, I have missed this.
Is it alright if I fix them in this PR?

[ben-schwen]

Hello @MichaelChirico , I have added the suggested changes. regarding the coding style for data.table as data.table(a=1L), in my previous (PRs) tests, I have missed this. Is it alright if I fix them in this PR?

Yes, since we follow the Boy Scout principle here, which is the practice of leaving code slightly better than you found it.

[MichaelChirico]

I don't follow, why would it be slow?

[manmita]

due to format(env) it might be slow, I had tried adding test to increase coverage for this line but atime didnt pass then. will try that again.

[MichaelChirico]

note that inherits(<data.table>, "data.frame") is TRUE (or at least we can assume it to be), so is_dt is redundant

[manmita]

removed is_df and kept only is_dt to count lists of df as well as the below comment says

[MichaelChirico]

anyway, why not also search data.frames?

DF = data.frame(a = 1:2)
DF$b = data.table(c = 3:4, d = 5:6)
str(DF)
# 'data.frame':   2 obs. of  2 variables:
#  $ a: int  1 2
#  $ b:Classes ‘data.table’ and 'data.frame':     2 obs. of  2 variables:
#   ..$ c: int  3 4
#   ..$ d: int  5 6
#   ..- attr(*, ".internal.selfref")=<pointer: 0x561d6ec08e30>

[manmita]

updated the code, and tested

[MichaelChirico]

let's use a helper to share logic between here & the depth=0 branch

[manmita]

@MichaelChirico this test was failing as indices and key column was missing, can you please check if this is the intended testcase? I have updated it to add index.