html-next/vertical-collection

Infinite Scroll and Occlusion at > 60FPS

@html-next/vertical-collection

Infinite Scroll and Occlusion at > 60FPS

vertical-collection is an ember-addon that is part of the smoke-and-mirrors framework. It focuses on improving initial and re-render performance in high-stress situations by providing a component for performant lists and svelte renders to match a core belief: Don't render the universe, render the scene.

Compatibility

• Ember.js v3.28.0 or above
• Ember CLI v4.4 or above
• Node.js v16 or above

TL;DR svelte render: the fewer things you need to render, the faster your renders will be.

Your web page is a universe, your viewport is the scene. Much like you wouldn't expect a video game to render out-of-scene content, your application should smartly cull the content it doesn't need to care about. Trimming excess content lets the browser perform both initial renders and re-renders at far higher frame-rates, as the only content it needs to focus on for layout is the content the user can see.

vertical-collection augments your existing app, it doesn't ask you to rewrite layouts or logic in order to use it. It will try its best to allow you to keep the conventions, structures, and layouts you want.

Installation

ember install @html-next/vertical-collection

Usage

<VerticalCollection
    @items={{items}}
    @tagName="ul"
    @estimateHeight={{50}}
    @staticHeight={{false}}
    @bufferSize={{1}}
    @renderAll={{false}}
    @renderFromLast={{false}}
    @idForFirstItem={{idForFirstItem}}
    @firstReached={{firstReachedCallback}}
    @lastReached={{lastReachedCallback}}
    @firstVisibleChanged={{firstVisibleChangedCallback}}
    @lastVisibleChanged={{lastVisibleChangedCallback}}
     as |item i|>
    <li>
      {{item.number}} {{i}}
    </li>
</VerticalCollection>

Actions

firstReached - Triggered when scroll reaches the first element in the collection

lastReached- Triggered when scroll reaches the last element in the collection

firstVisibleChanged - Triggered when the first element in the viewport changes

lastVisibleChanged - Triggered when the last element in the viewport changes

Support Matrix

vertical-collection version	Supported Ember versions	Supported Node versions
^v1.x.x	v1.12.0 - v3.8.x	?
^v2.x.x	v2.8.0 - v3.26.x	v12 - ?
^v3.x.x	v2.18.0+	v14+
^v4.x.x	v3.12.0+	v14+

Support, Questions, Collaboration

Join the Ember community on Discord

Features

Infinite Scroll (bi-directional)

Infinite scroll that remains performant even for very long lists is easily achievable with the vertical-collection. It works via a scrollable div or scrollable body.

• bi-directional scrollable div
• scrollable body
• dynamic content sizes
• as a table

Svelte Everything

If it can be trimmer, vertical-collection likes to trim it.

Status

Changelog

Documentation

For updated documentation and demos see http://html-next.github.io/vertical-collection/

Contributing

• Open an Issue for discussion first if you're unsure a feature/fix is wanted.
• Branch off of master (default branch)
• Use descriptive branch names (e.g. <type>/<short-description>)
• PR against master (default branch).

Testing

Make sure you register the test waiter from ember-raf-scheduler. So ember-test-helpers's wait is aware of the scheduled updates.

An example can be found here

License

This project is licensed under the MIT License.

vertical-collection's Issues

Add staticPool option

Depending on use case, there are times when it makes more sense for the collection to have a dynamic number of components based on the measured height of each component rather than a static number based on estimateHeight. Like staticHeight, this option should be opt-in to avoid confusion and allow users to optimize based on their use case.

Turn off vertical collection debug message

Just wondering if would you entertain a PR to add the ability to switch off the vertical collection debug message in the console? We try to keep our app console free of unnecessary debugging messages and deprecation warnings and would love to turn off the vertical-collection one if we could. We currently see it in dev and test.

vertical-collection/index.js

Line 142 in 6f19fcb

if (!/production/.test(app.env) && !/test/.test(app.env)) {

If so, should it appear based off of a flag?

scrolling fixed list renders incorrectly

I have a list of 50 objects, rendered in vertical-collection beta-7. As you can see from the timeline image scrolling is broken:

It starts by filling the height + a couple at the bottom. As you scroll it doesn't add new items until we reach the estimatedHeight at the top

scroll container max-height is calculated (does not render entire list)

Hi everyone,

thanks for the great addon!
Today I just ran into a problem.

The list is not completely rendered and just fills up when I start to scroll it.
Guess the reason is that the surrounding scroll container has a calculated height like this:

  max-height: calc(100vh - 170px);

Any idea how to redraw the list properly?

Thanks
Andreas

Expose the key used for an item when sending an action

See html-next/smoke-and-mirrors#154

Handle the collection starting hidden and off-screen

See: html-next/smoke-and-mirrors#95
See: html-next/smoke-and-mirrors#150
And: html-next/smoke-and-mirrors#151

Confirmation that this is still a problem in 1.0 beta 3: html-next/smoke-and-mirrors#151 (comment)

Add ability to insert block above or below scrollable content

Hello,

I have a use case where I want to insert a block of content at the very beginning of a collection (index 0 AND at very beginning of results e.g. no next link), the way I had thought to do this was a simple block inside the block form of vertical-collection but it seems there is no yield in the template to make this possible.

Could we add such a capability? Example:

{{#vertical-collection
    items
    as |item index section|}}
  {{#if (and isAtBeginning (eq section 'above') (eq index 0))}}
    <h1>Yay we are at the very beginning!</h1>
  {{/if}}
  ...
{{/vertical-collection}}

component/template

{{yield 'above'}}

{{#each virtualComponents key="id" as |virtualComponent index| ~}}
  {{~unbound virtualComponent.upperBound~}}
  {{~#if virtualComponent.element ~}}
    {{{unbound virtualComponent.element}}}
  {{~else~}}
    {{~yield virtualComponent.content virtualComponent.index ~}}
  {{~/if~}}
  {{~unbound virtualComponent.lowerBound~}}
{{~/each}}

{{yield 'below'}}

{{#if shouldYieldToInverse}}
  {{yield to="inverse"}}
{{/if}}

[META] can we loosen min-height requirements?

See (among others): html-next/smoke-and-mirrors#127

Focus out not being called on input fields

Bug: Yes

Version: 1.0.0-beta.9

OS: iOS 11

Browser: Safari

Description:
Table row contains input fields and one field is focused. When the focused input goes outside of the currently shown rows "focus-out" is not triggered.

Repo that demonstrates the issue:
https://github.com/dusanstanojeviccs/ember-vc-issue-no-focus-out

Steps to recreating the issue:

1. Get the repo
2. run "ember s"
3. Go to localhost:4200/demo
4. With console open click into the first input field
5. Scroll the table
6. Click into an input field

Expected:
In console it logs the number of focus outs. (in this case 1)

Actual:
In Chrome Desktop behaves as expected, in Safari on iOS nothing is printed to the console.

This issue makes building something similar to excel with VC impossible.

VC does not detect table body height change.

My table is using VC and only 2 rows are rendered at the beginning:

You can see entire white space below the first 2 rows. However, as you start scrolling all other rows start to render.

To reproduce this:

• Checkout my branch https://github.com/billy-addepar/vertical-collection/tree/billy/table-reproduction
• Go to route http://localhost:4200/#/examples/table

This happens because in the first RAF execution, the table only measures the header and the body height is 0. VC detects 0 container height and only renders 2 items. However, as the body has something inside, the table body expands to its full height leaving entire blank space below 2 rows. This is not a resize event. VC does not detect change in body height and does not update accordingly.

When you start scrolling, the VC sees the new container height and renders all rows as expected. All these events could be observed by putting a debugger in the radar.

Note that this only happens for my table since I have some special CSS applied to table. (check app.css).

Not sure how to fix this as I cannot find an easy way to detect change in table body height.

Vertical collection does not work very well with IE11 and Edge

The plugin is just great in Chrome and Firefox. Sadly if I watch the table demo in the Windows browsers, not so much.

Internet Explorer 11
First entries are loaded, but when scrolling down no new entries are drawn:

Edge
Only first entry row is drawn (when scrolling up and down, all entries are drawn, so it seems some initialization error):

Components not rendering

@erkarl is experiencing an issue where components seem to not render correctly at all:

The associated code is:

{{#vertical-collection
  content=jobs
  key='@index'
  containerSelector="body"
  defaultHeight=125
  as |job index|
}}
  {{job-item job=job}}
{{/vertical-collection}}

The project is open source so we can debug directly.

breaks on `{{#if}}` direct inside `{{#vertical-collection}}`

This template:

<section>
  {{#vertical-collection data
    minHeight=10
    containerSelector="body"
    alwaysRemeasure=true
    as |item|
  }}
    <section>
      Item {{item.id}}
    </section>
    {{#if item.detail}}
      <section>
        Detail
      </section>
    {{/if}}
  {{/vertical-collection}}
</section>

with this data:

data: Em.computed({
  get() {
    const arr = [];
    for(let i = 0; i < 1000; i++) {
      arr.pushObject({
        detail: true,
        id: i
      });
    }

    return arr;
  }
})

breaks with

Error: Failed to execute 'removeChild' on 'Node': The node to be removed is not a child of this node.
at Object.clear (ember.debug.js:45707)
at UpdatableBlockTracker.reset (ember.debug.js:46044)
at TryOpcode.handleException (ember.debug.js:54858)
at UpdatingVMFrame.handleException (ember.debug.js:55040)
at UpdatingVM._throw [as throw] (ember.debug.js:54753)
at Assert.evaluate (ember.debug.js:50122)
at UpdatingVM.execute (ember.debug.js:54740)
at RenderResult.rerender (ember.debug.js:54679)
at RootState._this.render (ember.debug.js:12161)
at runInTransaction (ember.debug.js:23478)

I'm not sure if this should be supported, but it AFAIK worked in smoke-and-mirrors.

Unknown plugin "transform-es2015-block-scoping"

Heads up folks! We're getting this error after ember install @html-next/vertical-collection.

ReferenceError: Unknown plugin "transform-es2015-block-scoping" specified in "base" at 0, attempted to resolve relative to "modules/vertical-collection/-debug/edge-visualization"

Thoughts?

ember -v

ember-cli: 2.9.1
node: 4.2.6
os: darwin x64

Thanks Folks!

Add documentation

Hello, do you guys plan to add documentation to this library? The aim of this project is promising but without proper documentation it's hard for people coming into it new...

Can't get a basic sample

Wanted to play around and test, but I can't even get a basic sample up and running.

see alexander-alvarez/vertical-collection-test@925638d

any ideas as to what I'm doing wrong?

Fix append positioning behavior for collections with `renderFromLast=true`

Currently, collections will never fix their scroll position when items are appended. When collections use renderFromLast=true and they are scrolled all the way to the bottom, appended items should push scroll top down more. This is the expected behavior for messaging apps and other users of this setting.

Removing items will throw an error

Removing items will throw an index out of bounds assertion error.

Threw together a branch illustrating this effect

https://github.com/srowhani/vertical-collection/tree/bug/remove-items

  removeItems() {
      const items = this.get('model.numbers');
      items.splice(0, this.get('numItemsToRemove'));
      this.set('model.numbers', items.slice());
    },

Uncaught TypeError: Cannot read property 'append' of null at DynamicRadar.append

We have been using this plugin on production and encountered some issues with the application not loading occasionally. It happens very rarely and I'm not sure what is causing this. It can happen randomly when we refresh the page enough times but we haven't been able to identify a pattern. The error that shows up in the console is:

-private.js:1406 Uncaught TypeError: Cannot read property 'append' of null
    at DynamicRadar.append (-private.js:1406)
    at Class.<anonymous> (component.js:115)
    at ComputedPropertyPrototype.get (ember-metal.js:3274)
    at get (ember-metal.js:2608)
    at RootPropertyReference.compute (references.js:203)
    at RootPropertyReference.value (references.js:70)
    at ArrayIterable.iterate (iterable.js:257)
    at IterationArtifacts.isEmpty (reference.js:502)
    at IterablePresenceReference.value (runtime.js:2889)
    at ReferenceCache.revalidate (reference.js:386)

At that point it looks like skipList is null for some reason:

DynamicRadar.prototype.append = function append(numAppended) {
    _Radar.prototype.append.call(this, numAppended);

    this.skipList.append(numAppended);
};

Any ideas why this could be happening?

Uncaught: Changing a view's elementId after creation is not allowed

Ember Version: 2.10.2
Vertical Collection Version: 58a7e4d

Getting an uncaught error being thrown during the scheduler's run.end():

Changing a view's elementId after creation is not allowed

Error
    at Object.EmberError [as Error] (http://localhost:4200/assets/vendor.js:27636:14)
    at eval (eval at <anonymous> (http://localhost:4200/assets/vendor.js:52422:11), <anonymous>:1:17)
    at Class.<anonymous> (http://localhost:4200/assets/vendor.js:52422:11)
    at Object.sendEvent (http://localhost:4200/assets/vendor.js:27972:18)
    at notifyBeforeObservers (http://localhost:4200/assets/vendor.js:31918:25)
    at Object.propertyWillChange (http://localhost:4200/assets/vendor.js:31701:7)
    at Object.set (http://localhost:4200/assets/vendor.js:32141:34)
    at http://localhost:4200/assets/vendor.js:32976:33
    at Object.changeProperties (http://localhost:4200/assets/vendor.js:31901:16)
    at Object.setProperties (http://localhost:4200/assets/vendor.js:32969:32)

Vertical collection is a bit janky when used with liquid-fire

I've got a table with rows that are rendered via vertical-collection. I'd like to use liquid-fire to fade the table in, but when I do so the rows in the vertical-collection render after the fade animation happens, so there's a noticable pop in of content post-animation. Ideally there'd be a way to coordinate row rendering with liquid-fire so the rows would be rendered prior to fading in, although I'm not sure what that would actually entail in practice...

Can't scroll to bottom on dbMon example in Chrome

When scrolled down to the bottom-most , scroll stuck , can't get go the bottom.

`index` is not passed into the component on initial render

we pass the correct item, but not index. This should be added here: https://github.com/html-next/vertical-collection/blob/master/addon/-private/data-view/radar/radar.js#L276

Vertical-collection is choppy compare to smoke-and-mirror in dbMon demo, when scrolled down.

https://html-next.github.io/vertical-collection/#/examples/dbmon

When scrolled to bottom , the performance become choppy on low end intel Core2Duo 3rd gen machine with 2GB memory. It is fine on high performance machine.

master currently failing all tests

Seems to have happened as a result of this PR: #62

Build is here: https://travis-ci.org/html-next/vertical-collection/builds/226719240

Unlike other broken builds, this failure happens consistently across all ember versions tested.

cc @runspired

Weird padding/margin on occluded-content

The landscape mode has way too much “padding/margin” when scrolling (on occluded content).

https://drive.google.com/file/d/0B4OTX4eSH_Plb0ExY25TdVc5eXc/view

Also tested it on real devices (iOS 10 Chrome which is just a Safari wrapper) and it seems a legit bug.

Source: https://github.com/erkarl/fronthat/blob/master/app/components/job-list/component.js
Live demo: https://fronthat.com/

Item height must always be above minimum value

I installed ember @html-next/vertical-collection and used vertical collection. When I switch mobile view and web view in browser. I got the following error.
Item height must always be above minimum value
Pls help me.

Acceptance Test Error in PhantomJS

Attempted to remove a handler from an unknown element or an element with no handlers

I suspect this has to do with the setTimeout in {{vertical-collection}} not registered with Ember run loop. So the test failed to wait for the render cycle to complete. This make sense too as the test I have fails on every fifth build.

Change `bufferSize` to be a static number of components

bufferSize is currently a multiplier of the scrollContainer height. This makes it hard to have granular control over the buffers, and because the point of them is to prevent flickering in general they don't really need to scale with the scrollContainer, so much as be a certain size relative to scroll speed.

Remove scroll handlers can be called before they are added

If teardown occurs before the first scheduled sync, then removeScrollHandler will error because the scroll handler hasn't been installed yet.

This can happen if a user navigates away from the page that displays the component at precisely the moment between init and sync.

`lastReached` and `firstReached` can continue firing forever

We don't currently guard against lastReached and firstReached firing more than once for the same item, so they can keep firing after appends/prepends that don't actually affect what is rendered.

Noticeable lag between rendering and actions being sent

ping @antidis

See previous conversation on this commit

It seems that using run.next to schedule actions causes the actions to lag a little bit before being sent. The lag is noticeable, @antidis I'm assuming that scrolling are the pattern between 3400ms and 4200ms and the actions are sent at 4800ms?

What version of Ember are you using currently?

Integration tests fail

Hi,

When I run an integration test of a component using vertical-collection I always get this error:

Error: Attempted to remove a handler from an unknown element or an element with no handlers

I was able to reproduce this in a new empty ember app so it's not because of my application. Sadly I couldn't create a twiddle somehow..

Doesn't work in zoomed containers

vertical-collection will blow up if we try to use an arbitrarily sized container (e.g. in tests). We need some way to account for this both using transforms (standard Ember testing container) and zoom ideally. Users should be able to drop vertical-collection into their app and expect tests to work without having to modify the test container's styles.

Tests fail sporadically

Tests are failing fairly regularly on Travis. I have a feeling it's a subtle issue we're not capturing, but it's hard to say.

Add logs for failures to this issue.

Cookbook Recipes

Tracking the list of recipes we need to add to the cookbook to show users ways to use VC in their apps/libraries.

Performance

• Proxying for transient state (storing component state for occluded items, like in ember-table)

A11y

• Dynamically loading more content in an accessible way (from @MelSumner "Replace automatic infinite scrolling with a “Load more results…” button or link that explicitly invites the user to add more. Once they do a few times, prompt them to ask if they’d like to turn auto-loading of more results on.")

Tests fail to connect to testem sometimes

Example log file from Travis: log.txt

Should use `max-height` property of scrollContainer if it exists

max-height is a valid way to define the bounds of a scrollContainer, so it makes sense to use that if it exists.

Mobile Safari "scroll to top" + Desktop keyboard shortcut jump to top

While implementing this on a mobile web app, I used the scrollable body demo and it works great. I noticed an issue when trying to scroll to the top by touching on the top bar in iOS. I am using an "infinite load" concept to load more items onto the page. When you scroll to the bottom, it makes an ajax call to the API to get more items. After those items load, the model is updated and the new items show on the page. When you touch the top to scroll up, it staggers and sends you up to random parts of the page. You essentially have to touch the top a few times in order for it to be fully back at the top.

The same can be reproduced on desktop by using the keyboard (on mac, Cmd + Up / Down).

Here is a gif of my experience: https://www.dropbox.com/s/blamakzadktn36g/issue.gif?dl=0

The infinity loader plugin I'm using is https://github.com/hhff/ember-infinity

Get vertical-collection ready for v1.0.0

Here's the shortlist of the things we need for this to be ready for v1.0.0:

• accessibility
  • renderEverything switch
  • ensure focus updates correctly on scroll
• Documentation
  • More examples in general
  • Better coverage of the API docs
  • Table quirks (@runspired will also blog about this)
• Rollup -private (@runspired thinks this should yield ~50% reduction in file size)
• Improve debug Visualization performance
• Fastboot
  • granular control for fastboot (can be moved post 1.0 if significant refactoring needed)
• Tests. All kinds of tests. For everything. Mostly should be integration tests, unit tests can follow when the internals are more solidified.
  • Tests with RecordArrays
  • Tests with ArrayProxies
  • Tests with PromiseArrays
  • Tests with normal arrays
  • Tests with Ember.A() arrays
• Yield to inverse logic. Should yield when items is empty
• Resizing logic. Whenever we resize the scroll container, we need to update the VCs and rerender.
• Yield heightDidChange action for when height changes on an already rendered item

Attempt to remove scrollHandler failing in tests

Intermittently in tests, I get an error when using vertical collection with an error like this:

Error: Attempted to remove a handler from an unknown element or an element with no handlers
    at ScrollHandler.removeScrollHandler (-private.js:1756)
    at removeScrollHandler (-private.js:1817)
    at Class.willDestroy [as _super] (component.js:199)
    at Class.willDestroy (debug-mixin.js:82)
    at Class.superWrapper [as willDestroy] (ember-utils.js:426)
    at invokeWithOnError (backburner.js:283)
    at Queue.flush (backburner.js:153)
    at DeferredActionQueues.flush (backburner.js:345)
    at Backburner.end (backburner.js:455)
    at Backburner.run (backburner.js:533)

This is causing some of my async tests to fail.

padding on the container and renderFromLast

So my scroll container has padding on it, and I want to load scrolled to the bottom so I've set renderFromLast to true, this seems to work except the scroll position doesn't seem to take into account the container top and bottom padding.

I can work around this in my app with different css but I wonder if this should be handled by the library?

Add mouseenter/mouseleave events

It's not clear to me what is the idiomatic way of adding mouseevent/mouseleave events on each entry of the collection. Using event delegation is not easy because those events don't bubble.

[Meta] CSS Debugging

Post all of our awesome refactoring, the visualizer and CSS debugger are largely broken. I figure this is a good time to revisit them and want to lay out a couple of things we should list as common traps.

Original Implementation Issue: html-next/smoke-and-mirrors#59
Original Implementation: https://github.com/runspired/smoke-and-mirrors/blob/feat/virtualization/addon/-debug/edge-visualization/debug-mixin.js#L53

The list below is incomplete, I just wanted to get it started

Visualizer

• visualize where we think each item is and how large it is.
• hovering over an item in the visualizer should show the index, key, item data, and sizing data

CSS Debugging

• Warn for use of table over tbody
• ensure touch CSS is set

 '-webkit-overflow-scrolling': 'touch',
 'overflow-scrolling': 'touch',
 'overflow-y': 'scroll'

• Warn if GPU acceleration is not used on base element or is over-used.
• Warn if positioning is done poorly
• Warn if floats are used without proper width

SkipList breaks when heights are not integers

Because the SkipList uses integer typed arrays as it's underlying store, it breaks if items are not pure integers. See the Scrollable Body example and scroll to the bottom to see this in action - it keeps detecting a difference between the actual height and the stored height, and it adds the delta to the total which then becomes inaccurate.

We could round out the values before we add them, but I think there could be small glitches due to the differences between actual heights and measured heights. Using Float32Arrays instead of UInt32Arrays is safer, and would only increase the memory usage for the bottom layer (which is currently a UInt16Array).

Cannot read property 'values' of undefined, in debug=true mode

When I pass debug= true,
I see the following error in console.

Wanted to debug why the implementation is not very smooth for us.
But with that error, the scrolling stutters even more.

Current invocation in our app.

{{#vertical-collection events
      classNames="vertical-collection__events-list"
      minHeight=200
      firstReached="loadAbove"
      lastReached="loadBelow"
      staticHeight=true
      bufferSize=4
      debug=true
      debugCSS=true
      as |event|
    }}

{{ our component}}
{{/vertical-collection}}

Granted our component is little on the heavier side, and there is too much stuff going on there. Is any anyway to speed the experience up?

Tracking of states/properties in components within vertical-collection

When the addon is adding and removing components with nested subcomponents, I'm noticing that if we quickly go to the bottom then to the top, it's mixing up the states of each individual component instance. To give some details on the set up of this list, we have a vertical-collection of activity-card components:

{{#vertical-collection activities containerSelector="body" lastReached=showInfinityLoader as |activity index|}}
    {{activity-card activity=activity}}
{{/vertical-collection}}

In each activity-card component, amongst the many types of activities that exists, it's possible to have a message type activity that shows contents of a message sent through the application, and if it's longer than 100 characters, we set a property "fullMessageShown" to false. When a user clicks "Show more" it sets that property to true.

When I'm following closely in the inspector to see what's going on, it looks like vertical-collection is re-using the component containers (at random) that has the state of that component's instance to display new content. This behavior reflects the description on your Medium blog post going in depth on how this was implemented:

To accomplish this, vertical-collection only ever updates the content of the components for Glimmer and Ember to rerender. The ordering of non-updated components, and their content, remains intact. The changed components are then manually moved. (This works because Glimmer only needs to maintain a stable reference to an existing DOM node, similar to the way that ember-wormhole works.)

Source

Therefore, whichever container that now has the new content, the properties are preserved so using the example before, the activity component instance that had "fullMessageShown" set to true, can hold random new content and so when we scroll back up, it's possible that the activity component now is rendered in a different container so the state of "fullMessageShown" is not correct.

It doesn't seem like there's a way to do this in the options the plugin exposes, so what would be a good workaround for this if we need to keep the states/properties of the instance in sync with the content of the card? The only option we seem to have is populating an array or object with the corresponding IDs and states, and managing these states independent of the component instance.

`private` fails to compile correctly on Ember-CLI 2.11

_verticalCollectionPrivate.Token is not a constructor error TypeError: _verticalCollectionPrivate.Token is not a constructor

Issues with race conditions and table crashing when browser resizes up

Possible bug in Firefox, scrollContainer too high

The FrontHAT app seems to have too much space at the end of the scroll container, specifically in Firefox