Design Meeting Notes - January 24, 2013

Buffering data reader perf

The change to make EF queries buffer by default caused some of our perf tests to regress. Changes were made to bring back some of this perf by, for example, switching to sequential access. After these changes:

• A small number of tests are still showing a regression
• Many more tests are not showing any regression or are showing improvements
• The spatial tests are still showing significant regressions

Preliminary numbers are below. Note that:

• These numbers may change as the results are stabilized.
• We don’t yet have numbers for all the other tests.

Conclusions/next steps:

• The spatial regressions are likely caused by a different aspect of the change than the other regressions
  • For example, possibly the serialization of spatial types for conversion
  • These will be investigated separately and there seems a good chance improvements can be made
• We need to understand better why joins and especially ordering are specifically regressing
• We need to get the full stabilized numbers and take a holistic view of perf across all scenarios
  • If we are getting significant improvements in other areas, then small regressions here may be acceptable
  • This is especially true since using AsStreaming brings perf back to where it was
• Other changes (such as o-c caching) may also help improve perf in important scenarios
• If, after the investigations above, the regressions are unacceptable, then buffering may need to be off by default
  • This means that MARS may still be required in many common scenarios
  • Switching on connection resiliency will need to enable buffering

Related: what should happen if AsStreaming is used while connection resiliency is enabled?

Options are:

• Let things continue, but then potentially get inconsistent state if a retry happens
• Throw only if a retry is needed, but this may not happen until the app is in production, and it is not very obvious that this could happen unless you know intimately the way AsStreaming and resiliency interact
• Fail fast for the combination. This is safe and helps drive developers to success. We will do this.CodePlex 822

Provider-agnostic testing

We want to allow more testing against non-SQL Server providers/backends. Investigation of our existing test suites shows that many of the tests assume that they are running against SQL Server. For example, the tests use:

• Store-specific types
• Store-specific SQL
  • Queries
  • Create table
  • Create procedure
  • Drop table
  • Execute sp_*
• Store-specific features
  • Schemas
  • Identity/Auto-increment
• Provider-specific metadata
  • Invariant name
  • Manifest token
  • SSDL
• Provider-specific types
  • SqlConnection
  • SqlCeConnection
• Optional provider functionality
  • Database.Create
  • Database.Exists
  • Database.Delete
  • Migrations
  • Spatial

Also many of the existing tests that would work with other providers do so because they don’t really make use of the provider at all. For example, tests that change states in the state manager. There is no real value in running such tests against different providers.

Provider agnostic tests should cover the following:

• Required
  • Transform delete/query/insert/update command trees into DML
  • Store schema mappings (for reverse engineer)
  • Translate between EDM and store types
  • Get supported facets for an EDM type
  • Get store functions
  • Get store types
• Optional
  • Create a database
    • Transform SSDL into DDL
  • Check if a database exists
  • Delete a database
  • Migrations support
  • Spatial support

Conclusions:

• It doesn’t really make sense to try to get all the existing tests to run against different providers for the reasons above
• We should instead create a new provider-agnostic test suite
  • Existing functional tests that make use of the provider and are already agnostic can be moved to this suite
  • Specifically written new provider agnostic coverage will be added here
    • Also, if a new test makes use of the provider and can be written in an agnostic way, then it should also be added here
  • The agnostic suite will be run as part of “build” for check-ins, but only against SQL Server
    • We could possibly change this to SQLite in the future
  • On the CI machines the suite will be run against multiple providers
    • We should contact provider-writers to get additional providers onto the CI machines

Porting existing SQL tests

Following on from the discussion last week, we will:

• Port the existing tests as end-to-end tests with assertions against the results
  • This is not as expensive as doing full-on CQT assertions
  • This provides some good end-to-end coverage we are missing right now
  • This is more robust that the SQL baselining
• We should still investigate CQT assertions