NumPy sprint: Nov 22-23 2019

Attending: Stefan vdW, Matti, Sebastian, Warren, Chuck, ZJ, Ralf, Inessa, Stephan Hoyer, Joe LaChance, Ross Barnowski

Topics: Day 1

Ralf will introduce each topic with one slide.

NumPy vision and roadmap (current roadmap):

• Purpose: come up with a forward looking vision and prioritize roadmap topics

• Vision:

  • Key terms:

    • Foundational
    • Stable, Adapt

  • Now:

    • High level-not API
    • Audience
    • Foundation for numerical computing in Python
    • Stability
    • De-facto exchange format / data container

  • Future:

    • Don't break it
    • Enable/Serve/Adapt
    • Interact with larger community
    • Support the data science community
    • Stable

  • Implementation: tied to Python

• Roadmap:

  • Testing: Chuck is concerned about the level of testing, coverage.
    • All common platforms: ARM (done), PowerPC, ...
    • Can combine last two items of current Continuous Integration; in essence, make sure we don't break other packages
  • Website
    • First version will be done soon; will still require work on:
      • Translation
      • Content that helps users (educational, installation, onboarding)
      • Marketing material
      • People, teams, roles
    • New sitemap
    • Cross-project documentation
  • Community building
  • Structural concerns about the document:
    • Add status field to roadmap items
    • Should we separate into active items vs aspirational? Hard to classify.
      • Perhaps a list of items that are of continual interest, but not active interest?
  • Performance:
    • Can probably be made 30% to 3x faster, but will require significant effort
    • Hard to justify from vision?
    • Other view is to focus on extensibility, to allow optimizations externally
    • Performance matters to NumPy users, of which there are many
      • We can't easily improve library performance through extensible mechanisms
    • Instead of individual improvements, look for infrastructure to improve (SIMD example, instead of adding one instruction set, add framework for adding all)
  • High level item "user experience" for tracking indexing and other similar items
  • Education & outreach
    • Material development
    • Workshops

Organizational structure & governance

• Purpose: How to organize the project—and the roles within it—to facilitate growth and decision making
  • org structure / governance. website, other, funding, etc. roles (project mgmt, etc.)
  • Meetings
    • Strategy for prioritizing work regularly
  • How to engage with companies (e.g. IBM bounty, np.erf Intel; Amazon hiring people for work on NumPy, etc.). explicit role?
• Organizational structure:
  • Currently only have steering council, everything else is implicit
  • Informally, we now have a web and documentation team
  • Triage team is possible now that GitHub supports those permissions
  • Get rid of core team terminology
  • "Owners" become "Admin team"
  • Weekly meetings:
    • Community meeting alternating with triage meeting
      • Triage: higher level community discussions, website, decisions, getting doc writers unstuck, big picture topics
      • Looks at specific issues and PRs
    • Currently we have several very low-level items on the agenda
    • Make BIDS updates during "high-level" meeting, a general update and keep it short.
  • Teams proposed:
    • Admin
    • Documentation
    • Web
    • Triage
    • API — team that discusses overall API improvements, approves PRs that change API (team can be pinged via @numpy/api)
    • Industry — more of a role than a team; mention this clearly on the website
    • Funding and grants
  • Make it clear that teams are not exclusive and can easily be joined
  • Identify key downstream constituents (astropy, pandas, sklearn)
  • To clean up contributor rights/steering council, we can ask voluntary movement to emeritus or active
    • In industry, every year a board has an election; board members elect to multiple-year (~3) terms
    • Also an opportunity to remind everyone of the responsibilities involved, what is expected
    • Current status: steering council has no defined size, everyone >1 contributor eligible, if member of SC suggests someone they can get added, inactive for 1 yr removed; decision making is complete consensus only
  • Roles proposed:
    • Industry liaison
    • Release manager
    • Internship coordinator
    • Mentorship coordinator
    • Community manager
  • Re: funding requests—we would like to write down a NEP with guidelines of how to be a good player in getting funding "on behalf of" NumPy
  • Typically boards (advisory councils) have monthly meetings to coordinate
  • Suggested: rolling process for replacing members
  • Responsibilities of steering council:
    • Decision making (breaking ties) — classically, this used to be only about technical issues
    • Ambassadors for the project
  • Voting group can be quite wide (in PSF, they ask everyone who spends >5 hrs per week on that)
  • Some prefer 'Advisory Council' so that steering council can steer

Community building

• Purpose: Come up with a plan to make the community friendlier to newcomers
  • A friendlier forum (Discourse)
  • Diversity
  • Welcoming newcomers
  • PR review practices
  • Onboarding for contributors --- Notes ---
  • A friendlier forum
    • Concerns about maintenance and moderation
    • Would be good venue for teams to communicate
    • StackOverflow has 66,900 questions with the NumPy tag
    • General sense to revisit next year, with team comms happening on Slack
    • Spoke to Chris H. from Jupyter about their Discourse experience
      • We will need a community manager to organize and (initially) moderate the forum
      • Maybe room for a scientific numerical computing forum (NumPy "ecosystem")
    • PyData Discourse available for experimentation
  • Outreach
    • Giving talks in person more effective than online
    • Webpack has a representative on Twitter who also gives a lot of talks—helped popularize the project
    • New adopters have to be helped, so how does that happen
    • We need more developers, not more users (already ~10M)
  • Reviewer guidelines
    • Take time to thank contributors, be gentle in rejections
    • Reviewer guidelines (see skimage core dev guide)
      • Less nitpicky reviews
      • Grammar / spelling change requests can be quite intimidating to foreign language speakers
      • Help out with difficult things (such as rebasing, do that for the contributor)
  • Mentorship program
    • We are willing to do this, but require contributors to reach out to take part
    • New role to organize this: have to figure out "entry point"
  • Diversity & inclusion
    • Helpful to have multiple roles in the project
    • Express interest and awareness
    • We have some project income that can be spent on this
    • Wider roles should not be only mechanism for diversifying team; we want to grow the core developer team as well
    • Resource: blog post on diversity and inclusion
    • sklearn WiDS sprints
  • University of Michigan partnership for NumPy survey
    • An annual survey that informs the roadmap would be very useful (Ralf has some feedback from several big projects available already)

API evolution

• Purpose: Define a strategy for sensibly limiting API growth, and reducing it where necessary
  • Propose minimal NumPy(TM) API
  • Generally, subtract rather than add, with guidelines for adding (e.g. not in C99, not in ...)
  • Use cases:
    1. Lots of libraries copy NumPy API, but don't want to copy all 600 odd API functions
    2. Projects like dask, mxnet, etc. try to be exactly compatible with NumPy, but none implement the entire API
    3. Libraries that also want to run on top of other libraries like dask have to be very conservative in which API functions they use
    4. User perspective: very confusing to beginners to know what "optimal API" usage looks like
  • Require light-weight version of API that can reliably be advertised and implemented
  • numpy.essential module for containing these functions, perhaps after they have been marked by a decorator
  • Two cases to consider that are slightly different:
    • What functions libraries should implement (transpose can do everything swapaxes does)
    • What we recommend users use (they'd use swapaxes)
    • We can probably just do the union of these, and it would still greatly reduce API surface
  • Discussed deprecation of Matrix (easy) and factoring out MaskedArrays into separate package (harder)
    • Can we reconsider NEP17?
      • There is a refactoring of MaskedArrays using the new protocols that need to be done, that complicates the matter
        • Feasible plan: write this version, and then gradually deprecate NumPy version in favor of that one
      • Should reach out to Allan H. to gauge his interest in this issue

Interest probe: new NumPy paper

• General interest in doing this, as long as someone takes responsibility for leading the process all the way through
• Ideally few initial writers that complete most of the paper, and then opens it up for community feedback
• Need to try and wrap it up in a short amount of time, otherwise process derails
  • What should the content be? - History - New features? - Future developments? - Success stories (overlap with SciPy paper?)

Topics: Day 2

Video call link

Polynomial

• Discuss "best practices" and ways to guide users toward them (Chuck, Stefan, documentation team)

NEP 36: industry rules of engagement

Dtypes and UFunc rework

• Recap of current work status (decisions)
• Discussion of general design and more detailed decisions such as:
  • UFunc inner-loop, setup/teardown
  • New implementations for ArrFuncs
  • …
• Discuss how to process/review may happen

SIMD support (10–11am)

Scheduled call with Sayed from 10-11am

participants: Ralf, Matti, Chuck, Ross, Sebastian, Warren, Sayed, Raghu

Design: at compile time we check the cpu and build for a baseline specified by the user.

Ralf: we would need to define a minimum baseline for the most general case: sse2 for x86.

Does it do runtime detection? Yes - via generated code for each variant for that CPU and detection code. see npy_cpu_dispatch.h

Raghu: compilers cannot optimize everything: things like gather-scatter for AVX512. How does that work with the unversal intrisics.

Sayed: universal intrinsics in simd.h supply basic building blocks for typical functionality when using SIMD functions.

Raghu: what will happen when someone wants to add new functionality for every architecture?

Sayed explained how the runtime and compile time mechanisms work.

Ralf: the first PR for cpu identification seems ready to go. We still have questions how to use the second PR, and will formulate some questions.

Sayed: I will be expanding some of the example loops and improving benchmarks and tests.

• Generic intrinsics are/can be implemented, allowing to write code which can compile for all SIMD featuresets.

• There are @target comments to generate multiple versions for runtime dispatching.

  • NPY__CPU_DISPATCH_BASELINE_CALL macro to generate the actual versions.
  • NPY__CPU_DISPATCH_CALL.

• PRs of Sayed Adel

• Generic infrastructure + PowerPC support

• Discuss high level design and who will review Sayed's PRs

• Call with Sayed and Raghuveer

• History: OpenCV had the same issue with compiler issues and supporting Power.

• Infrastructure (Compile time):

  • new distutils/checks/cpu_feature.c small programs to test compilation? Each file is a flag?
  • It is possible to compile a baseline which requires certain CPU features. In this case NumPy will raise an error on import if it is not available.

Duck array NEP discussion (11-12)

References:

• NEP 30 — Duck Array Protocol
• NEP 31 — Context-local and global overrides of the NumPy API
• NEP 35 — array creation dispatching with __array_function__
• NEP XX — numpy.api proposal (draft NEP)

Notes:

• Hameer wants a "reverse dispatcher"

args, kwargs = evaluate_dask_arrays(args, kwargs)
orig_func(*args, **kwargs)

with uarray.set_backend(something):
    x = unumpy.ones((5, 5))
    x = some_other_library(x)
x = some_other_library(x)

Sebastian (Opt-In required, but no implicit dispatching):

def library_func(array_like1, array_like2, unimportant_arraylike):
    array_like = np.asduckarray(array_like)  # can allow multipl array-likes.
    onp = np.get_np_lib(array_like1, array_like2)
    m = onp.mean(array_like)
    onp.square(m)
    # keep using onp.…

or:

def library_func(array_like):
    array_like = np.asduckarray(array_like, *args)
    m = np.api.mean(array_like)
    np.api.square(m)

See here for more details on np.get_np_lib.

Design options:

• Opt-In vs. Not Opt-In
• Explicit (which Type controls operation?) vs. Implicit
• Local vs. Non-local (passing around) vs. Global

Current designs or design ideas:

• Array function is Non Opt-In, Implicit and Local.
• UArray is Opt-In, Implicit supports Non-local and Global
• onp = np.get_np_lib(array_like) is Opt-In, Explicit and local or Non-Local (by passing onp out).
• np.api.… (S. H. idea) is Opt-In, Implicit and Local.

Decision:

• Jax, etc. require Opt-In (so we are looking for a new Opt-In solution)
• We need to map the other requirements and figure out what we need before making a final decision.

Hard cases:

• Error handling like np.errstate.
• Methods on objects, like np.random.Generator. Could we just call __array_function__(np.random.Generator.rand, random_state)? But we need array objects.

Stefan H. proposes np.api as a way to expressly call a generic function that will not change the underlying array. "Opt-In requirement" to address backward compatibility concerns (implicit dispatching like __array_function__ after this Opt-In).

Website

• Go over existing content and integrate it
• Make plan for what's missing
• Graphic design needs/ideas
• Make some progress on content & site

Numpy-wheels repo

• Move it under the numpy repo
• update macos,
• move windows builds to Azure
• use multilinux2010
• consolidate OpenBlas ARM and PPC builds to the rackspace location

Dtypes

• DTypeMeta->Dtype with a heirarchy of abstract and concrete dtypes. Concreate types will not be available as base classes, do that via composition or ???
• CastingImpl(dt1, dt2) returns object with methods: adjust_descriptor, get_transferfunction_func. They accept a casting mode.
• One of the design goals is to remove specialized if(dtype is/has X): else:
• Coercion

Some conclusions/questions:

• Can we avoid having lots of custom ufunc loop resolvers? How far can we get with just a single "smart" resolver that does some kind of "best fit" loop matching, presumably based on safe casting rules?

  • Need to review all the wacky special case resolvers currently
  • Do people actually want e.g. a bfloat16 to be implicitly upcast to float32 if there's no other loop available?
  • Impliciations for casting: to define the "best" loop we probably need to be able to ask about safe casting from DTypeType->DTypeType

• What exactly are the operations we need for casting? Probably like 12+ (can_cast, specialize_dtype_for_cast, do_cast, reversed versions, versions that DTypeType vs Dtype). How can we organize them logically so we don't get confused?

• Coercion:

  • can we not special-case scalars at the coercion level?
  • maybe for the pure np.array(obj) case:
    • fixed set of python->dtype mappings for built-in python types only, everything else has a dunder (to get an actual dtype, not a dtypetype) or nothing
    • final dtype is just common_dtype(all individual dtypes)
    • treat the str case where we have to go back and change the dtypes of previous objects as a special-case hack that we deprecate
    • to handle the arr+1 case, have a special coercion mode that ufuncs use, which tweaks the python->dtype mappings to use special "as vague as possible" dtypes, like 1 -> uint7, -1 -> int8, 1.0 -> vaguefloat, etc., where these are dtypes that have special coercion rules (e.g. uint7 can be safely cast to int8, vaguefloat can be safely cast to floatX for all X)
  • for np.array(obj, dtype=dtypetype) (e.g., dtype=Categorical), call a classmethod on the dtypetype and let it pick the output dtype + do the coercion

Typing ndarrays

API overrides

• NEPs 31/32
  • Make sure to tell Hameer the time, schedule it early for East Coast time

• Apply to main repo or maintain as a separate package
• Opt-in or not

• A large part of the discussion centered around Stephan Hoyer's API proposal

How NumPy funding is spent

• Sources
  • Tidelift
  • Donations to NumFOCUS
  • Grant guidance
  • Collaborative funding