gh-156443: Keep PyLong loop carries as twodigits in shifts and division by XiaohongGong · Pull Request #157060 · python/cpython

XiaohongGong · 2026-09-07T09:38:05Z

gh-156443: Keep PyLong loop carries as twodigits in shifts and division

Several functions in longobject.c (v_lshift, v_rshift, and x_divrem) narrowed a loop carry value to digit or sdigit, then widened it again on the next iteration.

On AArch64, that 64-to-32-to-64 conversion inserts an extra mov on the loop-carried critical path. Keeping the carry at twodigits until the function returns drops that mov and shortens the carry chain. Results are unchanged for valid limbs.

Use v_lshift as an example, the loop on AArch64 previously contains a redundant mov w3, w3 on the carry chain:

  ldr  w0, [x4, x2, lsl #2]    ; a[i]
  mov  w3, w3                  ; carry chain
  lsl  x0, x0, x24             ; a[i] << d
  orr  x0, x0, x3              ; carry chain
  and  w1, w0, #0x3fffffff
  ubfx x3, x0, #30, #32        ; carry chain
  str  w1, [x26, x2, lsl #2]
  add  x2, x2, #1
  cmp  x25, x2
  b.ne

Keeping the carry as twodigits removes that narrowing conversion. The loop is optimized to:

  ldr  w0, [x4, x2, lsl #2]    ; a[i]
  lsl  x0, x0, x24             ; a[i] << d
  orr  x0, x0, x3              ; carry chain
  and  w1, w0, #0x3fffffff
  str  w1, [x26, x2, lsl #2]
  add  x2, x2, #1
  lsr  x3, x0, #30             ; carry chain
  cmp  x25, x2
  b.ne

The instruction count on the loop carried chain is reduced from 3 to 2. We can observe ~20% performance improvement of the
pyperformance pidigits benchmark on an NVIDIA Grace CPU, while no material regressions observed on other platforms and benchmarks.

Tests cover divmod of saturated limbs with quotients near BASE (x_divrem's inner loop), and intra-digit shifts via float() and true division: full-limb values and powers of ten.

Fixes gh-156443.

Co-authored-by: Kyrylo Tkachov ktkachov@nvidia.com

…division

python-cla-bot · 2026-09-07T09:38:10Z

All commit authors signed the Contributor License Agreement.

bedevere-app · 2026-09-07T09:38:12Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

XiaohongGong · 2026-09-07T10:35:34Z

Hi, I'm an engineer from NVIDIA that has signed the CLA and PSF. What should I do to pass the CLA check?

picnixz · 2026-09-07T12:46:12Z

Hi, I'm an engineer from NVIDIA that has signed the CLA and PSF. What should I do to pass the CLA check?

Be careful with which email you sign the CLA (see https://devguide.python.org/getting-started/pull-request-lifecycle/#why-do-i-need-to-sign-the-cla-again)

XiaohongGong · 2026-09-08T03:53:47Z

Hi, I'm an engineer from NVIDIA that has signed the CLA and PSF. What should I do to pass the CLA check?

Be careful with which email you sign the CLA (see https://devguide.python.org/getting-started/pull-request-lifecycle/#why-do-i-need-to-sign-the-cla-again)

Thanks for the comment! I confirmed that my company (NVIDIA) has signed the CLA and my email and github are all correct. Any other checks/steps that should I do to pass the CLA?

StanFromIreland · 2026-09-08T06:42:10Z

You'll have to sign (again) by clicking the button above, I'm afraid otherwise we can't do anything here.

XiaohongGong · 2026-09-08T07:41:16Z

You'll have to sign (again) by clicking the button above, I'm afraid otherwise we can't do anything here.

If I sign again by clicking the button above, it means that I will sign on behalf on the individual contributor, which may not be recommended? The contribution is made on behalf of my company. My understanding is that NVIDIA has an existing PSF Contributor Agreement. Could you please let me know whether any additional action is needed to associate this PR/GitHub account with the corporate agreement?

Note that I'v clicked the sign button above and the CLA check has passed. But I'm aware that should be a mistake and I should not do that.

…presses Maintainer review fixes for PR gastownhall#217, iteration 1. The two gating defects are both in coverage_contract.py — the artifact this PR designates the tiebreaker — and both were reproduced with matched controls before being fixed. F1 (major) — _strip_code substituted a space for a code span, and a space is exactly the separator _CLOSING_KEYWORD_REF accepts between a keyword and its reference. Removing inline code from "Fixes `x` gastownhall#42" therefore *created* the keyword->reference adjacency GitHub does not honour: the stripper minted mechanical `closing-keyword` coverage for a body that closes nothing. This is the expensive direction — a body shaped like "fixes `--resume` #3849" makes the tiebreaker instruct an agent to bury a live issue. Code is now replaced by a NUL sentinel, which separates the two sides without ever joining them. F2 (major) — the fence-close alternative is anchored on a MULTILINE `$` that a trailing \r sits in front of, so a properly closed CRLF fence read as unclosed and the `\Z` branch blanked every reference after it. CRLF is the delivery form the GitHub API actually returns for web-authored bodies, and "description, code block, Fixes #N at the bottom" is the standard template. Line endings are now normalised before matching. Confirmed against the real body review identified: python/cpython#157060 classified no-coverage-evidence before this change and yields {156443} after. Folded in the same-character fence-close rule while reworking the regex — a ``` block "closed" by a ~~~ line ended the blanked region early, the same phantom direction as F1. F3 — mol-pr-triage's verdict branch asserted coverage_contract.py implements "exactly these rules" while omitting the evidence floor its sibling states and pins, so a triage agent following that prose could record a Tier-4 demotion citing a span the shared contract rejects: a false Tier 4, the expensive direction again, and the one finding all three review lanes filed independently. The floor is now stated on both sides and mirrored against _MIN_EVIDENCE_CHARS/_MIN_EVIDENCE_WORDS, and the baseRefName restriction names $DEFAULT_BRANCH instead of leaving the fetched variable unused. The suite passed on the shipped tree AND on every fix variant, so green tests are not evidence here and each new test was verified by breaking the thing it guards: reverting the sentinel, the CRLF normalisation, the fence backreference, the floor sentence, the $DEFAULT_BRANCH binding, the repo= forwarding, and the keyword separator each fail their own assertion. Neither regression test passes on a stripper that simply deletes everything — the positive arm is what rules that out. Picked up while in these files: a gate-level test for decide_competing_pr_gate(repo=), which 0 of its 13 call sites exercised; a pin for the no-space `Fixes:gastownhall#42` form, recorded rather than endorsed so a future relaxation is deliberate; and a fixture comment that paraphrased what it presented as a verbatim quotation. tests: 126 passed (was 120). All 6 pack formulas still parse. git diff --check clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

XiaohongGong · 2026-09-11T08:24:11Z

Hi, could any one please take a look at this PR? Thanks a lot!

XiaohongGong · 2026-09-14T03:10:24Z

As on AArch64, the x86 changes also eliminate the mov instructions in the loop-carried dependency chain. Below is the v_lshift loop before the change:

add    $0x1,%rsi
cmp    $0x1,%rdi
je     b9e3 <v_lshift+0x173>
cmp    $0x2,%rdi
je     b907 <v_lshift+0x97>
  //  twodigits acc = (twodigits)a[i] << d | carry;
lea    0x0(,%rsi,4),%r11
mov    (%r9,%r11,1),%eax                             ; a[i]
shl    %cl,%rax                                      ; carry chain
mov    %edx,%edx                                     ; carry chain (narrow)    <--
or     %rdx,%rax                                     ; carry chain
   //  z[i] = (digit)acc & PyLong_MASK;
mov    %eax,%edx
and    $0x3fffffff,%edx
mov    %edx,(%r8,%r11,1)
   // carry = acc >> PyLong_SHIFT;
shr    $0x1e,%rax                                    ; carry chain
mov    %rax,%rdx                                     ; carry chain

After the change, the code in loop is changed to:

add    $0x1,%rsi
cmp    $0x1,%r11
je     b981 <v_lshift+0x151>
cmp    $0x2,%r11
je     b8be <v_lshift+0x8e>
    // twodigits acc = (twodigits)a[i] << d | carry;
lea    0x0(,%rsi,4),%rdi
mov    (%r9,%rdi,1),%edx                              ; a[i]
shl    %cl,%rdx                                       ; carry chain
or     %rdx,%rax                                      ; carry chain
    // z[i] = (digit)acc & PyLong_MASK;
mov    %eax,%edx
and    $0x3fffffff,%edx
mov    %edx,(%r8,%rdi,1)
     // carry = acc >> PyLong_SHIFT;
shr    $0x1e,%rax                                     ; carry chain

I’m not very familiar with x86 systems, but my guess is that we did not see a noticeable improvement because this mov instruction may already be optimized at the hardware level, possibly through register renaming. Therefore, in theory, this should not introduce any regressions on x86 systems.

XiaohongGong · 2026-09-17T09:48:12Z

ping again~

Could anyone please help take a look at this PR? Thanks a lot in advance!

diegorusso

Hello, thanks for the patience with this. I reviewed the PR and both implementation and tests look good to me.
I'm approving it but I prefer to have another pair of eyes to have a look at it.

XiaohongGong · 2026-09-22T02:52:28Z

Thanks a lot for your review @diegorusso !

XiaohongGong · 2026-09-22T06:16:05Z

Hi, can anyone else please take a look at this PR? Thanks a lot!

eendebakpt

Two minor things, but the approach here looks good.

Note: in #157751 we modify the code to skip the carry altogether. @XiaohongGong Could you test that on your platform?

eendebakpt · 2026-09-22T08:35:58Z

@@ -0,0 +1,2 @@
+Optimize shifts and division of large integers. Speeds up the pyperformance


Suggested change

Optimize shifts and division of large integers. Speeds up the pyperformance

Optimize division of large integers. Speeds up the pyperformance

(the actual << on ints is not touched here)

Yeah, what I want to say is the shifts operations of digit, maybe I should change it to:

Optimize shifts and division of large :class:`int` values. Speeds up the pyperformance

Is that better to you?

XiaohongGong · 2026-09-22T09:30:36Z

Two minor things, but the approach here looks good.

Note: in #157751 we modify the code to skip the carry altogether. @XiaohongGong Could you test that on your platform?

Thanks for looking at this PR. Sure, I will take a look at this PR and have a test for that.

XiaohongGong · 2026-09-23T01:21:26Z

Two minor things, but the approach here looks good.
Note: in #157751 we modify the code to skip the carry altogether. @XiaohongGong Could you test that on your platform?

Thanks for looking at this PR. Sure, I will take a look at this PR and have a test for that.

With #157751 , the pidigits benchmark improves by about ~25% across all of my test machines, including AArch64 and X86_64 platforms. This is a wonderful change for those methods. I will look into the details of the PR. Thanks a lot!

pythongh-156443: Keep PyLong loop carries as twodigits in shifts and …

b1373ea

…division

bedevere-app Bot mentioned this pull request Sep 7, 2026

Keep PyLong loop carries as twodigits in shifts and division #156443

Open

bedevere-app Bot added the awaiting review label Sep 7, 2026

blurb-it Bot and others added 3 commits September 7, 2026 09:52

📜🤖 Added by blurb_it.

d2fa4b3

Merge branch 'main' into pythongh-156443-longobject

414792a

Update NEWS docs

5ff5f31

StanFromIreland reviewed Sep 7, 2026

View reviewed changes

Comment thread Misc/NEWS.d/next/Core_and_Builtins/2026-09-07-09-52-21.gh-issue-156443.tT40sm.rst

XiaohongGong added 3 commits September 8, 2026 09:28

Merge branch 'main' into pythongh-156443-longobject

d63c21e

Merge branch 'main' into pythongh-156443-longobject

7f9e7cd

Update NEWS.d doc

9016c9c

StanFromIreland requested a review from diegorusso September 11, 2026 12:25

diegorusso approved these changes Sep 21, 2026

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting review labels Sep 21, 2026

eendebakpt reviewed Sep 22, 2026

View reviewed changes

XiaohongGong added 3 commits September 23, 2026 02:34

Merge branch 'main' into pythongh-156443-longobject

ea44e35

Update unit test and NEWS.d

8b481db

Merge branch 'main' into pythongh-156443-longobject

28d2105

		@@ -0,0 +1,2 @@
		Optimize shifts and division of large integers. Speeds up the pyperformance

	Optimize shifts and division of large integers. Speeds up the pyperformance
	Optimize division of large integers. Speeds up the pyperformance

Uh oh!

Conversation

XiaohongGong commented Sep 7, 2026

Uh oh!

python-cla-bot Bot commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-app Bot commented Sep 7, 2026

Uh oh!

XiaohongGong commented Sep 7, 2026

Uh oh!

picnixz commented Sep 7, 2026

Uh oh!

Uh oh!

XiaohongGong commented Sep 8, 2026

Uh oh!

StanFromIreland commented Sep 8, 2026

Uh oh!

XiaohongGong commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XiaohongGong commented Sep 11, 2026

Uh oh!

XiaohongGong commented Sep 14, 2026

Uh oh!

XiaohongGong commented Sep 17, 2026

Uh oh!

diegorusso left a comment

Choose a reason for hiding this comment

Uh oh!

XiaohongGong commented Sep 22, 2026

Uh oh!

XiaohongGong commented Sep 22, 2026

Uh oh!

eendebakpt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

eendebakpt Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

XiaohongGong Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

XiaohongGong commented Sep 22, 2026

Uh oh!

XiaohongGong commented Sep 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

python-cla-bot Bot commented Sep 7, 2026 •

edited

Loading

XiaohongGong commented Sep 8, 2026 •

edited

Loading

XiaohongGong Sep 23, 2026 •

edited

Loading