Add extern "custom"

[folkertdev]

View all comments

Summary

An extern "custom" fn is a function with a custom ABI that is unknown to rust. Often these are low-level functions that pass arguments in different registers than any standard calling convention.

#[unsafe(naked)]
pub unsafe extern "custom" fn __aeabi_uidivmod() {
    core::arch::naked_asm!(
        "push {{lr}}",
        "sub sp, sp, #4",
        "mov r2, sp",
        "bl {trampoline}",
        "ldr r1, [sp]",
        "add sp, sp, #4",
        "pop {{pc}}",
        trampoline = sym crate::arm::__udivmodsi4
    );
}

unsafe extern "custom" {
	fn __fentry__();
}

History

• Lang proposal: extern "unspecified" for naked functions with arbitrary ABI rust#140566
• Tracking Issue for abi_custom rust#140829
• add extern "custom" functions rust#140770
• Stabilize extern "custom" rust#158504

Important

Since RFCs involve many conversations at once that can be difficult to follow, please use review comment threads on the text changes instead of direct comments on the RFC.

If you don't have a particular section of the RFC to comment on, you can click on the "Comment on this file" button on the top-right corner of the diff, to the right of the "Viewed" checkbox. This will create a separate thread even if others have commented on the file too.

Rendered

[traviscross]

Thanks @folkertdev for putting this together.

@rfcbot fcp merge lang

[rust-rfcbot]

@traviscross has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @scottmcm
• @tmandry
• @traviscross

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

[tmandry]

@rfcbot reviewed

[rust-rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[nikomatsakis]

@rfcbot reviewed

[ChayimFriedman2]

Seeing a RFC entering a final comment period 1 hour after it was opened is extremely rare, and personally makes me worried. Even if it was discussed extensively in the past and all relevant team members are on-board, I believe we should let the community more time to discuss it.

[programmerjake]

imo it's fine since that's what the final comment period is for, announcing and giving people time to look at something before it's accepted, when the team already thinks it's good enough.

[folkertdev]

Right, this is the (final) ping for concerns to be raised. This was coordinated during the T-lang triage meeting.

This feature is based on the following lang proposal, it is basically unchanged from that thread.

• Lang proposal: extern "unspecified" for naked functions with arbitrary ABI rust#140566

[clarfonthey]

I also felt a bit concerned by the sudden FCP, but the methodology here seems pretty canonical since it uses naked functions.

The only potential concern is bikeshedding the name "custom" which could potentially be worded as "none" (since this effectively is no ABI) but that's a pretty weak point and not worth blocking IMHO. Plus, it could be changed even post-acceptance but pre-stabilisation.

I see no reason to slow down on the FCP process.

[steffahn]

Plus, it could be changed even post-acceptance but pre-stabilisation.

I would expect this time frame to be fairly short in this case.

[Lokathor]

"none" is incorrect, as explained in the RFC, there is an ABI happening, it's just not an ABI the compiler knows how to use without assistance.

[clarfonthey]

"none" is incorrect, as explained in the RFC, there is an ABI happening, it's just not an ABI the compiler knows how to use without assistance.

Yes, this is another reason justifying the choice, although I do think that "none" that the compiler knows about is still appropriate: the compiler is not using any ABI for the function, even though yes, technically, ABI will always exist no matter what. Even just jumping to a label is still an ABI, after all.

My point is that this is technically a weak point of contention, but not one I think is worth discussing here, since the name is more than adequately justified.

[kennytm]

in the current implementation in 1.98.0-nightly (2026-07-01 4c9d2bfe4ad7a6566909) returning ! seems to be accepted (but returning an actual never_type i.e. (!) or std::convert::Infallible is rejected). please clarify

#![feature(abi_custom, never_type)]

use std::arch::naked_asm;

#[unsafe(naked)]
unsafe extern "custom" fn foo() -> ! {   // no error.
    naked_asm!("ud2")
}

#[unsafe(naked)]
unsafe extern "custom" fn bar() -> (!) { // error: invalid signature for `extern "custom"` function
    naked_asm!("ud2")
}

View changes since the review

[clarfonthey]

This just feels like a bug IMHO, since no ABI means no way to tell if the function never returns. That said, technically, since crates using this method would benefit from knowing this, perhaps it is a reason to allow it, perhaps via some unsafe(no_return) attribute or something similar.

[scottmcm]

Hmm, is there a reason it needs to care that you wrote arguments? Since you can't call it anyway, would it make more sense to just say "sure, write whatever arguments is helpful" and rust will just not use them for anything (other than rustdoc)?

[poseidon-rises]

I agree with @scottmcm here, since we can't make a direct call, why should we limit the signature? I don't see any disadvantages to adding arbitrary parameters and return types, but personally, I find it easier to parse __aeabi_uidivmod(numerator: u32, denominator: u32) -> (u32, u32) at a glance over __aeabi__uidiv_mod(). Specifically, I don't need to read the exact ABI to get a general gist of the purpose of this function.

[Lokathor]

I agree, the function still has semantic inputs and outputs, even if the compiler doesn't know the ABI for them, so we should prefer that the function show those inputs and outputs even if only as documentation.

[Darksonn]

Since all ways of actually calling an extern "custom" function are unsafe, it doesn't seem required that we also make the functions themselves unsafe. Yes, you must call it with the right ABI for the call to be safe, but that's true for all functions. Defining such a function just seems analogous to creating a raw pointer to me.

View changes since the review

[clarfonthey]

The justification to me seems to be to encourage people to write safety comments describing the actual ABI, but I actually kind of agree and think we could have a separate clippy lint for documenting calling conventions for extern "custom" methods.

Also from the perspective of, should these function definitions be allowed with #[deny(unsafe_code)], I think the answer is yes, since only calling them is unsafe.

[kennytm]

it doesn't seem required that we also make the functions themselves unsafe.

If an extern "custom" fn can be defined as a safe function, it follows that they should also be able to be declared as safe, invalidating lines 96-109 (In an extern "custom" block, functions cannot be marked as safe).

---

The justification to me seems to be to encourage people to write safety comments describing the actual ABI

Maybe that mandatory #[unsafe(naked)] is enough for encouragement

should these function definitions be allowed with #[deny(unsafe_code)], I think the answer is yes, since only calling them is unsafe.

But this argument applies to defining ordinary unsafe functions too, yet this will fire the lint

#[expect(unsafe_code)]
unsafe fn just_defining_a_function_not_calling_it() {}

So unless we further refine unsafe_code into unsafe_use & unsafe_define, this should continue to be denied.

[Lokathor]

If we change the unsafe requirements, lines 96-109 would also be changed. That's not a strong justification either way.

[folkertdev]

Maybe that mandatory #[unsafe(naked)] is enough for encouragement

The unsafe on naked is there to declare that the body faithfully implements the signature. Hence standard naked functions can in fact be safe to call, despite their implementation being inherently unsafe due to the use of inline assembly.

[Darksonn]

Yeah. I guess for extern "custom" functions, whether or not it's an unsafe fn is just as meaningless as which arguments appear in its arguments list. The only real difference is how people will interpret the function when they read it.

For instance, using __aeabi_uidivmod from the original motivation:

/// Multiplies two 64-bit integers.
///
/// # ABI
///
/// The first argument is passed across registers r0 (lower 32 bits) and r1
/// (upper 32 bits). The second argument spans registers r2 and r3. It returns
/// the final 64-bit result across r0 and r1.
#[naked]
pub extern "custom" fn __aeabi_lmul(multiplier: u64, multiplicand: u64) -> u64 {
    ...
}

/// Divides two 32-bit integers, computing both the quotient and remainder.
///
/// # ABI
///
/// The numerator (top number) and denominator (bottom number) are passed into
/// CPU registers r0 and r1. It returns the quotient in register r0 and the
/// remainder in register r1.
///
/// # Safety
///
/// The divisor must be non-zero.
#[naked]
pub unsafe extern "custom" fn __aeabi_uidivmod(numerator: u32, denominator: u32) -> (u32, u32) {
    ...
}

Here, we make __aeabi_lmul safe because it's safe for all inputs, and we make __aeabi_uidivmod unsafe because it's not safe for all inputs. In either case, we also document its ABI, and to actually call it with inline asm you must of course follow that ABI, but that's a separate matter from the function's own safety annotation.

Of course, neither the argument lists or whether it's marked unsafe has any real effect here except insofar it helps the reader understand the function.

[poseidon-rises]

#3980 (comment)

I'll remake the argument above here, these annotations make the functions easier for a reader who just wants to know what some code it doing, and doesn't care about the specifics of the ABI. I think, since it has no effect on the usage of the function, I see no reason to disallow it or add special handling just to deny safe custom ABIs.

[Jules-Bertholet]

Suggested change

	pub unsafe extern "custom" fn __aeabi_uidivmod() {
	pub unsafe extern "custom" fn __aeabi_uidivmod {

Since there's no parameter list, and you can't use a call expression with an extern "custom" function, arguably there should be no parentheses in the declaration. Not sure it's worth changing the language grammar for this, though.

Another option is to use the variadic dots syntax:

Suggested change

	pub unsafe extern "custom" fn __aeabi_uidivmod() {
	pub unsafe extern "custom" fn __aeabi_uidivmod(...) {

View changes since the review

[kennytm]

removing the () breaks the Function grammar, absolutely not worth it for a pretty niche feature.

making it (...) could work, though i like #3980 (comment) more.

[zachs18]

-1 for removing the parens entirely, seems not worth changing the grammar for.

+0.5 for ..., but I'd prefer just allowing arguments and return types for documentation reasons.

[bjorn3]

Variadic arguments are passed differently from regular arguments on some calling conventions though.

[folkertdev]

Given that this is a niche feature, that almost nobody will see much less use, I see no reason to spend any syntax budget here.

it's funny/frustrating how so many RFC threads devolve into discussions about syntax: everyone can have an opinion about syntax. The bar for adding syntax is, very deliberately, extremely high.

[steffahn]

@folkertdev it seems to be human nature or sth. Wadler's law in full effect 😄

[Jules-Bertholet]

Are extern "custom" function pointers allowed? The current implementation on nightly considers unsafe extern "custom" fn(), extern "custom" fn(), and extern "custom" fn(i32) -> u64 to all be valid types.

One option would be to have a single bare extern "custom" function pointer type, to represent a function with known address but unknown calling convention. All other function pointers would be able to coerce to it.

View changes since the review

[steffahn]

Good catch! IMHO, it would make sense if you can only define function items as extern "custom" when they're unsafe, argument-less and returning () (or !), to apply the same kind of restrictions on what fn pointer types are usable.

[Jules-Bertholet]

Would it be worth it to allow the naked_asm! blocks in extern "custom" functions to specify clobbered registers? This could be useful for documentation.

If we allow an explicit parameter list as suggested in #3980, perhaps we could even allow specifying inputs and outputs that reference those parameters (and also the return place?)

View changes since the review

[joshtriplett]

We talked in today's @rust-lang/lang meeting about function signatures on extern "custom" functions.

I proposed, and others on lang agreed, that any signature should be allowed, and used in type checking of custom function pointers.

For instance, if you have some hardware table that wants a function pointer to call, you may want to define a signature for that, to make it easier to catch cases where you used the wrong kind of custom function. You write the signature on the function pointer type in the data structure or similar, you then need to match that signature on the extern "custom" fn, and then having that signature will steer you towards writing the function's asm correctly.

[steffahn]

I’m looking at RFC 2972 constrained_naked - specifically the future possibilities section - and I think that it’s food for thought on this design. Here’s a quote of that section:

Future possibilities

It would be possible to define new calling conventions that can be used with naked functions.

A previous version of this document defined an extern "custom" calling convention. It was observed in conversation that calling conventions are really a type and that it could be useful to have calling conventions as part of the type system. In the interest of moving forward with constrained naked functions, it is best to limit the scope of this RFC and defer this (very good) conversation to a future RFC. As a simple workaround, naked functions which do not conform to their specified calling convention should be marked as unsafe and the caller requirements should be documented in the safety section of the documentation per standard convention.

It may also be possible to loosen the definition of a naked function in a future RFC. For example, it might be possible to allow the use of some additional, possibly new, operands to the asm!() block.

---

In particular the middle part

It was observed in conversation that calling conventions are really a type and that it could be useful to have calling conventions as part of the type system. In the interest of moving forward with constrained naked functions, it is best to limit the scope of this RFC and defer this (very good) conversation to a future RFC. As a simple workaround, naked functions which do not conform to their specified calling convention should be marked as unsafe and the caller requirements should be documented in the safety section of the documentation per standard convention.

seems very relevant IMHO to the discussion of allowing arguments, or allowing non-unsafe signatures to the function. I get the argument that there’s a distinction between cases where it’s just the ABI that’s “special”, but other than that the function is safe to call – from the cases where the function has additional safety predicates.

But I also don’t think this is really how Rust works. If there are still manually-upheld preconditions (not expressed in the type signature) then it simply must be unsafe; no compromises on that: The ABI being unspecified still means that the whole thing is inherently unsafe to use.

I believe that to really allow a "custom" ABI function (and consequently also function pointer type) to call itself truly safe (i.e. non-unsafe), it would need to indicate the ABI at the type level somehow.

I also believe that allowing function arguments would - practically - already allow a way to do this. People could establish a convention where the first argument to a extern "custom" fn(…) is actually a marker type that identifies the actual ABI. If you define your own ABI (by “define”, here I just mean documenting it, in the docs for the MyAbi marker type itself), then function pointers like

let f: extern "custom" fn(MyAbi, x: u32, y: u32) -> u32;

could be sensibly be considered fully safe. They could be safe in the sense that you can then provide a sound way to call them from safe code, e.g. as a callback:

fn call_my_abi(callback: extern "custom" fn(MyAbi, u32, u32) -> u32, x: u32, y: u32) -> u32 { … }

fn main() {
    let f: extern "custom" fn(MyAbi, x: u32, y: u32) -> u32 = todo!();
    let n: u32 = call_my_abi(f, 42, 123); // yay, no unsafe!
}

I also think that establishing such a convention is somewhat questionable for downstream users to do (without any official “blessing” of the practice), since different conventions could be incompatible.¹ Furthermore, if we ever wanted to have a different, more “dedicated” mechanism for specifying custom ABIs on a type level in the future², then it’d be incompatible with this kind of approach. I personally wouldn’t dislike the idea of just blessing this kind of approach though, and to leave any sort of better support for it up to future possibilities.

Speaking of future possibility: I could even imagine a trait like trait CustomAbi<Args, Output> { … }³ that would allow you to somehow define how to use an function of that ABI (on the argument and/or return types that the ABI supports) from Rust. So you can just call foo(x, y) on an extern "custom" fn foo(MyAbi, u32, u32) -> u32 function (or on a function pointer), provided there exists some implementation impl CustomAbi<(u32, u32), u32> for MyAbi.

---

Alternatively, if we don’t want to discuss any of this at this point, it would in my opinion make a lot of sense to uphold the restriction of only unsafe function for now - in order not to close this kind of future possibility.

View changes since the review

Footnotes

1. Imagine one convention is to put a marker type as first argument, another is to put them as last argument, and someone defines some ABIs where basically any Rust type could be handled by passing a pointer - then I can imagine overall-unsound APIs emerging once you create some extern "custom" fn(AbiConvention1, OtherArg, AbiConvention2) that fits both conventions. ↩

2. Something like extern "custom(MyAbi)" fn(u32, u32) -> u32 perhaps? Or extern "custom"(MyAbi) fn(u32, u32) -> u32, i.e. with somewhat more dedicated syntax to keep the type outside of the string literal? ↩

3. The question of what exactly goes inside of this trait is slightly tricky since there’s currently no way to write something like “extern "custom" fn(Self, Args…) -> Output” as a type, that “unpacks” a tuple Args into function arguments. ↩

[programmerjake]

The question of what exactly goes inside of this trait is slightly tricky since there’s currently no way to write something like “extern "custom" fn(Self, Args…) -> Output” as a type, that “unpacks” a tuple Args into function arguments.

There's currently an experimental implementation of fn f(..., #[splat] args: ArgsTuple), tracking issue: rust-lang/rust#153629

[Jules-Bertholet]

and used in type checking of custom function pointers.

I'm skeptical of this. It only gives very partial type safety, because the signature doesn't tell you what the actual ABI is. I could imagine that the illusion of safety might cause more damage than it averts. It also makes it a breaking change to change the signature, which would otherwise be pure documentation (and therefore able to be improved at any time).

I think we should have only a single extern "any" fn(...) pointer type that can represent any signature, at least for now. The ecosystem can experiment with type safety via newtypes, and building it into the language can be revisited once we have a better idea of the use-cases and design space.