Post by @daztucker@infosec.exchange

2026-09-28 23:18 UTC

@darrenmoffat@mastodon.social @JdeBP@tty0.social @ska@social.treehouse.systems I've been waiting until I had some time and a keyboard to reply to this, since it won't be short. Before getting into the question, I'll first explain the problem from my perspective. Well problem*s*: 1) PAM interacts with PAM applications (that's sshd in this case) using a callback "conversation function". You give PAM a function pointer inside the application, and if it wants to talk to the user (eg to ask for a password), it'll call that function with a pointer to an array[0] of structs that say things like "ask for the password and don't echo keystrokes". As far as the application is concerned, it calls pam_authenticate() or whatever, and that call blocks, then if it needs to PAM magically calls back into the app's conversation function. In sshd the call stack looks something like: sshd main -> sshd event loop -> sshd auth kbdint-> pam_authenticate() -> ???? -> sshd_pam_conv. 2) PAM modules get loaded into the address space of the application, and can do pretty much anything without restriction. The application has zero insight into or control over those things. From the application's perspective, any PAM call (eg pam_authenticate, pam_setcred, pam_session, pam_chauthtok) requiring interaction *blocks*. If interacting with your user is simple (eg a blocking read from /dev/tty) then this is OK. In sshd, the pam_authenticate call blocks until it's done, but in order to be done it needs to interact with the user, and to do that it needs to run the event loop, because it needs to send the prompt, then process replies, all while doing various other things needed by the SSH protocol. The first attempt to deal with this was to have the conversation function call the event loop *again*. This was pretty icky. Another attempt to deal with this came from FreeBSD, where they put the sshd event loop and the PAM calls in different threads. This solved the blocking problem, but introduced a new one: all of sshd, PAM and every PAM module are all loaded into the same address space, so they all need to be thread safe. I'm not sure if FreeBSD could guarantee that for their PAM stack and modules, but we were not at all convinced that all of those would be true on all platforms. This lead @damienmiller@hachyderm.io to take the FreeBSD code, but shim the pthread_ functions to make it use *processes* instead of *threads*. This solved the shared-address space problem, but introduced new ones. Remember how I said PAM modules could do whatever they wanted? Some of them do things like stash secrets in auth to be used later, or set random supplementary GIDs in the calling process. Since the auth part was now in a separate process, this state wasn't propagated. That lead some OS vendors to re-enable the threading code and disabling the split process again. I think the define was USE_POSIX_THREADS. I changed that to the current UNSUPPORTED_POSIX_THREADS_HACK to make it clear that anyone using it was on their own. Over the years we looked at other solutions such as swapping stacks using makecontext/setcontext (which would likely work well but were deprecated by POSIX) or tricks like sigaltstack (which would probably also work and would be portable but hacky) but changing it would be a large undertaking, and would require *years* of debugging, support and maintenance. It's mostly only Damien and me maintaining this, we both have full time jobs, neither of which are this, and so far neither of us have had the time or inclination to embark on anything like this. So where do we go from here? Well the sshd process model has change significantly since then, so it might be possible to do better now, but again, so far neither of us have had the time or inclination to embark on that. If PAM implementations could agree on and implement a non-blocking equivalent of conversation functions (similar to AIX's native authenticate[1]), we might switch and even drop support for the old functions, but again, that'd be a large undertaking. So that's where we are now. Hope that helps explain how we got here. [0] or an array of pointers, depending on the PAM implementation. But that's a separate problem. [1] https://www.ibm.com/docs/en/aix/7.1.0?topic=authenticate-subroutine

Replies (1)

@ska@social.treehouse.systems 2026-09-29 00:54

@daztucker@infosec.exchange @darrenmoffat@mastodon.social @JdeBP@tty0.social @damienmiller@hachyderm.io Thank you a lot for the explanation! If you're around, and can (at your own pace, bless asynchronous media such as fedi) answer a few more questions, this might really help me, and I hope us, going forward. Here's the context: I want to learn about the details of PAM usage and encountered problems and what clients wish they had instead. No project I know that uses PAM is entirely happy with it (for good reasons), and I may soon have an opening to do something radical: design and write a complete replacement. And so, I want to be sure of exactly where and how PAM fails, in order to make something better, in the sense of: it can do everything PAM does, it provides better security, it doesn't require clients to bend themselves into a pretzel to do things correctly, and the API is friendly enough so that clients are willing to use it. I am aware of the address space issue. Some time ago I wrote a PAM implementation that aimed to protect against that: https://skarnet.org/software/pamela/ but of course it's not usable as a drop-in replacement since so many PAM modules and applications want to use the fact that modules are loaded in the same address space to share global data, such as environment variables. Bleh. I initially thought that the reluctance of a lot of projects to use PAM conversations was simply due to pam_conv being cumbersome to use, but you also gave another explanation: PAM conversations are synchronous, so they're not usable in a project organized around an async event loop. This is a useful insight for me: in order to work, a PAM replacement must provide a way to have asynchronous conversations. Thank you for that. The API I have in mind looks roughly like: (it won't be a pam_ prefix but bear with me) pam_start() to start a session. It spawns a helper process, for address space isolation. The helper process loads the modules, rewritten not to share global data with the application (basically the whole module system gets rewritten, probably using executables instead of shared objects)Various primitives that don't block, don't make networking calls, don't require any callback to the application: these can be synchronous.Asynchronous primitives, let's say pam_send(), that send something to the helper but return instantly, they're usable in an async loop.You get a file descriptor with pam_fd() and add it to your select/poll fd list.When the fd is readable, you call pam_update(). That invisibly advances conversations, making callbacks that you have registered, etc. Depending on the return code from pam_update(), you may have gotten a result, which you then check in previously set up data structures, and decide what to do from there.For simple cases where you don't need async stuff, there's a fully synchronous layer that encapsulates all the fd/poll/update thing.pam_end() cleans up everything, the helper dies and is silently reaped, etc.The only code sharing address space with your application is the lightweight client-side that basically just serializes data and sends it to the helper process, and back. For those who like that, it can even be statically linked: no dlopen() there or anything of the kind. The goal is for the library to be used naturally by clients without needing threads or whatever isolation headaches PAM clients are using. It is a programming pattern that I use regularly, I have the necessary library infrastructure for it, I like it and it does everything I need; however, a PAM replacement would not be for me, it would be for other people. So I have the following questions: If you were an application needing to use a PAM-equivalent stuff to authenticate users and whatnot, would you use such an API? Would it be user-friendly enough that you decide you can work with it? would it integrate well enough with your workflow that it's natural, and if not, what pattern would you prefer?Assuming all the backends (previously known as PAM modules) are rewritten to take advantage of improved conversations rather than sharing global data: is this powerful enough? Would a PAM replacement built along these lines be able to do, over time, everything that PAM currently does - or at least, everything that you, as a client of PAM in OpenSSH, require from it? Or is there a system that I have missed and that absolutely needs to be addressed? I will stop here for now, thanks for reading all this already 😅 And if the project happens, it won't be before the summer of next year at the very least, so, absolutely no rush - but I am interested in your insights. And this is of course also valid for other PAM users out there, if this thread reaches them.

Open ##5015275