FastFed with Darin McAdams

Vittorio Bertocci: Bongiorno, everybody, and welcome. This is Identity, Unlocked, and I'm your host, Vittorio Bertocci. Identity, Unlocked as a podcast that discusses identity specifications, and trends from a developer perspective. Identity, Unlocked is powered by Auth0. The season is sponsored by the OpenID Foundation. In this episode, we focus on Fast Federation working group, FastFed for short in the OpenID foundation. And today we are chatting with Darin McAdams, software engineer at AWS and the author of all of the specifications in the working group. Welcome, Darin.

Darin McAdams: Hi, Vittorio, glad to be here.

Vittorio Bertocci: Thanks for joining me today. And as is tradition, let's start with how you ended up working in identity.

Darin McAdams: Common answer is by accident. And so, I'll explain my accident. So I have like an Amazon life, or I've been there since 1999 and people often ask me, what was your first team at Amazon? And I'll think, I'll be like I was on the team. You know, it was the team that did like ordering and the catalog and customer service tools and, A to Z. But what happens, like with all companies, is you get bigger and teams specialize and you fragment. And so I think I went from that to, I was on the ordering team and then that specialized further, I was on identity doing customer login to the amazon. com website. And of course specialized even further where at a certain point I was just doing addresses. Address books was what we did. But yeah, that's how I, I ended up there and then I, I sort of switched from there to Amazon's own internal sort of corporate identity and access management. So I got a flavor of that. And then about four years ago, I joined AWS side of the Amazon house where I work in AWS identity. So I guess if one reorg had gone a different direction, I would be on a podcast right now talking about order fulfillment optimization or something, but, identity... I've really enjoyed it. It's a lot of fun.

Vittorio Bertocci: Fantastic. And on behalf of identity community, we are all very grateful that you are working there because the work you're doing, the one that you're going to talk about today, is really, really important. So thank you for that introduction. It was really interesting. And that let's jump into the main topic. The main topic is FastFed. So what is FastFed, and what is the problem that we are trying to solve with it?

Darin McAdams: The one second pitch is like," takes Federation set up from weeks to minutes" is what we often say. I'll tell you how I figured it out. Cause it took me a while when I was first starting here to figure out what the problem was either. So Dick Hart, who I think you had on this podcast before, Dick's great. He actually kicked off the FastFed working group and Dick and I were on the same team at the time. And he was looking for an owner to take on editing and drafting the spec. And so he zeroed in on me and he said, I think you should do this. And to be honest, I didn't get it. I'm like, I read the charter. I know some standards; just, someone writes a draft and it sits there and they go nowhere. I'm like, do I want to waste my time on this? Dick's like, ono, just go try to set up Federation. Like I'd never done it before. I think this was just after I joined this particular team, I'd never done it. He's like, just go try to set it up. He nagged me and nagged me. So I'm like, okay. So I went to Google G Suite as it was called at the time, like just created a directory with a few users and I'm like, I'm going to set up single sign- on to, I picked AWS in Salesforce. And so I'm clicking buttons in the UI trying to find the right place to start. And eventually I get to the page to set up an app. And I think the very first thing it said is like... there is a form field and it said," please paste your entity ID and ACS URL." And at that point, the needle just scratches on the record. Cause I'm like, what? So first line, first page, I'm already stuck. So clearly I have some learning to do. SAML was the technology that was being used. So I'm like, well, okay, I'll go read the SAML spec. And so I paused that and I downloaded the SAML spec and I start reading and I'm reading and reading and I'm getting more and more confused. And that's when I realized the SAML spec is several hundred pages if you add it all up, I start skimming. And I come out of that, like Dick, I'm even more confused. Like I just don't get any of this. He's like, what did you do? I'm like, Oh, I read the SAML spec and he's just started laughing. He's like, nobody reads the SAML spec. I didn't know that at the time. Apparently there's only like one tenth of it that actually gets used in practice. So he's like, just follow the directions. So I find these directions, credit to Google, they had pretty good instructions. They're different ones for AWS, different ones for Salesforce. They'd be 42 steps long. They looked like a pre- flight checklist for launching an airplane. So I have this list up on one screen and it says, go to the app and look for this thing on the page and copy it and paste it to this box on this other page. And I'm like checking off these one by one. I don't really understand what I'm doing. I go through it. There's a point where you have to say like, what attributes should you convey? And like it had its own domain specific language for attribute mapping, because if you're sending the email, people want them spelled or capitalized different ways. I get through all that. I have few errors, had to figured out the steps of... I missed step 37. So I had to go back and fix that. I got AWS eventually working. I never got Salesforce working. And if this was the real world, I'd be on a call to their tech support and they'd bounce me back and forth to Google, probably. And maybe a week or two later, we might get it working. And then what I figured out is, I didn't even know this until later, but if I did get it working, it would fail a year later when their certificates, which are behind the scenes there in SAML expire.

Vittorio Bertocci: You really went all in and felt the pain of the people that are supposed to benefit from. That's great. That's fantastic.

Darin McAdams: Yeah. So I came back to Dick and I'm like, Dick, we have to solve this. Why has nobody solved this? And he's like, yep. Now you get it. So if you work at like a large organization and you do single sign- on to things like Salesforce or whatever, you owe a thank you to someone in your IT department who is setting up all this stuff. It's not fun. And that's what we're trying to go after.

Vittorio Bertocci: Nice stuff. So you describe beautifully, the challenges that traditionally one have when you try to set up a Federation. And what you're trying to do is to spare people from that pain so that Federations can be established without you going through that pilgrimage of pain.

Darin McAdams: Exactly. Why isn't this like on any consumer facing app or, login with Google, login with Facebook, click a button. They don't understand OpenID Connect when they're clicking that button. Like why can't we have that? And so that, that push button experience is what we're going after.

Vittorio Bertocci: So now we understand that what is the problem we trying to solve. So let's think for a moment about the how. Can you tell me, what are you doing in FastFed? What things are you describing that will make this problem better?

Darin McAdams: I might go through the journey we went through with the group, because I think it's a question maybe a folks might have when they're listening too, it's a question that came up a bunch, which is why we have to do anything at all. I mentioned login with Google, log in with Facebook, like that stuff exists at works. Why aren't we done? And what we found out was that, enterprise is a little bit different. It's some of the things that make it different, there's more standards in the mix. Some people using SAML, OpenID Connect, SCIM for provisioning. If you're building an app that's serving an enterprise audience, you're looking at all these standards and you're like, which ones do I implement? How do they work together? If someone's authenticating with SAML, but doing user directory synchronization with this other standard SCIM, how do they play together? How do I even know whether to do SAML or OpenID Connect? And, as I mentioned, SAML is 800 pages. So like which subset? And so there's just a whole bunch of confusion that resulted in every app kind of doing it differently. And so there's pretty much no interop to the large part. But in addition, enterprises have some differences from just a single individual clicking login with Google.

Darin McAdams: What is that?

Darin McAdams: Login with Google or login with Facebook, presumes that somebody knows they have a Google or Facebook account, which you can pretty much assume. If you go to anyone who works at a large organization or an enterprise, just pull someone in and be like, who's your identity provider? How do you sign in? You'd get a blank expression. And you'd be like, I don't do use Active Directory. Do you use Okta? Azure? Ping? It would just get more blank. They don't even know how they log in. Some other differences: you're setting it up, not just for yourself. You're connecting an entire organization. And now your security team is probably waking up going like, Oh wait, what? You're going to let everyone in the company put corporate data into some system and you're giving them employee directory like, whoa, I want to look at this one. So there's reviews that happen. And you have to handle that. I think the other big difference was the fact that... The fancy word we often say is there's no pre- established trust. But what that means is the app and the corporate directory have really no awareness of each other. And that's different than like login with Google. Like, I don't know if anyone's ever set up login with Google, but you do that. And what, you know, what you're doing as the app developers, before you had that button on your website, you've gone to Google, you filled out some forms, you've registered, Google gives you some stuff. Now you and Google know about each other. And they probably even give you a library to put the button on your page. And so that's sort of like pre established relationship, pre- established trust. And you get into enterprise and that's impossible to do. How many thousands, millions of enterprises, a lot of them just running their own LDAP or Active Directory and Federation server, like an app like Salesforce can not possibly register or even know about all these in advance. And similarly a lot of the corporate folks don't know about all the apps they want to use in advance. It's just so, so many of them it's. We called it bootstrapping from zero trust. How do you get these two parties who have never talked to each other or even know each other and be like, yeah, we're going to trust each other for sign in or whatever. So that was one of the differences. And one of the other things we really had to tackle.

Vittorio Bertocci: So all of these things point to a lot of options, possibilities, and you don't know in advance whether the parties chose or... So you mentioned the attributes, which is another area where everyone has their own vocabularies and similar. So you brought order and method to this chaos. So ways of declaring some of those options so that you have not left guessing, but you are restricting their space of possibilities. So how did you guys do it? What aspects did you describe? What processes did you introduce?

Darin McAdams: The logical problems we realized we had to solve was, how did the computers find each other? Cause in a way, like our ultimate goal was that we don't want a human being copying and pasting 42 steps, that configuration back and forth. How do we say like, Hey, computer one, meet computer two and go ahead and exchange the information you need to exchange. And don't make me copy and paste 42 different things as a human being. Humans are the worst possible data bus you could ever pick to convey data between two computers, which is why it takes several weeks to set up. Cause we always screw it up. So you look at that and you're like, what are the logical questions? Well, how did the computers find each other? What's the URLs? How did they talk to each other? How do they establish trust that this thing's allowed to talk to this thing? And so there was a bit of a handshake there. And then we often talk about FastFed is our carrier frequency for configuration data. So you're connecting two parties. You're telling them they can trust each other to exchange information. And then there's... JSON is what it comes down to in formats that we sort of prescribe, but to extensible to say, Oh, here's how you can sort of declare... One party can declare what I do. I do SAML or OpenID Connect or SKIM. The other party can declare the same. And then they can sort of exchange a negotiation over that carrier frequency about cool. I think we both speak the same language. So let's agree to do SAML and turn on SCIM. And here's some of the end points you need to do that. So it's that discovery, that carrier frequency for configuration and, and that contract negotiation, what can we both do? And let's go set it up. But in the end, the end result is exactly what a human being would have copied and pasted. We're just letting the computers copy and paste between each other.

Vittorio Bertocci: SAML has metadata already, right? What is different with using FastFed that is using directly their metadata in SAML and the SCIM API is for doing provisioning? All of that stuff.

Darin McAdams: We definitely did not want to reinvent SAML metadata or OpenID Connect Dynamic Registration. Like there's a lot of good stuff there. Don't throw that out, but we keep coming back. It's the bootstrapping problem, which is how do I know this app does SAML or this IDP support SAML versus OpenID Connect. And if they do, how do I find their SAML metadata file? Is it hosted on the internet somewhere? Or what's the URL? How do I get permissions to retrieve it? So that sort of bootstrapping is what you need to do. And, and if you actually look at what's going on in the wire, you'll eventually just see, if you're SAML, for example, one of the messages that goes across as the parties just say, Oh, here's my SAML metadata file. You can just go read it now.

Vittorio Bertocci: I see. So do you have metadata?

Darin McAdams: It is. I mean, it's one of the funny things because," why do we need this?" is a question that comes up all the time? But if you and I asked myself, when I was looking through some of the specs, there's OpenID Connect Dynamic Registration, aren't we done? And so I'm reading it and there's the fundamental question that, which is how do you know that a certain app is allowed to actually do the Dynamic Registration with a certain organizational directory? You're reading it like a mystery novel being like, I can't wait to read this part of the spec where it explains how that happens. And you get to the last page and it's like, this is an implementation detail. That's out of scope. So, I'll say, Oh yeah, magic happens. Somehow. Don't ask me. That was our problem. If we want a push button experience, we can't just pin it to magic. The buck stopped with FastFed, and so we had to figure out how to actually solve all that.

Vittorio Bertocci: Yeah. Which is a bigger responsibility because getting a consensus in a standard body is really hard. Like very often most standards are under specified, because it's a failure of reaching consensus. Whereas in here you had to get to push button. So you had to make some choices, right?

Darin McAdams: Yeah. It was interesting. You mentioned under specified, once you set a goal of, this is a push button experience and the user pushing the button doesn't know a single thing about any of these standards. That was really our goalpost. You've run into a lot of issues with the fact that the things are under specs are under or over scoped in some way. Under one, like SAML certificate rotation isn't described anywhere in the SAML specs. But if we didn't make that work, people would press button set these things up and in a year would fail and then they have no idea why it failed and suddenly they'd have to become SAML experts again. So we had to sort of extend the spec in a way. Sometimes we had to diff profiles, SCIM wasn't one, for example, where everyone had turned out, had implemented SCIM provisioning differently because the spec is just really generic. And so when people were connecting our products together, it never worked. So I was like, obviously that's not a push button experience. So that's one we're on the other end of the spectrum expected a lot of things. And we had to profile down and say like, if you're doing FastFed, this is the way you do it. And we just know it'll work,

Vittorio Bertocci: SCIM has this thing about the... I unfortunately stumbled upon in spec I'm leading, which is the fact that groups, roles and entitlements don't have a fixed schema. Like they have some advice, but then ultimately avoid implementation decides. All you know is they are lists, but then the shape or the entity is not defined. So I guess what you guys had to dig into that as well. And on the SCIM stuff: again, I'm very ignorant on the spec because I just wanted to make sure to represent an audience that doesn't know. So I didn't really get any time recently, so I might be saying something silly, but I have a vague recollection that there you can actually declare the attributes that you wanted to exchange versus is using SCIM directly. So that's a big acceleration, right?

Darin McAdams: Yeah. I mean, SCIM lets you... Maybe we'll explain what SCIM is in case people haven't looked at it. So you'll hear the acronym come up a bunch for synchronizing users from one directory to another. So just to pick an example, if you're like at Okta or Azure ID or Ping or... There's many of these identity providers. And they have your directory and then you want to use Salesforce, but you want to sort of keep the list of users that are in Salesforce, in sync with the list of users that are in your corporate directory on an ongoing basis. And so SCIM is that technology that sort of synchronizing the directory and the way you say. The interesting thing though, is that if you sit down and read the spec, this is what I went through, I sat down and I'm looking for the section called synchronization and there isn't one, like my Ctrl + F search came up empty and I'm like, that's interesting. So then I started reading it and the spec itself is actually pretty cool, but it's modeled almost like a modern LDAP where it's got APIs, it's CRUD operations, put users, get search, delete. Basic. That kind of stuff. And you can also, I think, Vittorio, this is what you were saying you could declare users. There's a space where you can declare your schema. This is how I define a user. Now it's cool, for the most part. They've also added a core shared definition of what a user is and what an enterprise user is. So almost everyone uses that. The challenges though, are one: how do you build a recurring synchronization mechanism with CRUD APIs? And it turns out everyone made up their own way. So, that's what we had to sort of rationalize. And the other one is, when it comes to that schema and declaring what, you know, what attributes you want, everyone just pretty much says like, Oh, I speak to the core user schema, but it's huge. And nobody really uses that to shrink it down and say like," of this subset, here's the two I need and the four that are optional and I don't understand the rest. So don't send them to me." That's another thing we had to solve.

Vittorio Bertocci: So more profiling, more scoping done. That makes complete sense.

Darin McAdams: Yeah. I mean, one of the ways a lot of these SCIM setups fail today is, an app might mandate that you give a display name... that's one of the fields in SCIM. But if you don't send it, the person can't log in. How do you know that you got to send the display name? And so that's, that's the kind of challenge.

Vittorio Bertocci: From the nature of the problem, it seems like you had to solve a long list of tactical issues, which on paper, on the white board, it seems solved, but if you actually want to make it happen in the world, then you had to. So I'm curious if there is something particularly challenging, something really surprising that, maybe you tried an approach, and then you had to change course. Some" at war" story of your journey in trying to develop these standards.

Darin McAdams: We definitely went through some iterations just trying to figure out where does this journey start? So at some point someone says, aha, I want to set up single sign- on, probably cause their security department just yelled at them to set up single sign- on. So they're like, well, what do I do? And so someone's there trying to press a button. And so when you, one of the first questions was, well, where do they start? Do they start in the app? Do they start in the identity provider? We went back and forth and we actually had one attempt that started at the identity provider and it just didn't work. Missed some use cases. It was really kludgy. So we ended up starting at the app, ended up making a lot more sense. So, that was one where we had a few fits and starts. Interesting, one of the things we worried about a lot that may not be quickly jump out, was we worried about making it too easy, which isn't that intuitive, but you know, there's some security consequences that could flow if we make it too easy. And you can imagine, maybe I'll explain this a bit, Vittorio. Let's say you have your own Salesforce instance for your company and I'm a bad person. And I would love to poke around in there and find all your sales leads or whatever. How would I get in? Well, there's hard ways. I can try to compromise one of your employees passwords or their workstation. That's a lot of work. It would be a lot easier if I could just trick you, Vittorio, as the administrator of that Salesforce instance to set up my own directory as a valid way to log into your Salesforce instance. And then I can just log in with my own password. That would be, that would be way easier. Now, of course today, when it takes two weeks and a call to tech support and 42 steps to set up Federation, kudos to anyone who could trick anyone into doing that, your powers of persuasion are amazing and fearful, but it's never really been a thing that people worried about. But of course now, it's pushed by it. Could I trick Vittorio into a pushing a button that would let me sign into a Salesforce? We had to think about that a bunch. And to be honest, it gets into UX, risk scoring, things like that. I think when we were first doing the prototypes, we realized we zipped through the technical backend implementation pretty fast. And then we got to the UX and we're like, how do we make this safe and intuitive? And there it's like, oh that's where all the time is. I mean, we still think of best practices once we figure this all out for ourselves, some best practices, guidance on that.

Vittorio Bertocci: That's really interesting. It's such a counter intuitive thing. Like we fight the complexity all the time, but you are, you raise an excellent point. If you lower every bar too much, you might be adding to the attack surface. Now I'm going to throw you a curveball. I was recently talking with someone who works in decentralized identity and, as usual I was challenging them saying, what is this thing good for? And one of the answers I got was, well, you could get away with all of these setting Federation up where everyone would be able to just use that VC, like verify credentials and just without any setup. And to which, my reaction was like, well, but people do want to do that setup because there are things that the administrators wants to decide and streamline, so it is overhead, but it also plays a function. So I wanted to hear if you ever had a similar discussion, if what are your thoughts about it?

Darin McAdams: Yeah. Just make sure I get the scenario. Are you thinking about maybe Vittorio, you wanted to use Salesforce and it was your first time going there- that from the very first moment, when you clicked sign up, that you could only enter like your own corporate creds and do single sign on from the moment of sign up, is that kind of what they were thinking?

Vittorio Bertocci: The idea is that when you use the these decentralized identities, you get the these special kind of credentials that you can as a user, you can use it without the identity provider knowing, and then yeah, it's back there. These credential brings with it a number of attributes that you got from your provider and then you can choose to use, but it's kind of like a, your driving license that you can pull out and use without the department of driving license, knowing that you are using it. And then like someone can decide to accept that credential. But at that point you don't have much choice in term of negotiating these attributes, you're get the attributes that you found. So to me, it seems a bit odd to put it in the context of Federation, because Federation, typically you decide what's your scope with the, what are the things you want to get. Whereas in that case, it will get what shows.

Darin McAdams: So it's that decentralized identity kind of discussion. Yeah, it's interesting. I think we never explicitly said this, but maybe people might've gathered from the context is FastFed is definitely targeting first and foremost, the enterprise audience, it has its own unique things. One of the things that is unique about the enterprises... Security teams at an enterprise will want the opposite of decentralized. They want a centralized as you can get, they want to know everyone who's in the directory. They want to know what they're doing. They want to tightly control all the attributes on your employees. I don't get to choose, I wish I could choose what my salary and job level was, but you know, someone else controls that. If I were to leave the company, they want to make sure I stop accessing those resources for the company as quickly as possible. And so, maybe I just haven't followed as closely, but I think there's, I haven't seen the corporate enterprise community talking as much about decentralized identities.

Vittorio Bertocci: Thank you. I know it was a big side detour, but I was just curious to see if you heard anything about it. Fantastic. So this has been something that you guys have been working on for some time. So, where are we? Are we close to this being a standard, is there adoption out there? What's the situation?

Darin McAdams: On the internet? You'll see some demos that we did before, but where we are with the spec is... we just recently got it to what's called implementers draft. It really implies this spec is stable enough that you can start putting some money behind the implementation, like putting it into roadmaps, putting some engineers down on the building, some things. And we'll probably learn a few more things in that process and catch a few bugs in there, but it's really expected to be pretty stable and ready to implement, hence the implementers draft. We just had a recent worker group meeting and you've recognized a lot of the names in there. And we were all trying to figure out what our roadmaps are. Who's going to start on the app side, who's going to on the identity provider side. We're trying to figure out when, we've all sort of penciled into our roadmaps, it's 2020, 2021 and we've lost all meaning of space and time. So normally we would be like, let's pick a major conference and work backwards from that and doing an announcement there. And now everyone's like, I don't even know what day of the week it is. I think we're just circling around, like, what do we want to actually put as a milestone, but it's definitely starting implementation. There's an open source implementation, a Java one that's in progress out there for the most part we would expect people would look at that and not go read this spec and reimplement everything from scratch.

Vittorio Bertocci: If you were to issue a call for action about this, what would you want the listeners to do?

Darin McAdams: Maybe, for the engineers out there? Certainly if you're curious, if you want to join the group or look at the spec, it's all out there. I think if you're maybe just as a customer, almost saying I'm tired of setting up, spending two weeks and, and pulling my hair out, setting this stuff, like when do I get FastFed, that's one where you should really just reach out to, if you have an account manager or whatever the right channel is to submit the feature requests, whether it's to the app or to the identity provider. Those matter, how often we're hearing from people about I want X, Y, or Z is really how people decide whether to focus on X, Y, or Z. So I think that's the other thing, if this sounds interesting and you want it, ask for it.

Vittorio Bertocci: Makes a lot of sense. And then we add the links to the show notes. So if any of these is actionable, we make sure that people have the right tools to get there. Well, these was extraordinarily interesting as I knew it would be. So thank you so much for being with us today and walking us through FastFed. You'll get a lot of people reaching out because it truly does solve an important problem. So I'm looking forward to see this more adopted.

Darin McAdams: Thanks, Vittorio. Yeah, it was fun to do this.

Vittorio Bertocci: Thank you. And thanks everyone for tuning in. Until next time. The OpenID Foundation is a proud sponsor of the Identity, Unlocked podcast. Since its formation in 2007, the foundation has committed to promoting, protecting, and advancing the OpenID community and technologies. Please consider joining the foundation and contributing to current working groups. To learn more about the OIDF, please visit www. openid. net. Thanks everyone for listening. Subscribe to our podcast on your favorite app or at identityunlocked. com until next time, I'm Vittorio Bertocci, and this is Identity, Unlocked. Music for this podcast composed and performed by Marcello Wolovsky. Identity, Unlocked is powered by Auth0, copyright 2020, Auth0 Incorporated. All rights reserved.