"We've Disabled MFA for You": An Evaluation of the Security and Usability of Multi-Factor Authentication Recovery Deployments

Multi-Factor Authentication is intended to strengthen the security of password-based authentication by adding another factor, such as hardware tokens or one-time passwords using mobile apps. However, this increased authentication security comes with potential drawbacks that can lead to account and asset loss. If users lose access to their additional authentication factors for any reason, they will be locked out of their accounts. Consequently, services that provide Multi-Factor Authentication should deploy procedures to allow their users to recover from losing access to their additional factor that are both secure and easy-to-use. In this work, we investigate the security and user experience of Multi-Factor Authentication recovery procedures, and compare their deployment to descriptions on help and support pages. We first evaluate the official help and support pages of 1,303 websites that provide Multi-Factor Authentication and collect documented information about their recovery procedures. Second, we select a subset of 71 websites, create accounts, set up Multi-Factor Authentication, and perform an in-depth investigation of their recovery procedure security and user experience. We find that many websites deploy insecure Multi-Factor Authentication recovery procedures and allowed us to circumvent and disable Multi-Factor Authentication when having access to the accounts' associated email addresses. Furthermore, we commonly observed discrepancies between our in-depth analysis and the official help and support pages, implying that information meant to aid users is often either incorrect or outdated. Based on our findings, we provide recommendations for best practices regarding Multi-Factor Authentication recovery.


INTRODUCTION
Username and password are the de facto standard for user authentication on the web and beyond.However, they come with many security problems: Users tend to choose easily guessable passwords and re-use them for multiple accounts [18,69,87,89,90], or suffer from insecure or hard-to-use password policies [35,52,84].Additionally, service providers may implement insecure and inadequate password storage, leaving millions of passwords unprotected due to data breaches [17,29,32,70,72,95].Multi-Factor Authentication (MFA) adds an extra factor and additional security to passwordbased authentication schemes, and has become important in many authentication deployments on the web.With MFA, users authenticate themselves using additional factors, e. g., biometric features, hardware tokens, smartphone applications, or secret information only they know [67].
Prior work has found tensions between security and usability in many MFA implementations, revealing that the complexity of MFA setup is a barrier to widespread use [74,75,76], and user acceptance of long-term MFA use can suffer due to the increased log-in and setup time [6,31].While most prior work focused on MFA methods and deployment, recovery procedures for loss of MFA are less well understood.Hence, in this paper, we focus on this critical aspect of MFA security and usability: We investigate the security and usability of MFA recovery procedure deployments on the web.Recovery procedures are crucial to regaining access to accounts in case MFA factors are lost, stolen, or broken.The deployment of a recovery procedure by service providers has a significant impact on authentication security and usability.While locking users out of an account after losing access to MFA factors keeps up security assumptions of MFA, it might contribute to users' frustration [61], users leaving a service, or even avoiding MFA in the future.However, allowing users to access their accounts after losing their MFA factors by returning to password-based authentication significantly decreases security, allowing attackers to more easily gain access to user accounts than with MFA.While security and usability seem to be directly at odds in many current implementations of MFA recovery, it is essential for service providers to relieve this tension through better communication, and policy and technical changes in order to provide both security and usability to support users.
We aim to better understand the experience of users who have lost access to their MFA, and systematically evaluate MFA deployment and recovery, asking the following research questions: RQ1: "What MFA recovery procedures are commonly deployed on the web?" Due to their crucial impact on authentication security and usability, the variety of deployed MFA recovery procedures need to be better understood in the research community.We investigate the deployment and documentation of common MFA recovery procedures on the web.RQ2: "How are MFA recovery procedures on the web implemented and what is their impact on authentication security?"MFA recovery procedures aim to support users getting back into their accounts in case of losing access to MFA factors.We investigate how service providers balance the usability and security of MFA recovery procedures.RQ3: "How can the security and usability of MFA recovery procedures be improved?"Standards or established best practices for MFA recovery procedures are missing.Based on our results, we make recommendations to balance the security and usability of MFA recovery procedures for future deployments.First, we created a list of 1,303 websites offering MFA.We systematically evaluated MFA recovery procedures deployed on these websites using their official help and support documentation.In a follow-up ERB-approved study, we performed an in-depth analysis of the security and usability of deployed MFA recovery procedures for a subset of 71 websites.We created accounts, enabled and configured MFA, and went through the entire recovery process.
In this paper, we make the following contributions: • MFA Recovery Procedure Deployment: By analyzing official help and support pages, and FAQs, we find that most websites only offer one recovery procedure, which is most commonly a backup code or direct support contact.A quarter of pages do not mention recovery in their help.• User Experience of MFA Recovery Procedures: We perform an in-depth investigation of the user experience of MFA recovery procedures on 71 websites.We could recover 52.11% of the accounts.In most cases, access to the associated email inbox was sufficient.• Recommendations: Based on our results, we provide recommendations for future research on secure and usable MFA recovery procedures.We also provide guidance for web developers to help deploy easier-to-use and secure MFA recovery procedures, spanning all areas of recovery such as setup, communication, and recovery itself.• Replication: We provide all material including communication templates, the study protocols and our codebooks to support methodological transparency, replicability, and validity, and to help future research on this topic as part of a replication package 1 .This work is structured as follows.In Section 2, we describe previous work on the subject.We present the methodology and results of our deployment evaluation in Section 3.1, followed by our practical recovery test in Section 3.2.In Section 4, we elaborate on the ethics and limitations of our work.We discuss our findings in Section 5, and conclude in Section 6.

RELATED WORK
In this section, we present previous work on related areas and discuss our novelty and contributions compared to them.We first showcase related papers regarding MFA security and usability, then discuss research on the recovery of account access regarding both single-factor authentication and recovery of additional factors.

Multi-Factor Authentication Usability
Several previous works have investigated the usability of different popular MFA methods.
In 2018, Colnago et al. observed the enrollment of Duo twofactor authentication at their university.While adopters found it annoying, they also perceived it as easy to use and secure [25].In the same year, Reynolds et al. conducted two studies with Yubikeys, asking participants to set them up or to include them into their daily lives.Despite participants embracing hardware keys, the study uncovered severe usability problems, especially during setup [76].In 2019, Reese et al. conducted a usability study with 72 participants comparing five different 2FA methods, confirming findings by Reynolds et al. [74].
Ciolino et al. also conducted both a lab and longitudinal study comparing SMS OTP and hardware keys, and similar to previous works reported that SMS is both faster to use and set up, and that hardware keys are perceived as less usable [24].In 2020, Reynolds et al. evaluated authentication data sets from two universities that introduced 2FA, finding that more than five percent of authentication attempts failed, mostly due to technical difficulties such as timeouts, or users canceling or mistyping the codes [75].Similarly, Abbott and Patil examined authentication logs and surveyed university users after 2FA became mandatory for them.They found that user acceptance was not influenced when 2FA was required for some sensitive authentication purposes, but decreased when it was enforced for every login [6].Also in 2020, Farke et al. accompanied the introduction of FIDO2 keys as a single-factor (i.e., a password-less login, not a MFA method) in a small company, finding that while participants considered them usable, they stopped using them due to concerns such as login efficiency or missing browser support [31].
A more recent work by Lyastani et al. evaluated 85 websites regarding their MFA usability by creating accounts and evaluating their MFA communication and settings.They found that MFA implementations are largely inconsistent between different websites, and that many designs have previously been identified as obstructive or problematic [36].
Overall, previous work shows that the usability depends on the MFA method used, e. g., SMS OTPs are often faster and more usable than hardware keys, although they are known to be less secure.Additionally, users commonly struggle with setup and continued usage, obstructing widespread voluntary adoption of MFA.While our work does not investigate user sentiments, we discuss the user experience of MFA methods and recovery procedures, and how website and service providers communicate MFA and its security benefits for online security.

Account Recovery
While not many previous works have discussed the recovery of inaccessible MFAs, several have worked on account recovery in general, and its potential obstacles.
One of the currently still popular forms of account recovery, security questions, was criticized as early as 2008 when Rabkin addressed the issue of using security questions to regain account access.The work highlighted several issues, including how questions ask for easily obtainable knowledge, especially with the rise of social media [71].A similar work by Schechter et al. in 2009 confirmed this, finding that in 17% of attempts, user acquaintances were able to guess the answer to security questions, 13% were easily guessable within the first five attempts in general.Further issues included usability problems, as 20% of users forgot the answers themselves six months after setting them up [77].The usability of security questions was further researched by Bonneau et al. in 2015 in a survey.They found that 37% of participants lied when answering security questions, and 40% did not remember their answers when requiring them to recover their account [19].
In 2006, Brainard et al. proposed a novel approach for account and MFA recovery, in which a trusted person can vouch for account owners and help them regain access.They further discussed limitations and variants, like additional measures to prevent users from abusing this recovery method as its own MFA [21].A similar approach was investigated in 2009 through a lab study by Schechter et al. [78].In their study, 17 of 19 participants successfully used trusted individuals, however, similar to the answers to security questions, some users forgot who their trusted individual was.
In 2015, Hang et al. researched fallback authentication methods for smartphones, finding that current methods were largely sufficient, but that a minority of users struggled with it due to a lack of knowledge or alternative authentication options [42].While most of these studies examined specific account recovery procedures, Neil et al. evaluated real-world account recovery advice on 57 popular websites.Their results indicated that help sections were often Compare documented MFA recovery procedures with our experiences in the in-depth recovery procedure evaluation in step 2. incomplete and that 39% of websites were failing to address the topic at all [65].
In 2021, Kunke et al. evaluated twelve account recovery procedures for passwordless authentication with FIDO2 regarding different criteria within the areas of usability, deployability, and security [48].They found the recoveries to be lacking and, in some cases, such as security questions and backup passwords, jeopardize the idea of passwordless authentication.In 2023, Gilsenan et al. performed a usability analysis of 22 popular TOTP 2FA mobile apps regarding their backup implementations, finding that most rely on less secure methods such as SMS or email for backups [37].
Furthermore, a recent study by Gerlitz et al. has also investigated how websites act when additional factors are lost.They were able to recover about half of their created accounts, and find that there are no best practices for recovery behavior or communication [34].
To summarize, previous work has evaluated the security, usability, and communication of fallback authentication methods.However, this typically happened in the context of account recovery, and with no regard for MFA loss.In contrast, we evaluate existing MFA recovery procedures by experiencing them first-hand, with special regard to its usability and (loss of) security benefits.

MFA RECOVERY PROCEDURE ANALYSIS
This section presents the method and findings of our analysis of deployed MFA recovery procedures.Our analysis spans two parts; we first investigated deployed MFA recovery procedures for 1,303 websites using their publicly available help and support pages.Second, we performed an in-depth analysis of MFA recovery procedures for a subset of 71 websites.We simulated a user's experience after losing access to their MFA factor.See Figure 1 for a summary of all steps in our methodology.

Deployment of MFA Recovery Procedures
First, we conducted a systematic evaluation of MFA recovery procedures deployed on 1,303 websites that use MFA, based on the information provided on help and support pages.Overall, we investigate the impact of MFA recovery procedures on authentication security and usability.
3.1.1Methodology.We aimed to simulate users who lost access to their MFA factor and tried to find information on how to recover access to their accounts.While our overall focus was on the users' perspective on MFA recovery procedures instead of a measurement of deployed MFA methods on the web, we provide some context on the spread of MFA methods and recovery procedures in the following.
Our dataset included all 1,303 websites with MFA deployments from 2f a.directory [2], a high-quality crowdsourced collection of websites with information about MFA deployment on the web.Most websites included on 2f a.directory offer services to users, and typically collect user-connected information, e. g., addresses for online shopping, or financial information for online banking.The 2fa.directory excludes self-hosted websites and websites that are not within the top 200,000 according to SimilarWeb [5].We verified the rank requirements by cross-checking our sample with the Tranco list [50] 2 , and found 1,173 (90.02%) of the websites in our dataset within the top 200,000 websites, of which 692 (53.11%) were even within the top 20,000 (cf.Overall, we found 2,021 websites on 2fa.directory.From these, we excluded 44 duplicates that were included in multiple categories, 7 unreachable websites.We further only examined websites that were either in English or German and excluded 155 websites in other languages.Finally, 512 pages did not offer MFA and were therefore not relevant for our study.This gave us a total of 1,303 websites.We focused on German and English websites, since all authors were either proficient in English or German.We excluded other languages since in some cases automated translations of help and support pages in other languages were ambiguous in the MFA methods or recovery used, too vague, or inaccurate.While some websites provided official translations, they were not always wellmaintained or even outdated.We further removed 15 websites we could not reliably reach due to, e. g., geo-restrictions, or unavailable services. To evaluate help and support pages, we used the support links provided in the 2f a.directory database, and the websites' help and support sections that we could either find by manually going through the websites, or using Google with the following search term "<website> two-factor authentication"3 .Within our Google search, we followed all links that were officially affiliated with the website in question (i.e., no third-party news websites) within the first page of Google results [22,43,44].We visited all of them and examined them for input regarding MFA methods and recovery procedures.We followed all links and relevant prompts, including "Trouble logging in?" or "Forgot your MFA?" prompts.This did not include interactive processes to reset passwords, as we focused on lost MFA.
Two researchers analyzed the help and support pages for documented MFA recovery procedures using an inductive categorization approach [59].We used a semi-open coding approach and an initial codebook including the five MFA methods given on 2fa.directory and several initial recovery procedures based on personal knowledge and previous work [48,71].We extended or updated the codebook accordingly whenever we identified a novel MFA recovery procedure, and revisited previous codings.The full codebook and a more detailed description of MFA methods and recovery procedures can be found in the replication package in Table 4, and Section A in the Appendix respectively We resolved all coding conflicts as they came up, leading to a hypothetical agreement of 100%.In line with common practice and previous high-quality qualitative research, we, therefore, chose to omit the calculation of an inter-rater reliability score [20,60,62,92].
For the initial categorization, we coded for documented MFA and recovery procedures.We used counts, means, and other aggregation measures to report our findings and generate descriptive statistics.During our evaluation, we made notes of every recovery procedure that stood out, either positively or negatively.This includes constraints on MFA, e. g., if it was only provided for certain user groups or customers, interesting security-related statements, such as stressing that lost MFA could not be recovered in any way, or interesting methods, such as receiving recovery codes via mail.We used these notes as a basis for the detailed coding described in Section 3.2 and our best practice recommendations in Section 5.

Results
. Overall, mobile applications were the most popular MFA method, deployed on 1,036 pages (79.51%), mostly as TOTP generator apps.Second most popular were SMS (559, 42.90%).All other methods, i. e., phone calls, email, hardware tokens, and various other methods were noticeably less widespread and present in between 10.67%-15.73%(139-205) of our sample.On average, websites offered 1.75 (median: 1.0) MFA methods.We provide details for MFA methods and their recovery procedures in Table 1.
Websites offered an average of 1.28 (median: 1.0) different recovery procedures.Most websites used direct support contacts (R01) (491; 37.68%) and backup codes (R02) (451; 34.61%).However, 321 (24.64%) websites that deployed MFA did not provide publicly accessible information about MFA recovery procedures at all (R17).While some websites might provide MFA recovery information behind a login, we argue that users locked out due to MFA loss would benefit greatly from public recovery information.
While 179 (13.74%) websites offered MFA using hardware tokens, no website in our sample offered them as a recovery procedure.In most cases, users could only use hardware tokens with another MFA method -a requirement we did not find for other MFA options.While websites did not justify this decision, we suspect known usability issues of hardware tokens and a perceived higher likelihood of becoming inaccessible [24,31,57,76].
We compared MFA recovery procedures between website categories provided by the 2f a.directory categories, e. g., Banking, Table 1: Summary of deployed MFA methods and recovery procedures in our sample of 1,303 websites.For example, we find that of 559 websites using SMS (42.90% of all websites), 199 (35.6% of websites that offer SMS as main MFA) offer support contact as a recovery procedure.Because some websites offered multiple MFA and recovery procedures, not all percentages add to 100.We provide a more detailed description of these methods in our replication packageand in Section A in the Appendix Most websites used backup codes or direct support contact as their recovery procedure.A quarter of websites did not provide public MFA recovery information.

Recovery User Experience
The previous experiment sheds light on the deployment frequencies of different MFA recovery procedures, including recovery instructions for 1,303 websites supporting MFA.
In this section, we investigate the user experience of deployed MFA recovery procedures.
Therefore, we selected a subset of 71 websites, created accounts, and performed the entire MFA recovery procedure process.We focused on authentication security and usability and how well the implemented MFA recovery procedures were documented.

Methodology.
Since we were further interested in detailed but also diverse insights into MFA recovery procedures that cannot be achieved from collecting help and support pages alone, we selected a subset of 71 websites for an in-depth analysis of the user experience.Similar to previous work [37,52,65,68,81], we chose to comprise our sample from different sources, including 28 of the highest ranking, 27 random, and 16 hand-picked websites.We chose the 16 handpicked websites based on interesting edge cases in our larger dataset.This included 6 websites that offered security products such as SSL certificates or antivirus programs, and 6 websites that mentioned unusual recovery procedures.Examples of unusual recovery procedures were preventive measures such as enforcing downloads of recovery codes, offering users to store various types of personal data which would be later used to recover the account, or multiple tests of the availability of their chosen recovery.Similarly, we chose some websites based on their recovery procedures, including a website that involved team members as witnesses to vouch for or against a recovery decision or enforcing fixed waiting times before MFA would be reset.Finally, we chose 2 websites that explicitly mentioned that the account would be lost if users lose their MFA, including advice to buy multiple YubiKeys, and 2 websites with unclear or no instructions to see what would happen in these cases.
From an initial qualitative sample of several websites, sampled after our three criteria above, we excluded websites that required pre-existing contract or group memberships (25), or data we could not or did not want to provide to protect the privacy of the researchers conducting this study, e. g., credit or ID card data (23).We further excluded websites with terms of service that prohibited our use case (20).Other reasons included that the website shared its account with another website already in our sample (12), that we were blocked on due to unspecified reasons (11), or websites whose free versions were only time-limited trials (17).Finally, 3 domains forwarded us to other websites or were offline since the deployment evaluation.An anonymized overview of excluded websites is given in our replication package and in Table 7 in the Appendix B.2 Whenever we could not create accounts, we replaced a website with the next best alternative where possible-e.g., if a top-ranked website required us to have a contract with them to acquire an account such as a bank, we replaced it with the next most popular website not yet included.We ended up with a sample of 71 websites, on which we tested MFA recovery.Despite the exclusion of websites, we feel confident that our in-depth analysis of the actual MFA recovery process on 71 websites provides deep and diverse insights valuable to the research community.Additionally, the distribution of offered MFA methods in our final sample is comparable to the overall distribution on all 1,303 websites.
We created accounts on all 71 websites using free account plans, enabled MFA, and triggered the respective MFA recovery procedure.Using screencasts, we made video recordings of the entire procedure to compare it with the official documentation during later evaluation.To receive email and text messages for account creation, MFA setup, or MFA recovery, we used a Google email address created for the study and an eSIM with a US-based phone number.
During the MFA setup, we simulated a user who follows instructions but only fulfilled the minimum requirements, following findings from previous work [45,90,96].During the process, we noted all steps, whether we regained access, and how long recovery took.We also captured security and usability details.For each website, we followed the protocol below: (1) We enabled the first MFA method provided by a website.We set specific or additional MFA and recovery procedures if the website gave explicit instructions to do so.This decision was further driven by the assumption that the first listed MFA method and recovery procedure is preferred by a website and likely best supported.
(2) After one week had passed, we revisited the websites.This decreased suspicion because we did not immediately lose access after creating the accounts.
(3) We tried to log in without our supposedly lost MFA and recovery procedures.If the respective website allowed account recovery for lost or inaccessible MFA devices, we followed it without using our configured recovery procedure (i.e., we did not use our backup codes to regain access), to simulate the experience of a user who has lost MFA completely.(4) If a website did not provide an automated emergency solution for login without MFA or the recovery procedure we configured, we contacted the website's support directly.In our initial request(cf.Appendix B.1) we said that we recently lost a backpack containing our phone and wallet, and found ourselves without access to the configured MFA factor.(5) If there was no way to contact the website (e. g., because support channels were only available for paying account plans), we stopped, marked the account as lost, and continued with the next website.
For an in-depth analysis of MFA recovery procedure experience and comparison with their documentation, we qualitatively assessed the help and support pages of the 71 websites in our data set.We collected all relevant help and support pages for all websites: • We performed a Google search for "<website> two-factor authentication" 3 similar to the one for our deployment evaluation.• We collected the help and support sections linked in the 2fa.directory database.• We manually navigated websites, starting with their main pages and using their search functions if available.
Following the steps above, we saved screenshots of help pages and expanded our codebook from the initial categorization (see Table 4) We added details such as whether a website explicitly warned their users of account loss in case of losing access to their MFA, if MFA was required in general, and how often it needed to be entered, or whether users were required to set a recovery procedure (see Table 5 for the codebook expansion) We coded both the MFA account setup processes and the information provided on the help pages.We compared them to identify potential shortcomings or contradictions in how providers communicate MFA to their users.Our institution's ethical review board (ERB) approved the study, and we took measures to protect the websites' identities, their resources, and the time of their staff (cf.Section 4.1).

Results
. Below, we provide details for our MFA recovery procedures evaluation.Furthermore, we compare our user experience of MFA recovery procedures with their official MFA recovery documentation.
Account Recovery Success Rates.Overall, we created 71 accounts in August and September 2022.We regained access to 37 (52.11%), and lost access to 30 (42.25%).As of the time of submission, we did not receive an answer from 4 (5.63%)service providers.Figure 5 in the Appendix connects our website sample to MFA methods and recovery success.We configured mobile apps (49, 69.01%), SMS (15, 21.13%), email (6, 8.45%) and finally hardware token (1, 1.41%) as primary MFA methods.We found that websites that deployed SMS-based MFA had slightly worse recovery success rates than TOTP or other app-based approaches.As we retained access to our email inbox during our study, i. e., we never lost access to the main MFA method on the respective website, email had a perfect success rate.We decided not to further bias our results due to a change in methodology by treating websites with email as the main MFA differently from those that deployed other MFA methods.Overall, we set up at least one MFA recovery procedure on 50 (70.42%)websites.However, 21 (29.58%)either did not offer any MFA recovery option or did not prompt us to configure a recovery option during MFA setup.The recovery rates for all MFA recovery procedures are illustrated in Figure 2.
We configured and stored backup codes (R02) for 39 websites.Only 6 (15%) of those websites tried to verify backup code availability during setup, e. g., by asking for the code or confirming that we stored the code.We simulated having lost access to our backup codes for the recovery requests.We could regain access to only slightly more than half (22, 56.41%), and lost 16 accounts (41.03%).The remaining 1 website did not reply to our recovery request.
We were unable to regain access to any account that used TOTP apps or their seeds for MFA recovery (5, 100.0%).On 5 websites, we tried to recover our account using available, but unrelated procedures, i. e., alternative contact methods such as backup email (R06) or phone numbers (R05) we set up during account creation that were never officially dedicated as recovery or mentioned during MFA setup.Despite the vague communication, this led to indeed 4 (80.0%)accounts being recovered from MFA loss.For other recovery procedures, mostly consisting of phone-based (R05) recovery or the direct mention of support channels (R01) as a backup solution, we regained access to 2 (40.0%) accounts.Finally, we did not set up (R16) MFA recovery procedures on 21 websites, as this was not part of the MFA setup process.Despite the lack of a recovery procedure, we could regain 10 (47.62%) accounts.Overall, we configured MFA recovery for 54 accounts.We successfully regained access to 27 (50%), and lost access to 23 (42.59%) accounts.We received no answer from 4 (7.41%).
Account Recovery User Experience.During our user experience, we found MFA recovery implementations to differ vastly, e. g., websites offering different MFA or recovery procedures, or some including the recovery procedure configuration in MFA setup while others do not.We provide details on the reasons for accepting or denying manual account recovery, finding similarly inconsistent and contradictory recovery philosophies, e. g., some websites argued that because our account was barely used, there was no harm in allowing us back in, while others argued that this meant there was no harm in us losing access and creating a new account.Our findings hereof are described in the following paragraphs and illustrated in Figure 3.Among the 30 accounts to which we did not regain access, 6 (8.45%) were because we could not contact the website's administrator or customer support team.Reasons included that on some websites, our free account plan was not eligible to receive any support, leaving us, and any user with a free account, no way to reach out and ask for help.Additionally, some websites required us to use contact forms for which we did not have all the necessary information, e. g., credit card information or banking transactions, IMEI phone identifiers, or information about the specific program from the vendor's offer we were using, the information we were not asked to provide or set up during account creation.Another website required us to provide our location and an order number, as the contact form assumed that only paying customers with questions or problems regarding their orders would reach out.Finally, one website had removed their contact information from their website, arguing that due to a high request volume, only specific user groups were allowed to contact them.We refrained from reaching out with a dishonest request.
Of the websites we were able to contact, 10 (14.08%) had strict policies regarding recovery and would not allow access without the configured MFA or recovery procedure.Finally, 13 (18.31%)websites required us to provide various sensitive and identifying documents, such as governmental ID cards or bank transaction receipts; when we refused, we lost access to these accounts.However, we successfully recovered 6 (8.45%) accounts due to being able to access our primary email MFA method and 10 (14.08%) due to being able to provide specific knowledge about the account during the recovery procedure.This encompassed metadata or contextual information, such as our organization name, our address, the account creation date, typical login locations as well as our ISP or IP.One website asked us a series of questions about our typical behavior and settings, many of which were potentially public knowledge, such as the date of our last broadcast or other connected platforms.We recovered one account by answering security questions that had been part of the account creation process.Three websites asked about potentially sensitive, personal, or identifying information such as, e. g., recent orders or payment information-which we could not provide as we never completed a transaction on the website-or use of an SSH shell or GitHub recovery-which we had never connected-but several websites allowed us to regain access despite failing these requirements.
For 4 (5.63%)accounts, we regained access immediately after our initial request with no further questions asked and no requirements or needing to go through any recovery process.In 3 (4.23%)cases, the recovery was time based.Here, MFA was disabled after a waiting period of 3-14 days.In these cases, our primary email account received multiple reminders and warnings, designed to give the legitimate account owners time to interrupt the recovery in case a malicious or unknown third party made the recovery request.
Overall, we found that most of our requests to regain access were fulfilled after just a few emails from our primary email account and that we rarely required more than contextual knowledge about either the account or us as the main user.Our findings suggest that the security of many deployed MFA recovery procedures is similar to that of the email account connected to the MFA we wished to recover, or that of security questions, as many requested data points were often easily guessable or public knowledge such as connected platforms or our companies address, or accessible via our email inbox such as account creation dates or ISPs.
Correspondence with Website Support Teams.The majority of accounts, to which we regained access, it was sufficient to have email access (26, 70.27%); this encompasses accounts that used email as their main MFA or recovery procedure, as well as accounts that asked us to confirm our request by email to disable MFA and regain access.While only confirmed by one website explicitly, we assume this was required as a form of phishing protection.
In two cases, websites typically used for commercial purposes tried to ask our account owner to allow them to disable MFA for us.As they assumed a hierarchy within the account -e.g., a dedicated main user, admin or owner, and several team members-but found us to be the only user, we received messages to the same email both as a user who applied for a MFA reset, and as account owner that can allow or deny this request.Hence, we were able to allow ourselves back into the account.While this might be an edge case of their recovery procedure, it might lead to a decreased security of important administrator roles in comparison to normal users, as this cancels the extra obstacle of requiring admin approval.
Overall, we set up a recovery procedure during the account and MFA setup for 50 (70.42%)websites.When we initially applied for account recovery (cf.Appendix B.1 for the message) we did not mention these recovery procedures, but only stressed that we lost our wallet and phone.While this would render most pre-set recoveries, such as backup codes, still usable (47, 87.04%),only a minority of 10 (14.08%) websites mentioned using the pre-set recovery procedure, with one of them disabling our MFA without further requirements after we responded that we lost the backup codes as well.10 (14.08%) websites insisted on only allowing us access if we used either the original MFA or the recovery procedure we set up.Only a few websites enforced and verified the configuration of a MFA recovery procedure.During our experiment, 10 (14.08%) websites required us to set up and test the recovery procedure before being allowed to complete the MFA setup or required a manual confirmation that backup codes were stored before finalizing the MFA configuration.Documented MFA (Recovery) Procedures.In this paragraph, we discuss our evaluation of how websites communicated the details of MFA and recovery procedures with users, encompassing both our user experience during the account and MFA setup process, as well as the publicly accessible help and support pages.
Building on our evaluation of help pages in our initial categorization (Section 3.1.1),for our user experience, we revisited each of the 71 websites' help and support pages in a more in-depth investigation of their MFA and recovery procedures and formed a basis for comparison with our user experience.We collected additional data about, e. g., whether the recovery setup was part of enabling MFA, if users were warned about account loss in case of MFA loss, or whether account restrictions or forced setup of recoveries were mentioned.We next discuss interesting findings and differences between the documentation and our own experience.Figure 4  Our detailed analysis also reveals that at most two-thirds of websites discussed recovery procedures during MFA setup, slightly more frequently present in documentations (66.2%) than in (our) user experience (59.15%).
Although a perfectly secure MFA implementation would include account loss for users unable to prove their identity, we found only 16 (22.54%)websites to explicitly warn us about the potential danger of irrevocably losing access.This was even rarer within official documentation (13, 18.31%).However, 35 websites (49.3%) included more vague warning phrasings that mentioned account loss but refrained from portraying it as a guaranteed consequence of losing MFA and its recovery.Overall, we find that most of these do not convey the urgency of keeping the MFA secured, or do not even clearly state the connection between losing one's MFA and therefore losing access to the account.In the following, we examine aspects from Figure 4 in tandem with our recovery rates.
We further find that the warnings did not always portray the truth: 9 (12.68%)websites allowed us back into our accounts despite having warned us explicitly that they would not.On the other hand, we found that we lost 6 (8.45%) accounts without receiving any prior warning.As such, we emphasize that when warning users about account loss in case of MFA loss, websites should also introduce users to the recovery process, particularly during MFA setup.We find that of the websites that do warn about account loss, only 14 (19.72%)adhere to this, suggesting that users are often left alone with the task of researching and enabling MFA recovery procedures, despite the danger of account loss.
In our initial evaluation of deployed MFA and recovery procedures, we found SMS to be a popular MFA solution despite its known insecurities [15,46,51,80].We were therefore interested in the number of websites that acknowledged these insecurities and warned about SMS and found that while SMS was available for roughly half of all pages, websites rarely (0% during user experience;9.86% of documentations) admitted its insecurity.
Further, our analysis indicates that websites were often inconsistent regarding their communication of mandatory MFA usage or the enforcement of recovery procedures.We find usage restrictions mentioned more commonly within official documentation (30.99%) than we encountered (4.23%).This is likely because websites most commonly mentioned requiring account administrators to enable MFA, and the accounts we created might not have had this status.Regarding enforcing recovery setup, only a small portion of websites includes this (14.08% during user experience; 7.04% of documentations).As websites cannot verify that users did, e. g., store their backup codes, we considered every mechanism that tried to only allow users to continue the process after the recovery was set, e. g., by using checkboxes in which users confirm that they stored the backup, asking users to use the backup as a proof, or setting up recovery before MFA itself is set.
Websites often mention in which situations MFA is triggered, e. g., any login, only logins from new locations or devices, or specific functionalities such as accessing security-relevant settings or withdrawing funds.However, we find the MFA trigger more commonly mentioned on help and support pages (92.96%) than as a part of the MFA setup (70.42%), although, e. g., how commonly users are required to provide the information can be an important reason for or against enabling MFA.
Finally, we examined the number of pages within the documentation, help sites, and FAQs that discussed MFA and its recovery.In cases where a website offered multiple services, we included only those related to the specific account we set up, and argue that ideally, all information should be present on a few pages to help users find information quickly without getting lost within the documentation and to help website providers keeping all information current as there are fewer pages requiring updates.We find that 18 (25.35%)websites within our sample used more than three different pages to discuss all details of MFA and its recovery, therefore potentially obstructing a user's search for help, and involuntarily burying information within too many pages.
Sample Characteristics.This study explored different MFA recovery implementations that services deployed in an in-depth study of recovery procedures.While our sample is not generalizable, we provide an overview of the websites within our sample and the respective recovery results to contextualize our findings.An anonymized overview of websites in our sample can be found in our replication packageand in Table 6 in the Appendix.In general, it is difficult to quantify the value of accounts, as this is not only dependent on financial information linked to them, but also on social, physical, or sentimental value, and therefore subjective for each user.For example, while losing a social media account can be meaningless for some, it might be the base of income for others.Likewise, the precise threat model is dependent on how valuable users estimate their accounts to be, which results in a similar subjective outcome.However, we deem the value of an account to be highly relevant to users, as this has an impact on their security decisions and how they protect accounts.
To estimate the risk and damage for users when they lose access to an account, or when a malicious third party is wrongfully given access, we took notes on which types of data a website requires to function, including financial data (e.g., credit card or banking information), addresses (typically in the form of delivery or invoice addresses), or others such as telephone numbers.We find the majority (42) of websites to require financial data to utilize the website, typically because its main purpose is related to investments, shopping, or money pooling, and while accounts can be created without providing payment data, they are unusable with regard to the website's purpose.Additionally, 25 websites do not require storing payment data, but offer e. g., voluntary subscriptions.In our recovery experiment, we were able to recover half of these websites (50% for required data; 56% for voluntary data), showing that the improper MFA implementations we encountered indeed endangered valuable user data.Similarly, a large portion requires (30) or optionally collects (20) an address, and again was able to regain access to a significant number of websites (33.33% for required data; 60.00% for voluntary data).
Overall, most websites in our sample offer paid services, therefore collecting personal data.While financial data is typically not displayed in plain text (i.e., attackers cannot gain the credit card number as only the last digits are shown), it might still be abused by an attacker if malicious orders are placed using stored payment information.Other information, such as personal addresses or phone numbers, could be leaked and abused.In cases where the account is lost without a malicious party gaining access, users might lose access to products they legitimately bought, or lose content with sentimental value, such as social media they maintained for years, or photos and memories they shared.However, we found some websites to re-allow access in case an invoice or bank statement was provided, as this was accepted to identify the user.
When regarding the different sample sources, i. e., whether the account was added to our sample because it was interesting, topranked, or randomly selected, we see almost no differences regarding the success of our recovery study.Overall, we successfully regained half of all samples, with the largest difference being that we received any kind of answer or reaction from all top-ranked accounts, therefore losing more accounts than in other sample types.However, the non-answer of 4 handpicked and random accounts can also be considered as a form of account loss.Similarly, we found top-ranked accounts to more commonly offer contact forms rather than email addresses, which might be due to them being more professionally structured due to their popularity and large user bases.While the recovery results were similar for all samples, we found handpicked accounts to diverge from the other two samples in some cases, e. g., by less commonly offering SMS or software tokens and overall having less variety in their offered MFA methods, as well as offering no contact forms, but usually, email addresses to reach out to support staff.
While we excluded websites that required the provision of, e. g., governmental IDs at start-up to protect the sensitive documents of the researchers conducting this study, we found that 13 websites later required them for recovery purposes, leading to account loss on our side.This includes websites that mainly deal with, e. g., cryptocurrencies, payments, or IT-focused websites that offered, e. g., domain registrations, and website hosting.However, we also encountered similar websites within the sample that allowed us access without requiring a governmental ID, i. e., we regained access to potentially valuable accounts with typically only access to the respective email inbox.Key Points | Recovery User Experience We created 71 accounts, lost access to our MFA, and recovered access to half of them.Three quarters of the websites prompted us to set up any kind of recovery during MFA setup, typically backup codes.Reasons for success included email access and contextual knowledge; reasons for account loss were strict policies, lack of ID, and lack of contact options.User experience and documentations rarely matched, with documentations usually providing additional information.

ETHICS AND LIMITATIONS
Below, we discuss ethical concerns and the limitations of our work.

Ethics
This work was approved by our institution's ethical review board.In the first part of our evaluation, we manually examined nonpersonal, publicly available data.We further took care to not cause unusual resource burdens, and refrain from naming any website in the paper.In the second part of this work we contacted human support staff who were not initially aware that our support requests were part of an academic research project.However, we carefully designed our study according to the Menlo Report [28] and its recommendations for a deception study.Overall, we deem the harm caused by our study to be justified by the benefits to society offered by the evaluation of current MFA setups and recovery procedures and the resulting recommendations for developers and website operators.During our work, we caused an additional workload with our dishonest support requests.To minimize the burden on individuals, we took care to exclude websites working solely with volunteers.Furthermore, we limited ourselves to creating only one account per service and sent only one support request.We kept all communication as concise as possible and did not send out reminders.We also refrained from collecting sensitive data, and did not evaluate the support staff members themselves.
For each website, we studied the terms of service and refrained from registering with 20 services for which we would have violated them.As we were unable to ask for prior consent due to the nature of the study, we followed best practices and sent a post hoc notification and debriefing email to all services we evaluated (see Appendix B.1) We informed them about the fully anonymous use of their data and allowed them to drop out of the study, in which case we deleted all collected data for the respective website, and did not use it in our work.A total of 4 websites declined participation and were subsequently removed from our data set.Overall, we deem the deception in this work to be necessary for our goal of evaluating unbiased MFA recovery procedures, as otherwise support staff members might have adapted their behavior.All details of our methodology, including all texts we used to communicate with human support workers, were part of our approved ERB application.

Limitations
In observational studies like ours, there are multiple potential sources of bias or error.First, regarding our data source, we obtained a list of websites offering MFA using the 2fa.directory database.As we illustrate (c.f.Section 3.1) this database offers a public repository, where volunteers can contribute data according to certain quality criteria [3,4,5].The quality criteria nicely align with our research goals, since they require high popularity ranks.We, therefore, decided to use 2fa.directory instead of manually going through top websites.During our evaluation, we noticed that the list is slightly biased towards including more technical websites (e. g., 43.59% of all 1,303 websites belong to technical or related categories).However, these websites are likely more tech-affine and therefore may be more likely to offer MFA.This suggests that our measurement constitutes an upper bound to MFA recovery procedure quantity.To categorize the websites, we restricted ourselves to publicly available information, and might therefore have missed information only accessible to authorized users.However, this reflects the perspective of users who lost access, search information on MFA recovery procedures, and can also only rely on publicly available data.
The user experience study had additional limitations.We could not create accounts on 111 websites, including websites that required us to provide a government-issued ID, sign a (paid) contract for, e. g., a bank or investment service and utility providers for, e. g., gas or electricity, be part of a certain group such as enrolled students, or own certain devices such as IoT devices.This decision was made to protect the privacy of the Ph.D. students conducting this research.Additionally, our sample of service providers is globally distributed, and some services abroad did not allow us to create accounts located in Germany.We further focused on free account plans and did not pay for any service.While premium account tiers could have led to different results (e. g., because we would have received premium support), we think that our results still provide valuable insights for the community, and reflect the experience of users who are unable to pay for such services.Furthermore, we excluded 155 websites that were not primarily in German or English.While we aimed to sample high-ranked, handpicked, and random websites to increase diversity, and resampled whenever we needed to skip a website (cf.Section 3.1.1),the number of handpicked websites was limited and could therefore not be arbitrarily extended, resulting in uneven group sizes.Our study is further limited by our choice to not utilize the recovery codes we received on some websites and to retain our email inbox.However, our goal was to treat all websites equally, and we argue that losing our recovery procedure along with the main MFA sufficiently reflects reality, especially since the configuration is rarely enforced.
Finally, during our detailed analysis of our experience and official help and documentation pages, we might have missed information due to not following every path during the account and MFA setup.We especially did not follow links to the documentation given during MFA setup, but only the texts and prompts were shown during the process.

DISCUSSION
Below, we discuss our results and address our research questions.

RQ1: Diverse Landscape of Recovery Procedures
We identified 17 different recovery procedures based on the public help and support pages of 1,303 websites (cf.Section 3.1.2).Most providers offered support telephone hotlines, email addresses, or local IT staff, followed by backup codes.All procedures offer limited usability and security: Users may reach out to support channels and IT staff over insecure channels, such as unencrypted email [53,82].While backup codes are easy to distribute, they have similar issues as passwords: users can lose them or store them insecurely.
The high diversity of MFA recovery procedures we found is in line with work by Ghorbani Lyastani et al. who investigated MFA deployments without evaluating recovery.Implementations between websites and service types differed vastly, and we could not identify best practices regarding MFA recovery procedures.Due to the lack of standards or best practices, most website providers deployed custom procedures, often leading to contradictions between websites and making the lives of their users unnecessarily complicated.For example, we both encountered websites suggesting setting up multiple backup methods, while others encouraged their users to only configure one recovery procedure to reduce their attack surface.While we agree that too many backup options decrease security, it is sensible to not rely on a single method and to clearly communicate the offered methods and the significance of a working recovery.Similarly, some websites warned us about using SMS-based services due to various possible attacks on mobile networks, while others praised SMS for its availability and usability.In general, SMS should be avoided due to several vulnerabilities [15,46,51,80] -however, due to its advantages, it can be a valid method for low-risk services if there is an alternative for users that are uncomfortable with sharing their private phone numbers.After reaching out to their support, some providers recommended we create new accounts instead of recovering the existing ones.They made this recommendation after manually checking our accounts and seeing that they were not frequently used.
Another provider re-allowed us access without identification because the account was empty and not much used.In this specific case, we deem both methods valid for empty accounts but argue that account deletion is the safer option to give users full control over the stored data in case recovery procedures are abused.In all of these cases, the deviating arguments have a well-meaning, true core, and in many cases, compromises based on the nature of the service are necessary.We discuss shortcomings and recommendations in Section 5.3.

RQ2: Inconsistencies Compromise Security & Usability
Overall, we found examples of both websites that prioritized security, and did not allow us access without the proper MFA or recovery procedure we configured, and websites that helped us to regain access without having access to our second factor, lowering authentication security (cf.Section 3.2.2).However, from a user's point of view, this means either sacrificing security or usability, which can be especially critical for accounts that manage important resources such as monetary funds or medical data.
We identified inconsistencies between the authentication security associated with the use of MFA, and the recovery procedures implemented on many websites.Using MFA is expected to strengthen the security of authentication by limiting account access to users who know the account password and are in possession of the configured additional authentication factor.However, having access to the email address we used for account creation was sufficient for most of the accounts we regained access to.This effectively reduces the security of MFA to that of email security, which has been criticized in prior work about account recovery [8,53], and would allow an attacker with email access to circumvent MFA.Whenever we were required to provide contextual evidence such as our address or the nature of connected platforms, the security decreased to the level of security questions, which are also known to be insecure and guessable [19,71].
Although we never said anything about having lost backup codes as well, services typically did not request them to recover our MFA.This might be based on experiences with prior users that typically lose both MFA and recovery, or that many users did not configure a recovery procedure.However, it also means that services refrained from utilizing their own suggested recovery.Websites that distribute backup codes should adhere to these security decisions and not accept other, even less secure alternative identifications.
We ascribed many of the rejected recovery requests to usability issues.This includes not being able to contact the support, e. g., because contact is only available to logged-in or paying users.This can be especially frustrating when users are left without any other recovery option.
In other cases, providers requested any form of ID or data without previously informing us that they would be part of the MFA recovery procedure.While governmental IDs are a valid way to verify identities in some situations, they are not suitable to do so for accounts that often know little more about their users than full names, birthdates, or email addresses.Furthermore, even videobased identification processes have been successfully circumvented, allowing attackers to falsify documents and successfully impersonate their victims [30].
We find that websites rarely prepare users to provide ID identification, as they are usually not prompted to provide it during account setup, or informed that it will become relevant during recovery.However, the availability of these documents can otherwise be an issue, and lead to higher obstacles during recovery later on.For example, not all countries, including the USA and Switzerland, require their citizens to own governmental IDs.Further, the causes of MFA loss might also affect the availability of recovery: a lost or stolen purse might mean that not only the TOTP app on a user's phone, but also their wallet and ID might be gone, or an emergency, such as a house fire, could destroy both mobile phones and identifying documents.Finally, users who are not aware of the relevance of identifying information for MFA recovery might provide false information to protect their privacy.Some websites had exceptional expectations.For example, some help and support pages suggested setting up multiple hardware keys in case one was lost or broken.While having a backup configured is, in general, sensible, hardware keys are expensive, and having to set up multiple keys can impose a financial burden excluding users from setting up MFA or its recovery.However, this is also encouraged by official FIDO guidelines regarding security keys [10].
Other websites downplayed the process of regaining access to their users' phone numbers, by suggesting users get a replacement SIM card with which they can access their original MFA again.However, this replacement can require significant time to arrive, especially if the user needs replacement during, e. g., a vacation.
We find the documentation to largely match our experience, but none of them matches perfectly.While we expect help and support pages to be more detailed, we argue that some details, such as warnings about account loss, or the intended recovery procedure should also be communicated during MFA setup.However, they are often not found at all in either, therefore not sufficiently preparing users for a potential loss of MFA.We further find that websites often do not match their documentation regarding the most basic elements, such as the offered MFA and setup methods, which suggests that help and support pages might be outdated and not reflect the current recovery procedures.This could lead to unpleasant surprises as users are not properly prepared for MFA recovery.

RQ3: Improving MFA Recovery Procedures
The three most pressing problems we identified related to unreachable teams, insufficient MFA recovery procedure documentation, and information that was needed for recovery but not configured during setup.Based on our findings, we make the following recommendations: Prepare.Our findings illustrate that recovery procedures require a sufficient setup that should be part of the MFA setup.We suggest that all information service providers requests during recovery are collected at account setup.Users need to be warned about everything that is strictly necessary, i. e., they should be told about required documents and how to avoid recovery failures due to security reasons.However, asking for too much or too sensitive information can be risky in regard to, e. g., information abuse and data breaches [47,58] and might deter users.Future research should work on identifying the right balance between the interests of both user privacy and MFA recovery.While we agree that limiting the number of MFA recovery procedures is useful to not open up multiple attack vectors, having fallbacks available is in line with general security recommendations depending on the respective use case and threat model [27,56].As some procedures, such as SMS or security questions, are known to be insecure or impractical, we suggest offering backup codes, additional unique communication channels, e. g., bank statements where possible to identify the user as account owners, or additional (desktop) devices with TOTP.
Communicate.The process of setting up, disabling, or recovering MFA and what to expect in the worst case should be clearly and directly communicated during the setup process and be part of publicly available help and support pages.We argue that having too many help pages can be confusing and lead to both users not finding the information they need [73], and website maintainers struggling to keep them all up-to-date.We, therefore, suggest keeping the number of different help and support pages as low as possible and using structural elements like HTML tabs to help users more easily find the required information for their platform or MFA method.
Maintain.Recovery procedures should be tested regularly.Websites could prompt users to, e. g., enter one of their backup codes or a TOTP to make sure that the recovery procedure is still available and working.Previous work has shown that pop-ups and security warnings can be perceived as obtrusive or might be ignored by users [7,14,83,91].Hence, future work is required to find a good trade-off between verifying that recovery procedures are still available, and obstructing user workflows too much.
Recover.For MFA recovery, we urge websites to adhere to their documented recovery procedures.When websites deviate from their documented recovery procedures, e. g., by restoring access despite warning about definite account loss, or by allowing additional recovery measures, they might create opportunities for attackers that users are not aware of, or create (false) expectations for other websites.Additionally, honest and clear communication about the consequences of MFA loss can help users make informed and empowered decisions about their authentication security.We advise against using sensitive documents, such as governmental IDs or utility bills, as identity proof, as they often cannot reliably identify authorized users.However, if this kind of identification is sensible and necessary, websites should take precautions to collect it early on and make sure that users are informed of this part of their recovery procedure.

Putting our Work into Context
Previous work often focused on investigating the theoretical usability of recovery procedures in different areas [36,37,42,48,65,71,75].In contrast, our work studies the first-hand user experience of MFA recovery procedures on websites: We went through the actual MFA recovery procedure processes implemented by 71 websites and reported lessons learned.We could regain access to almost half of the websites we tested based on having access to our email inbox.Hence, our study illustrates that MFA recovery procedures' security on websites is affected by similar security risks as TOTP apps or smartphone security implementing insecure methods as fallback [37,48,75,78].
Previous work evaluated security documentation for setting up MFA [36,76], and encountered issues around unclear instructions.Our work shows many websites did not sufficiently document their MFA recovery processes, e. g. instructions described recovery processes that did not match the implemented deployments.Some websites did not provide documentation at all.

CONCLUSION
MFA recovery procedures need to balance security and usability by allowing authorized users to access accounts without locking them out due to MFA loss.We are the first to analyze deployed MFA recovery procedures and related documentations on the web.In this work, we first categorized the recovery procedures deployed on 1,303 websites, then conducted an in-depth analysis in which we created 71 accounts, configured MFA and recovery procedures, and finally tried to recover them from supposed MFA loss.We found that most websites only offered a limited selection of recovery procedures and that these were not necessarily part of the MFA setup process.In the recovery process, websites often did not adhere to their documentation and allowed us access without requiring the configured MFA or recovery.In most cases, having access to the account's associated email address was sufficient for account recovery.Overall, we recommend websites clearly and correctly document their MFA recovery procedures.Measures to detect loss of MFA or recovery procedures before users are locked out should be in place.
enable third parties to intercept traffic or change the user's phone number to the attacker's device via social engineering.Similar to email, a secondary phone number can be used to offer SMS or phone calls as MFA recovery 4 .

A.3 Mobile Applications
Mobile applications, such as time-based one-time password (TOTP) apps like Google Authenticator [54], are a common MFA method.They generate one-time passwords [55] (OTPs) that users can provide to prove their identity 4 .The app is set up by scanning a confidential QR code or entering a secret key provided by the respective website, which serves as a seed for the TOTP algorithm.As long as both server and client have the same current time and share the same secret, they can independently compute the same TOTP values.Besides this, some apps like Authy [85], or password manager apps such as LastPass [49] or 1Password [1] require users to register an account and offer a cloud backup of the supported MFA as a trade-off.While this eases recovery, it also means that users are not fully in control over their data, as service providers also have access to the backups.Related to this, LastPass leaked user data during two hacks in 2022 [33].Lastly, some websites provide custom apps for their respective services, and include MFA in the forms of, e. g., TOTP, usage of biometric sensors embedded in smartphones, or push notifications.The provided security and usability in these cases is limited by the app vendor and its implementation [64].

A.4 Hardware Tokens
Hardware tokens (e. g., smart-cards), code generators (e. g., Transaction Authentication Number (TAN) generators), or Universal Second Factor (U2F) [11] hardware keys, are established MFA methods.WebAuthn supports the use of trusted platform modules, including biometric identity checks [9].The adoption of U2F and WebAuthn has been limited [26], often due to their complex initial setups and device requirements.However, recent improvements provide better usability [74].The protocol offers verified security guarantees [16] and the main assumption is the secrecy of the chosen hardware token, resulting in the credential proof being something you have.

A.5 User-Dependent Methods
Below, we illustrate approaches that are dependent on the user, their input, or inherent features.First, biometry-based MFA commonly relies on apps that use the internal biometry security options of, e. g., mobile phones to authenticate.The main advantage compared to mobile applications is that authentication uses biometric features, requiring no additional device or memorization.While technically secure, there have been reports of malfunctioning biometrics [12,39,40,79,94], and they further require very sensitive personal information for authentication.Another approach are secret questions that supposedly only the authorized user can answer, and backup codes that users are given and supposed to store securely or print, or the storage of the TOTP secret, i. e., the secret key or QR code shown during MFA setup.While these methods are commonly deployed as recovery or alternative to MFA, the security is often lacking: security questions can easily be guessed, especially since they tend to require information that is often known to acquaintances or social media contacts [19,71,77], and user-set passwords are often weak or re-used [18,69,87,89,90].Methods such as backup codes or stored secrets are easily lost or allow attackers to bypass MFA completely if they gain access.

A.6 Website-Dependent Methods
We also distinguish methods that are dependent on the websites.In case access to MFA is lost, users can often contact the website support.In other cases, especially for team-accounts or self-hosted tools, the account is managed by local administrators that can be contacted.Both cases can include fixed waiting times or identity verification for account recovery.In some cases, this process is more straightforward, as users are asked to provide a photo proof of them holding, e. g., handwritten notes or governmental IDs.Other websites utilize automated dedicated recovery systems, or disable MFA automatically when the password is reset.The security of these methods depends on the respective communication channel, such as email or HTTPS.While easy to use, the usability can suffer from, e. g., long response times or queries for additional information, as methods involving any form of communication are often dynamic and resolved on a case-by-case basis.

A.7 Additional Methods
In some cases, MFA is not needed for login, but required for certain service features such as withdrawing funds.Some websites allow to omit MFA for trusted devices.While used as recovery, those methods are often bound to a time-limit of, e. g., 30-90 days after which MFA is required again, leading to a decreased backup usability if MFA is lost outside this time frame.
Other recovery advice might combine some of the previously mentioned factors, or give additional advice to, e. g., contact mobile providers to replace inaccessible SIM cards.Finally, the recovery can simply consist of setting up additional MFA methods, i. e., extra methods that can all act as a backup for whichever method was lost by the user.While this has been an official FIDO recommendation [10], it also increases the attack vectors for malicious third parties.
While neither a MFA nor recovery procedure, we encountered additional edge cases in which we were unable to determine the methods.This includes inaccessible help pages we were unable to visit due to, e. g., geo-blocking or deleted pages, and websites either mentioning explicitly that they had no recovery methods, or websites that did not mention recovery at all.

B GENERAL APPENDIX B.1 Recovery Test Templates
Recovery Request Message.Subject: Lost Second Factor Hello, I recently registered an account on your website and enabled 2FA.Yesterday, my backpack with my wallet and phone was stolen, so now I cannot access anything, and am unable to get past the 2FA.What do I need to do to regain access?My username/email/account number is data.

Best regards, First Author
Debriefing Message.Dear website support, We are a research team at the Anonymized For Review Institution, Country, and study the usability of multi factor authentication (MFA).We are currently investigating the trade-off between usability and security of multi factor authentication recovery.As a part of this research, we created accounts on several websites, enabled MFA, and tried to recover our accounts after a while due to supposedly inaccessible MFA.These websites were chosen because they were listed on 2fa.directory [1], ranked highly on Tranco [2], or because their documentation implied interesting MFA methods.As you might already guess, we write to you today because your website was among the ones we chose.We wanted to inform you about your involuntary participation, as well as give you the time to decline and have your data removed.On date, we send you an email/chat request (request ID if possible), which was part of our research, i.e., it was not a real users' request to recover their MFA.While we are not content about deceiving you and sending a false request, we wanted to assure you that it was necessary for the goal of our research.We wanted to truly gauge the experience of a typical user, and to assess the usability and security of MFA procedures on the web.Due to this goal, we were sadly unable to communicate the true purpose of our request upfront.Please be assured that we only sent one line of request, and that we kept it as short as possible to decrease the load on your staff caused by our inquiry.As mentioned, we would like to give you the opportunity to have your data removed and drop out of our data set before we de-identify it and use it as part of a scientific article.If you let us know until October 10th5 , we will irrevocably delete all communication with or data about your website.Please be assured that we currently only store the data on encrypted, self-hosted servers, and that only involved researchers have access to our results.Furthermore, we will never publish the name of your website or any involved staff member.We will de-identify your website in our work, and we will never mention your service's or website name.We hope that our study did not cause too much harm, and that you do not have objections against us using the data.In any case, we thank you for being able to create an account and collect valuable experiences, for the helpful support you gave us, and want to apologize for any inconvenience caused by our request.If you are interested in more details of our research, please visit our project website: https://anonymizedforreview.tld/projects/multifactorrecovery/.In case you are interested in the results of our research, we would be happy to share the paper once it is published.Finally, we are also interested in deepening our understanding of MFA and its recovery by also including your perspective in future research (e.g. an interview).If you would be available for a follow-up study, we would be delighted to hear from you! Best regards, First Author (PhD Candidate) Anonymized For Review Institution [1] https://2fa.directory/int/[2] https://tranco-list.eu/We were further interested in the top-level domains (TLDs) of websites in our sample.Due to our removal of 155 websites in languages we were not fluent in, we found the vast majority of remaining websites (944, 72.45%) to be .comwebsites, with the remaining being other English (e. g., .ca,.com.au, .co.uk) or German (.de, .ch)TLDs.Additionally, we find some neutral ones (.io, .net,.org),and .eduand .gov,which we almost exclusively found in the respective categories of Education, Universities and Government (see Table 2).We find no significant differences between recovery methods offered on different TLDs.

Figure 1 :
Figure 1: Summary of our methodology, consisting of the MFA and recovery procedure categorization (cf.Section 3.1) and a follow-up in-depth evaluation of deployed MFA recovery procedures for 71 websites (cf.Section 3.2)

Figure 2 :
Figure 2: Overview of the recovery success rates per recovery procedure in percent per method.Other encompasses two phone-based recoveries and contact support.TOTP also includes the TOTP secrets.Numbers do not add up to 71 due to websites offering multiple recovery procedures.

Figure 3 :
Figure 3: Summary of account recovery and loss split by root cause.We distinguished between recovery due to email as the main MFA method, and email as the main recovery communication channel due to which access is regained.

Figure 6 :
Figure 6: Presence of MFA recovery methods within the initial 1,303 websites over different categories on 2fa.directory in %.As websites can offer more than one recovery option, the numbers may not add up to 100%.
Table 3in Appendix B.2) displays the frequency with which the topics in this section are present in either our experience or the official help and documentation pages.Overall, our recovery experience never matched the documented Percentage of occurrence of certain topics or hints within either our own experience when setting up accounts and MFA, or the respective help and support pages.Note that not all services offer SMS, i. e., related topics are not necessarily applicable for all tested websites.
recovery procedure perfectly.Even if only the offered MFA methods are considered, we still only find 9 (12.68%) of all help and support pages to match our experience regarding available MFA methods and recovery procedures.This difference suggests that recovery procedures are commonly not communicated properly during setup, or that documentations are outdated or incomplete.

Table 2 :
Distribution of the initial 1,303 websites over various TLDs.

Table 3 :
Distribution of the initial 1,303 websites over the listed bins of Tranco ranks.

Table 6 :
List of websites examined during our recovery experience described in Section 3.2, including their approximated Tranco rank, category or sample group.We chose to omit details on present MFA and recovery methods, as well as the recovery result, within the public version of this work to ensure the anonymity of included websites.* Specific purpose * Always required