Inside an ADA Assessment: What the Laboratory Does

App Defense Alliance

The full walkthrough of a laboratory engagement, from the checks a lab must make before accepting your money to what happens to the records afterwards.

What is this?

When somebody says an independent laboratory assessed our app, that phrase is doing a lot of hidden work. Behind it sits a defined process with mandatory controls, and a laboratory that skips them is not entitled to issue the report.

ADA does not test apps itself. It publishes the requirements, then authorises independent laboratories to do the testing. A lab has to earn that authorisation, and part of earning it is holding accreditation to ISO/IEC 17025, the international standard for testing laboratories. The point of the arrangement is that the people judging your app have no commercial interest in the answer.

The controls exist because independence is not a matter of good intentions. A lab wants your repeat business. Its salespeople want you happy. Left to itself, that pressure bends verdicts. The process described below is what stops it, and every part of it is auditable by the lab's accreditation body.

Who is required to follow this?

Every laboratory authorised by ADA. It is not optional and it is not a matter of house style. An accreditation body inspects the lab against ISO/IEC 17025 and against the ADA requirements, samples real engagements, and can suspend or withdraw the lab's accreditation.

As the customer you are not required to follow any of it, but you are entitled to see that it happened. Section headings below double as a list of things you may reasonably ask about.

Which assurance level this describes

What follows describes a laboratory engagement at AL2, where the laboratory does the testing. Most of it applies at AL1 too, but the division of labour changes: at AL1 you run the test procedures and produce the evidence, and the laboratory's work is to judge whether that evidence is complete and sufficient. The acceptance checks, the independent review, the records and the dispute process are the same either way.

If you are working at AL1, read the evidence standard section particularly closely, because at that level the evidence is your deliverable rather than the laboratory's.

The seven phases

An engagement moves through phases, and each has a gate that has to be passed before the next begins. The gate is not a formality; it is a record that has to exist.

Phase 1: what a lab must check before taking your money

This is the phase customers know least about and it is where the most important protections live.

• Competence for your assessment type. ADA specifies the certifications the engagement team must hold, per assessment type. The lab has to confirm the people assigned to you actually hold current ones. A lapsed certification does not count, and a lab should suspend somebody's authorisation automatically on the expiry date.
• Authorisation, which is not the same as certification. Holding a certificate does not entitle somebody to work on your engagement. The lab has to have formally authorised that individual for that specific activity. This distinction catches a lot of laboratories out at audit.
• A conflict of interest check. Performed before acceptance and recorded. A lab that built, advised on or previously assessed your application cannot validate it. Neither can a lab where somebody on the team has a financial or family interest in your company, unless that person is formally recused and it is written down.
• An available independent reviewer. Somebody who will take no part in the testing. If the lab has only one person qualified for your assessment type, it cannot separate the tester from the reviewer, and it should decline the engagement rather than fudge it.
• Capability and capacity. The right tools, an approved test environment standard, and enough time. A lab is expected to decline work it cannot do properly.
• Method verification. The lab must already have demonstrated and recorded that it can properly perform the assessment method for your specific assessment type. Verification for mobile is not evidence of capability for web.

Phase 2: how your software is handled

• The public build is preferred. For app-store software the lab will normally obtain the store version rather than one you send, because that proves it tested what your users actually get. If it uses a build you supplied instead, the reason has to be recorded.
• A hash is taken on receipt. A cryptographic fingerprint of exactly what arrived. This protects both sides: you cannot be assessed on the wrong build, and the lab cannot be accused of it.
• An environment is built for you alone. Isolated from other clients' engagements and from the lab's own corporate network. Your data must not be reachable from anybody else's engagement.
• The environment configuration is recorded. Operating system version, platform, tool versions, privileged access state. Not the configuration intended, the configuration actually built.
• Tools are checked before use. Each tool is confirmed to be at the qualified version and to produce the expected result on a known input. A tool that fails is not used on your engagement.
• Deviations are recorded and disclosed. If the lab has to depart from its standard approach, that requires approval and appears in your report as a deviation from method.

Phase 3: the evidence standard

This is the single biggest difference between a laboratory report and a consultancy report, and it is worth reading carefully because it determines what your report is actually worth.

For every requirement, the assessor must record evidence sufficient that a different competent assessor could repeat the work and reach the same conclusion. In practice that means each piece of evidence has to show:

Two protections follow for you. A lab cannot record a Fail on a hunch, because a finding has to be shown rather than asserted. And a lab cannot record a Pass because you assured them something works: your own statements are recorded as your statements, and the report has to disclose which requirements were self-attested rather than validated.

Phase 4: how a verdict is decided

There is no partial credit and no aggregate score that conceals the detail. If the programme defines an overall outcome, it is derived from the individual verdicts and reported in addition to them, never instead of them.

Three situations get confused constantly and must be reported distinctly: a requirement examined but undetermined is Inconclusive; a requirement you self-attested that the lab did not validate is disclosed as self-attested; a requirement excluded from scope is disclosed as not evaluated. None of the three may be reported as a Pass.

Phase 5: the independent review

The reviewer is the control that does the most work, and the one customers least expect. They are not proofreading. They re-examine the evidence and reach their own view.

A properly run review works through, at minimum:

• whether the build tested matches the build received, by hash;
• whether the approved environment and qualified tools were actually used, and whether the pre-use check was done;
• whether every verdict is supported by contemporaneous evidence meeting the standard above;
• whether any Inconclusive verdict is standing in for a Fail the evidence would support;
• whether self-attested and excluded requirements are properly disclosed;
• whether remediation guidance is specific to your application rather than a restatement of the requirement;
• whether the statement of conformity and the decision rule are correctly expressed;
• whether the record set is complete;
• whether the impartiality clearance is on file and nobody recused took part.

The reviewer can authorise release, send it back for correction, or withhold the report entirely. Their determination stands. If the engagement partner disagrees, the disagreement is recorded but the report is not released against the reviewer's decision, and the lab may not simply appoint a more agreeable reviewer.

Two hard rules protect that authority: the reviewer's pay cannot depend on the verdict or on engagement margin, and no commercial function of the lab may alter, delay or suppress a finding.

The review record is retained whether or not it changed anything. A review that found nothing is evidence the control was operating; a missing review record is evidence it was not.

What happens if the laboratory gets it wrong

This is a real process, not an embarrassed silence. If a lab discovers that something it released may be wrong, it is required to:

Disputes

If you disagree with a verdict, finding or report content, you can dispute it, and the lab must have a documented process. The important features:

• The decision-maker is independent. The person who recorded the verdict does not decide the dispute, and neither does your engagement partner.
• The evidence is re-examined, not defended. The question is what the evidence supports, not whether the original assessor was reasonable.
• You get to make your case. Your representations are recorded and addressed in the decision, not merely received.
• The programme is told either way. On resolution the lab either provides an updated report to the ADA certification body or informs it that the dispute is resolved. That notification is a record.
• A verdict change triggers more than a correction. If a released verdict was wrong, the lab's nonconforming work process runs too, because something in its own controls let it through.

Ask for the dispute process in writing before you engage a lab, not when you need it.

What happens to your records afterwards

The lab keeps the whole record set, not just the report: every verdict and its evidence, the environment configuration, the tool versions, the review record, every version of the report, and every attempt including any that failed.

Retention is typically at least two years after your certificate expires, and often longer under the lab's own policy. That exists so a question arising later can actually be answered.

It also means history cannot be quietly revised. If a lab suggests a failed attempt can be made to disappear, that is a serious problem with that lab and worth walking away from.

Your material is treated differently from the records. Build artefacts, credentials and test data not needed as evidence are destroyed at the end of the engagement. Ask what is destroyed, what is retained and for how long.

Frequently Asked Questions

Click any question to expand the answer.

Related reading

If this walkthrough is useful, the two decisions either side of it are covered separately: how to prepare for your first ADA assessment covers the four weeks before a lab starts, and how to choose an ADA authorised laboratory turns the sections above into questions you can actually ask.

For the requirement sets themselves, see CASA for web applications and APIs, DASA for desktop and server software, and the Cloud App and Config Profile for the infrastructure underneath.

About Adayptus

Adayptus Consulting Private Limited is an application security testing firm based in Noida, India. We have been doing web, mobile and cloud security testing since 2018.

We are currently building our laboratory management system to ISO/IEC 17025:2017 and working towards authorisation as an App Defense Alliance Security Test Laboratory. We are not an ADA-authorised laboratory today. We will say so plainly on this page until that changes, because you should be able to trust what a security firm tells you about itself.

What we can do for you right now:

• Readiness testing. We test your app against the relevant requirement set and tell you what would fail, before you pay for a formal validation. Finding out early is much cheaper than failing late. That is web application, API or cloud configuration work depending on the profile.
• Fixing what we find. We give you guidance specific to your application, pointing at the actual component and the actual change needed, not a link to a general guideline. Where the cause sits in the code, secure code review is usually the faster route.
• Evidence preparation. We help you assemble the documentation, architecture information and test accounts an authorised lab will ask for, so your validation does not stall on paperwork.

Attribution and source