feat: add conformance test for authorization server metadata by Michito-Okai · Pull Request #170 · modelcontextprotocol/conformance

Michito-Okai · 2026-02-25T11:38:13Z

Motivation and Context

Described in #169

How Has This Been Tested?

npm test (Test Files 6 passed, Tests 85 passed)
npm test -- src/scenarios/authorization-server/authorization-server-metadata.test.ts (Test Files 1 passed, Tests 1 passed)
npm run start -- authorization --url http://localhost:8080/.well-known/oauth-authorization-server/realms/mp
--url URL of the authorization server metadata endpoint
npm run start -- authorization --url http://localhost:8080/.well-known/oauth-authorization-server/realms/invalid
--url invalid URL
npm run start -- list

Breaking Changes

add authorization server mode (-- authorization)
add url option for authorization server mode (--url )
add a conformance test for authorization server metadata

Types of changes

New feature (non-breaking change which adds functionality)

Checklist

I have read the MCP Documentation
My code follows the repository's style guidelines
New and existing tests pass locally
I have added appropriate error handling
I have added or updated documentation as needed

tnorimat

@Michito-Okai Hello, I added some review comments. Could you check them?

tnorimat · 2026-02-26T05:07:38Z

src/scenarios/authorization-server/authorization-server-metadata.ts

+  implements ClientScenarioForAuthorizationServer
+{
+  name = 'authorization-server-metadata-endpoint';
+  specVersions: SpecVersion[] = ['2025-03-26', '2025-06-18', '2025-11-25'];


To avoid duplication, instead of defining spec version here, could you import SpecVersion of types.ts like

import { ClientScenarioForAuthorizationServer, ConformanceCheck, SpecVersion } from '../../types';

?

I fixed. Could you check this?

To avoid duplication, could you remove ['2025-03-26', '2025-06-18', '2025-11-25'] ?

SpecVersion of types.ts is a type declaration. Therefore, these are not duplicates.

tnorimat · 2026-02-26T05:13:13Z

src/scenarios/index.ts

+export function getClientScenarioForAuthorizationServer(
+  name: string
+): ClientScenarioForAuthorizationServer | undefined {
+  return clientScenariosForAuthorizationServer.get(name);


getClientScenarioForAuthorizationServer declares its return type as ClientScenarioForAuthorizationServer but that type isn't imported. Could you add it to the import from '../types' on line 1 ?

I fixed. Could you check this?

tnorimat · 2026-02-26T05:18:03Z

src/index.ts

+    versionScenarios = listClientScenariosForSpec(version);
+  } else if (command === 'authorization') {
+    versionScenarios =
+      listClientScenariosForAuthorizationServerForSpec(version);


The function is called inside filterScenariosBySpecVersion but was never imported from './scenarios'. It's exported from index.ts but missing from the import block at line18-34 of index.ts. Could you import that ? like

import { listScenarios, listClientScenarios, listActiveClientScenarios, listPendingClientScenarios, listAuthScenarios, listMetadataScenarios, listCoreScenarios, listExtensionScenarios, listBackcompatScenarios, listScenariosForSpec, listClientScenariosForSpec, getScenarioSpecVersions, listClientScenariosForAuthorizationServer, listClientScenariosForAuthorizationServerForSpec, ALL_SPEC_VERSIONS } from './scenarios';

I fixed. Could you check this?

tnorimat · 2026-02-26T05:20:33Z

src/index.ts

+        authorizationServerScenarios = filterScenariosBySpecVersion(
+          authorizationServerScenarios,
+          specVersionFilter,
+          'client'


authorization instead of client ?

I fixed. Could you check this?

tnorimat · 2026-02-26T05:23:16Z

src/index.ts

+          console.error(`  ${err.path.join('.')}: ${err.message}`);
+        });
+        console.error('\nAvailable authorization server scenarios:');
+        listClientScenarios().forEach((s) => console.error(`  - ${s}`));


listClientScenariosForAuthorizationServer instead of listClientScenarios ?

I fixed. Could you check this?

I confirmed it.

tnorimat · 2026-02-26T05:29:03Z

src/index.ts

+        console.log('');
+      }
+      console.log(
+        'Authorization server scenarios (test against a authorization server):'


an authorization server instead of a authorization server ?

I fixed. Could you check this?

tnorimat · 2026-02-26T05:33:42Z

src/types.ts

  run(serverUrl: string): Promise<ConformanceCheck[]>;
 }
+
+export interface ClientScenarioForAuthorizationServer {


Both interfaces ClientScenario and ClientScenarioForAuthorizationServer have the exact same shape. Consider using a type alias (type ClientScenarioForAuthorizationServer = ClientScenario) or just reusing ClientScenario directly?.

The shape of ClientScenarioForAuthorizationServer is likely to change with enhancements, so I defined ClientScenarioForAuthorizationServer.

Thank you for the clarification.

tnorimat · 2026-02-26T05:35:55Z

src/scenarios/authorization-server/authorization-server-metadata.ts

+    let errorMessage: string | undefined;
+    let details: any;
+
+    const fail = (msg: string) => {


The fail is never called.
All validation methods throw instead, and the catch block sets status and errorMessage directly.

For example,

} catch (error) { status = 'FAILURE'; errorMessage = error instanceof Error ? error.message : String(error); }

I fixed. Could you check this?

tnorimat · 2026-02-26T05:42:04Z

src/scenarios/authorization-server/authorization-server-metadata.ts

+      );
+    }
+
+    const expectedIssuer = serverUrl.replace(


It only removes the first occurrence. If the URL were e.g. https://example.com/.well-known/oauth-authorization-server/tenant, the replace would produce https://example.com//tenant (note the path portion is kept). Per RFC 8414 §3, the issuer for a path-aware metadata URL /.well-known/oauth-authorization-server/<path> should be https://example.com/<path>. The current approach works correctly for the root case but may be incorrect for path-based issuers.

For example,

const url = new URL(serverUrl); const wellKnownPrefix = '/.well-known/oauth-authorization-server'; const suffix = url.pathname.startsWith(wellKnownPrefix + '/') ? url.pathname.slice(wellKnownPrefix.length) : ''; const expectedIssuer = `${url.origin}${suffix}`;

I think that if the URL were e.g. https://example.com/.well-known/oauth-authorization-server/tenant, the replace would produce not https://example.com//tenant but https://example.com/tenant. Therefore I think that the current approach works correctly for path-based issuers.

Thank you for the clarification.

tnorimat · 2026-02-26T05:44:29Z

src/index.ts

-    if (options.server || (!options.client && !options.server)) {
+    if (
+      options.server ||
+      (!options.client && !options.server && !options.authorization)


The server command supports running a single scenario (--scenario), suite selection (suite), expected-failures baselines, and verbose output (--verbose). The new authorization command has none of these, making it less flexible. This may be intentional for an initial implementation but is worth noting. It is up to you to consider this point by this PR.

The scenario, suite, expected-failures and verbose options are outside the scope of this PR. I am considering to add these options in a future enhancement. How does that sound?

Yes I understand.

pcarleton · 2026-03-17T11:29:19Z

src/scenarios/authorization-server/authorization-server-metadata.ts

+    body: Record<string, any>,
+    serverUrl: string
+  ): void {
+    this.assertString(body.authorization_endpoint, 'authorization_endpoint');


I like the throw / catch pattern for issues that block further progress (e.g. a non-200 response), but for issues where there may be multiple distinct issues, I'd prefer we either add multiple failing checks to the list of checks returned OR include all validation failures in one check so that the user gets feedback on the maximum possible list of failures to check when they get a test failure.

e.g. I want to avoid this situation:

Run test, fails w/ missing authorization_endpoint

Run test again, fails w/ missing token_endpoint

Run test again, fails with missing response_types_supported

And instead have:

Run test, fails w/ missing authorization_endpoint, token_endpoint, response_types_supported

@pcarleton
I fixed as follows. Could you check this?
When test fails in validateMetadataBody, failures are not handled with a throw / catch pattern, but instead all checks are consolidated and verified within a single test run.

pcarleton · 2026-03-17T11:29:49Z

src/scenarios/authorization-server/authorization-server-metadata.ts

+- Return a JSON response including issuer, authorization_endpoint, token_endpoint and response_types_supported
+- The issuer value MUST match the URI obtained by removing the well-known URI string from the authorization server metadata URI.`;
+
+  async run(serverUrl: string): Promise<ConformanceCheck[]> {


mentioned in the issue, but I think we want serverUrl to be the issuerUrl, and we test that the metadata is at one of the .well-known paths.

@pcarleton
I fixed as follows. Could you check this?
run receives argument as the authorization server issuer.
createWellKnownUrl create the authorization server metadata url based the authorization server issuer

This looks good for OAuth metadata, but I was assuming we would also check OIDC metadata routes.

Basically I would imagine we wouldn't want a server to marked as failing if it's using OIDC metadata, since we treat that as valid.

I believe that would require us probing URL paths.

@Michito-Okai Hello, as we discussed, to incorporate @pcarleton comments, how about the following by following the MCP 2025-11-25 spec https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#authorization-server-metadata-discovery ?

The test receives an issuer(server) url (mandatory) including or not including a tenant/realm part.

If the tenant/realm part exist, creating an well-know url and access the url by the following the order that the MCP 2025-11-25 spec described.

1. OAuth 2.0 Authorization Server Metadata with path insertion: https://auth.example.com/.well-known/oauth-authorization-server/tenant1 2. OpenID Connect Discovery 1.0 with path insertion: https://auth.example.com/.well-known/openid-configuration/tenant1 3. OpenID Connect Discovery 1.0 path appending: https://auth.example.com/tenant1/.well-known/openid-configuration

If the test can successfully fetches the metadata from one of them, the test proceeds (no need to do all the three patterns above). If the test cannot fetch the metadata from any of them, the test returns a failure.

If the tenant/realm part does not exist, creating an well-know url and access the url by the following the order that the MCP 2025-11-25 spec described.

1. OAuth 2.0 Authorization Server Metadata: https://auth.example.com/.well-known/oauth-authorization-server 2. OpenID Connect Discovery 1.0: https://auth.example.com/.well-known/openid-configuration

If the test can successfully fetches the metadata from one of them, the test proceeds (no need to do all the two patters above). If the test cannot fetch the metadata from any of them, the test returns a failure.

@pcarleton @tnorimat
Thank you for your comments. I see. I added OIDC metadata routes.
If the test cannot fetch the metadata from OAuth metadata route, the test fetches from OIDC metadata routes.

tnorimat

@Michito-Okai Hello, I reviewed the PR and it seems good to me. I think that the revised PR incorporates @pcarleton the last comment.

pcarleton

This looks great, thank you!

nbarbettini · 2026-03-25T16:39:52Z

src/scenarios/authorization-server/authorization-server-metadata.ts

+
+    if (
+      !Array.isArray(body.response_types_supported) ||
+      body.response_types_supported.length === 0


We should also require that response_types_supported include code. The AS may also support other response types (that's okay), but MCP requires the auth code flow + PKCE which should be code in OAuth 2.1.

@nbarbettini
Thank you for your review. I modified the check from verifying that the array has one or more elements to verifying whether the array contains code.

good catch, ty!

@Michito-Okai Looks good, thanks!

nbarbettini · 2026-03-25T16:40:52Z

src/scenarios/authorization-server/authorization-server-metadata.ts

+
+    if (body.issuer !== serverUrl) {
+      errors.push(`Invalid issuer: ${body.issuer ?? '(missing)'}`);
+    }


Exact issuer equality is correct, but we do need to be careful about path normalization before this check.

RFC 8414 §3.1 says that if the issuer identifier contains a path component, any trailing slash MUST be removed before building the metadata URL.

In other words, we should not assume that serverUrl is already normalized; we should normalize it earlier in this scenario and then this exact-issuer check will be correct in all cases.

@nbarbettini

RFC 8414 §3.1 says that if the issuer identifier contains a path component, any trailing slash MUST be removed before building the metadata URL.

The removal of the trailing slash from the issuer (serverUrl) when constructing the metadata URL is performed in L98. Since RFC 8414 §3.1 does not says that the issuer identifier must exclude a trailing slash, I think that removing the slash during issuer comparison is not necessary. What do you think?

hm yea maybe we just want to assume the URL you pass in is the issuer URL?

agree with @Michito-Okai that it doesn't say the issuer itself can't have a trailing slash, just that you have to remove it before inserting the /.well-known to find the metadata

@Michito-Okai I missed L98, sry! That looks correct on the discovery side.

I still think there is technically a gap on the comparison side, but it may not matter in the real world. My reasoning chain is:

RFC 8414 §3.1 says that if the issuer identifier contains a path component, any trailing / “MUST be removed” before constructing the metadata URL. (L98 ✅)

RFC 8414 §3.3 then says the returned issuer “MUST be identical” to the issuer identifier value into which the well-known string was inserted to retrieve the metadata.

Additionally, OIDC Discovery §4.3 says it this way: "The issuer value returned MUST be identical to the Issuer URL that was used as the prefix to /.well-known/openid-configuration".

In both RFC 8414 and OIDC Discovery the language is slightly vague because it does not explicitly say "issuers with a path component MUST NOT have a trailing slash in their canonical form". But I do think that is strongly implied by (1) and (2), and reinforced by (3) - if the issuer value must be identical to the prefix, the returned issuer value can't have a trailing slash because GET /issuer1//.well-known/openid-configuration is invalid.

Note that this only applies to issuers with a path component: https://as.example.com/issuer1 not https://as.example.com

Now, it may not practically matter because many authorization servers are not path-based, and the ones that are probably do this correctly already. MCP conformance isn't trying to be a canonical OIDC conformance suite. If we don't want to tackle this right now, or it's too edgy of an edge case, we could revisit later.

Thank you for your explanation.
I agree that the language is slightly vague in both RFC 8414 and OIDC Discovery, and that the issuer URL should not include a trailing slash.
However, since the specifications do not explicity state that issuer URL should not include a trailing slash, I think that we need not to arbitrarily remove a trailing slash.
For example, there may be a case where the issuer URL could be https://auth.example.com/issuer/ and metadata URL could be https://auth.example.com/.well-known/openid-configuration/issuer.
I would like to perform the check assuming that the value set in the url parameter is the issuer, i.e., the iss claim returned in the metadata.
What do you think? @nbarbettini @pcarleton

pkg-pr-new · 2026-03-26T13:09:30Z

Open in StackBlitz

npx https://pkg.pr.new/@modelcontextprotocol/conformance@170

commit: 60a0b4e

tnorimat reviewed Feb 26, 2026

View reviewed changes

Michito-Okai force-pushed the authorization-server-metadata branch from 9617458 to eacc9e7 Compare February 27, 2026 02:37

pcarleton requested changes Mar 17, 2026

View reviewed changes

Michito-Okai force-pushed the authorization-server-metadata branch 2 times, most recently from 29f04b5 to 899a5f3 Compare March 18, 2026 23:40

tnorimat reviewed Mar 19, 2026

View reviewed changes

pcarleton previously approved these changes Mar 25, 2026

View reviewed changes

pcarleton enabled auto-merge (squash) March 25, 2026 10:57

nbarbettini reviewed Mar 25, 2026

View reviewed changes

feat: add conformance test for authorization server metadata

60a0b4e

auto-merge was automatically disabled March 26, 2026 03:12
Head branch was pushed to by a user without write access

Michito-Okai dismissed pcarleton’s stale review via 60a0b4e March 26, 2026 03:12

Michito-Okai force-pushed the authorization-server-metadata branch from 899a5f3 to 60a0b4e Compare March 26, 2026 03:12

Conversation

Michito-Okai commented Feb 25, 2026

Motivation and Context

How Has This Been Tested?

Breaking Changes

Types of changes

Checklist

Uh oh!

tnorimat left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

tnorimat Mar 18, 2026 •

edited

Loading

tnorimat left a comment •

edited

Loading