Add content negotiation to content app for JSON distribution representation

[dkliban]

Problem Statement

The pulpcore content app has no content negotiation — the Accept request header is never inspected. When a client requests a distribution's base path or content path, it always gets an HTML directory listing or a binary file stream. There is no standardized way for a client to request a JSON representation of a distribution or its contents.

Some plugins have worked around this with custom URL patterns (e.g., PythonDistribution.content_handler serves JSON at pypi/*/json), but this is ad-hoc and plugin-specific. There is no common mechanism that all distribution types can use.

Proposed Solution

1. Content negotiation in the content app

Handler._match_and_stream() should inspect the Accept request header. When a client sends Accept: application/json, the handler should call a new method on the resolved distribution to produce a JSON response instead of serving files or HTML directory listings.

2. New Distribution extension method

Add a method to the base Distribution model that plugin distributions can override:

class Distribution(MasterModel):
    def content_handler_json(self, path):
        """
        Return a JSON representation of the distribution's contents at the given path.
        
        Plugin distributions override this to provide content-type-specific data.
        Return None to fall through to default behavior.
        """
        return None

3. Default behavior

The base Distribution could provide a minimal default JSON response with distribution metadata (name, base_path, repository info). Plugin distributions would enrich this with content-type-specific data. For example:

• Maven: list MavenPackage content (GAV coordinates, POM metadata)
• RPM: list packages by NEVRA
• Python: list Python packages by name/version
• File: list files by relative path

4. Integration with existing content_handler

The content negotiation check should happen in _match_and_stream (handler.py) after resolving the distribution but before calling the existing content_handler. When Accept: application/json is present:

1. Call distribution.content_handler_json(rel_path)
2. If it returns a response, serve it
3. If it returns None, fall through to existing behavior

This preserves backward compatibility — distributions that don't override the method behave exactly as before.

Use Cases

• CLI tools and automation scripts can query distribution contents programmatically
• Web UIs can fetch structured data about available packages without screen-scraping HTML
• Plugin distributions get a standardized extension point instead of inventing ad-hoc JSON endpoints