I assume something like this as robots.txt would do the job... but the base HTTP support level would need to be changed to admit this? User-agent: * Disallow: /agent Disallow: /object