# All robots will spider the domain User-agent: * Disallow: /Merchant2/ # Disallow directory /cgi-bin/ User-agent: * Disallow: /cgi/ User-agent: EmailCollector Disallow: / User-agent: TurnitinBot Disallow: /