Not op, but a common tactic is to put honeypots in a page set to be forbidden in Robots.txt. Not respecting Robots.txt is making you a bad crawler. I have marginal blog, but its somewhat indexed and it gets tons of traffics from crawlers
Yep this is how I'm doing it. It's marked as my super exciting client archive full of high-res RAW photos that all bots are told not to index in robots.txt, only for some weird reason the link never gets rendered to the page :>
I also explicitly tell all crawlers not to index the images on my site, actually, so I feel even better about serving the ones that do complete junk.
Not sure why I get more crawler traffic than I do legitimate traffic tbh - it's a relatively new website that I've been trying to sort out the SEO for as I'm trying to get photography work, so it's probably started cropping up on the radar of scrapers looking for training data.
If you have a website and you're optimizing it for GEO, you can't assume that the agents are going to have the glue. So as the person maintaining the website you implement as much of the glue as possible.
That sounds completely backwards. It seems, again, obvious to me that it would be easier to add HTML->markdown converters to agents, given that there are orders of magnitude more websites out there compared to agent.
If your agent sucks so bad that it isn’t capable of consuming HTML without tokenizing the whole damn thing, wouldn’t you just use an agent that isn’t such a mess?
This whole thing kinda sounds crazy inefficient to me.
Mintlify | Senior SWE | Onsite San Francisco
Mintlify is a developer documentation platform that helps companies and businesses build and maintain public-facing documentation (X, Coinbase, Anthropic, Scale AI, and over 20% of the last YC batch.
The team is full of people who are builders at heart and are genuinely passionate about elevating developer experiences. Our team is small but mighty (28 people, including 10 engineers) with over 10,000 companies using us.
We’re hiring all across the board - but in particular, we’re focusing on maturing our engineering organization and hiring more senior team members. If you’re ready to do impactful, hands-on work, we’re ready to chat. Apply at https://mintlify.com/careers
Join us in our mission to empower developers worldwide! We're a developer documentation platform that helps companies build and maintain public-facing documentation (X, Anthropic, Scale AI and over 20% of the last YC batch).
As for the team, we're builders at heart. Our small-but-mighty team is going from 0 to 1, laying down the foundations, and scaling (yes, all of the above). We prioritize attention to detail and are passionate about building tasteful user experiences.
If you’re ready to do impactful, hands-on work, we’re ready to chat.
Apply at https://mintlify.com/careers
Love it - I'm one of the cofounders of Mintlify and I love the thoughtfulness and intentionality you put here. We've found the navigation and hierarchy to mean a lot.
I also love your future docs roadmap - it aligns with our roadmap and some things are already there!
- API keys are locally cached across the API reference. Let me know if there are more fields you think would be convenient to cache and we can investigate.
- Whichever programming language you choose for a code block should also sync across pages.
- We also launched AI translations so you can support docs in any language!
- We also have AI Chat and are planning on devoting more resources to it so that it's best in class.
Great to hear! Lots more we want to add for sure. One thing that would be nice is a way to directly embed code snippets referenced in my OpenAPI spec into the content of a particular page. I know you have <RequestExample /> and <ResponseExample /> but you have to write the code snippet every time for those (unless you use a custom snippet, but I can see that being cumbersome at scale). Plus these are positioned outside of the main page content. Feels like if I've given mint my openapi spec, with code snippets already in there, then in the same way mint can generate an api playground, surely it could generate code snippets as well. This would be nice as the snippets I have defined in my api spec use our official sdks, so are more relevant than those mint generates itself.
Mintlify | mintlify.com | AI Engineer | On-site in San Francisco
Mintlify is building the modern standard for developer documentation. Our customers include the fastest growing AI companies such as X, Anthropic, Cursor, Perplexity, Scale AI, alongside 6000+ other companies.
As for the team, we're builders at heart. Our small-but-mighty team is going from 0 to 1, laying down the foundations, and scaling (yes, all of the above). We prioritize attention to detail and are passionate about building tasteful user experiences.
Mintlify is looking for an AI Engineer to help us build, train, and optimize the first applications of AI in our product.
As an AI Engineer, you will be working to fine-tune LLMs, create RAG pipelines, and understand model performance. You will work closely with the founding engineering team to integrate AI directly into our products.
Mintlify | mintlify.com | Product Engineers | On-site in San Francisco
Join us in our mission to empower developers worldwide! We're a developer documentation platform that helps companies build and maintain public-facing documentation (X, Anthropic, Scale AI and over 20% of the last YC batch).
As for the team, we're builders at heart. Our small-but-mighty team is going from 0 to 1, laying down the foundations, and scaling (yes, all of the above). We prioritize attention to detail and are passionate about building tasteful user experiences.
If you’re ready to do impactful, hands-on work, we’re ready to chat.
